2023-11-20 15:43:22,554 INFO [train_asr.py:1289] (0/4) Training started
2023-11-20 15:43:22,560 INFO [train_asr.py:1299] (0/4) Device: cuda:0
2023-11-20 15:43:22,562 INFO [train_asr.py:1311] (0/4) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 50, 'reset_interval': 200, 'valid_interval': 3000, 'feature_dim': 80, 'subsampling_factor': 4, 'warm_step': 2000, 'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '2b2ac14b326d61d79d04e53fbd69b1ff6d630411', 'k2-git-date': 'Thu Aug 24 05:58:26 2023', 'lhotse-version': '1.16.0', 'torch-version': '2.0.1+cu117', 'torch-cuda-available': True, 'torch-cuda-version': '11.7', 'python-version': '3.1', 'icefall-git-branch': 'multi_KD', 'icefall-git-sha1': '16e77b48-dirty', 'icefall-git-date': 'Mon Nov 20 11:32:19 2023', 'icefall-path': '/star-xy/softwares/icefall_development/icefall_multi_KD', 'k2-path': '/star-xy/softwares/k2_development/k2/k2/python/k2/__init__.py', 'lhotse-path': '/star-xy/softwares/anaconda3/envs/multi_KD/lib/python3.10/site-packages/lhotse/__init__.py', 'hostname': 'de-74279-k2-train-6-0423201309-7c68fd68fb-qfn6b', 'IP address': '10.177.58.19'}, 'world_size': 4, 'master_port': 13490, 'tensorboard': True, 'num_epochs': 40, 'start_epoch': 15, 'start_batch': 0, 'exp_dir': PosixPath('multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0'), 'bpe_model': 'data/lang_bpe_500/bpe.model', 'base_lr': 0.045, 'lr_batches': 7500, 'lr_epochs': 3.5, 'ref_duration': 600, 'context_size': 2, 'prune_range': 5, 'lm_scale': 0.25, 'am_scale': 0.0, 'simple_loss_scale': 0.5, 'ctc_loss_scale': 0.2, 'audio_tagging_loss_scale': 1.0, 'seed': 42, 'print_diagnostics': False, 'inf_check': False, 'save_every_n': 4000, 'keep_last_k': 30, 'average_period': 200, 'use_fp16': True, 'do_finetune': False, 'init_modules': None, 'freeze_modules': None, 'finetune_ckpt': None, 'num_encoder_layers': '2,2,3,4,3,2', 'downsampling_factor': '1,2,4,8,4,2', 'feedforward_dim': '512,768,1024,1536,1024,768', 'num_heads': '4,4,4,8,4,4', 'encoder_dim': '192,256,384,512,384,256', 'query_head_dim': '32', 'value_head_dim': '12', 'pos_head_dim': '4', 'pos_dim': 48, 'encoder_unmasked_dim': '192,192,256,256,256,192', 'cnn_module_kernel': '31,31,15,15,15,31', 'decoder_dim': 512, 'joiner_dim': 512, 'causal': False, 'chunk_size': '16,32,64,-1', 'left_context_frames': '64,128,256,-1', 'use_transducer': True, 'use_ctc': False, 'do_audio_tagging': True, 'use_encoder_projection': False, 'encoder_projection_dim': -1, 'freeze_encoder': False, 'freezing_encoder_layer_index': '-1', 'freeze_encoder_steps': -1, 'encoder_lr_scale': 1.0, 'full_libri': True, 'mini_libri': False, 'use_vox2': False, 'use_libriheavy': False, 'libriheavy_subset': 'small', 'use_audioset': True, 'audioset_subset': 'unbalanced', 'manifest_dir': PosixPath('data/fbank'), 'max_duration': 1000, 'bucketing_sampler': False, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.0, 'on_the_fly_feats': False, 'shuffle': True, 'drop_last': True, 'return_cuts': True, 'num_workers': 2, 'enable_spec_aug': True, 'spec_aug_time_warp_factor': 80, 'enable_musan': True, 'enable_audioset': False, 'use_musan_separately': False, 'input_strategy': 'PrecomputedFeatures', 'drop_features': False, 'return_audio': False, 'use_beats': True, 'use_ecapa': True, 'use_whisper': True, 'whisper_mvq': False, 'beats_ckpt': 'data/models/BEATs/BEATs_iter3_plus_AS2M_finetuned_on_AS2M_cpt2.pt', 'whisper_version': 'small.en', 'blank_id': 0, 'vocab_size': 500}
2023-11-20 15:43:22,563 INFO [train_asr.py:1320] (0/4) About to create model
2023-11-20 15:43:23,568 INFO [train_asr.py:1324] (0/4) Number of model parameters: 65819362
2023-11-20 15:43:24,367 INFO [checkpoint.py:112] (0/4) Loading checkpoint from multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-14.pt
2023-11-20 15:43:27,114 INFO [checkpoint.py:131] (0/4) Loading averaged model
2023-11-20 15:43:27,446 INFO [train_asr.py:1352] (0/4) Setting the lr scale of parameters in encoder and encoder_embed to 1.0
2023-11-20 15:43:32,075 INFO [train_asr.py:1361] (0/4) Using DDP
2023-11-20 15:43:32,390 INFO [train_asr.py:1384] (0/4) Loading optimizer state dict
2023-11-20 15:43:33,178 INFO [train_asr.py:1392] (0/4) Loading scheduler state dict
2023-11-20 15:43:33,181 INFO [train_asr.py:1414] (0/4) Getting audioset cuts
2023-11-20 15:43:33,181 INFO [kd_datamodule.py:796] (0/4) About to get the audioset cuts.
2023-11-20 15:43:33,184 INFO [train_asr.py:1420] (0/4) Using mux to combine Librispeech with audioset
2023-11-20 15:43:33,184 INFO [train_asr.py:1430] (0/4) CutSet(len=2748469) [underlying data type: <class 'lhotse.lazy.LazyIteratorMultiplexer'>]
2023-11-20 15:43:48,692 INFO [kd_datamodule.py:396] (0/4) Enable MUSAN
2023-11-20 15:43:48,692 INFO [kd_datamodule.py:397] (0/4) About to get Musan cuts
2023-11-20 15:43:52,253 INFO [kd_datamodule.py:427] (0/4) Enable SpecAugment
2023-11-20 15:43:52,253 INFO [kd_datamodule.py:428] (0/4) Time warp factor: 80
2023-11-20 15:43:52,253 INFO [kd_datamodule.py:438] (0/4) Num frame mask: 10
2023-11-20 15:43:52,253 INFO [kd_datamodule.py:451] (0/4) About to create train dataset
2023-11-20 15:43:52,257 INFO [kd_datamodule.py:487] (0/4) Using SimpleCutSampler
2023-11-20 15:43:52,257 INFO [kd_datamodule.py:495] (0/4) About to create train dataloader
2023-11-20 15:43:52,274 INFO [kd_datamodule.py:814] (0/4) About to get the audioset eval cuts.
2023-11-20 15:43:52,276 INFO [train_asr.py:1494] (0/4) CutSet(len=20681) [underlying data type: <class 'lhotse.lazy.LazyMapper'>]
2023-11-20 15:43:52,368 INFO [kd_datamodule.py:529] (0/4) About to create dev dataset
2023-11-20 15:43:53,145 INFO [kd_datamodule.py:550] (0/4) About to create dev dataloader
2023-11-20 15:43:53,146 INFO [train_asr.py:1508] (0/4) Loading grad scaler state dict
2023-11-20 15:44:29,656 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.76 vs. limit=15.0
2023-11-20 15:44:29,790 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.32 vs. limit=22.5
2023-11-20 15:44:30,185 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 0, loss[loss=0.08484, simple_loss=0.09368, pruned_loss=0.01497, audio_tagging_loss=0.02302, over 16188.00 frames. ], tot_loss[loss=0.08484, simple_loss=0.09368, pruned_loss=0.01497, audio_tagging_loss=0.02302, over 16188.00 frames. ], batch size: 58, lr: 4.68e-03, grad_scale: 32.0
2023-11-20 15:44:30,191 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-20 15:45:06,716 INFO [train_asr.py:1253] (0/4) Epoch 15, validation: loss=0.06153, simple_loss=0.05347, pruned_loss=0.005654, audio_tagging_loss=0.02914, over 4681554.00 frames. 
2023-11-20 15:45:06,718 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-20 15:45:07,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1122200.0, ans=0.2
2023-11-20 15:45:11,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=13.55 vs. limit=15.0
2023-11-20 15:45:12,413 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.106e+01 8.292e+01 9.006e+01 9.945e+01 1.226e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-20 15:45:23,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1122266.6666666667, ans=0.125
2023-11-20 15:45:31,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1122266.6666666667, ans=0.1
2023-11-20 15:45:32,142 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168350
2023-11-20 15:45:37,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1122333.3333333333, ans=0.125
2023-11-20 15:45:38,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1122333.3333333333, ans=0.0
2023-11-20 15:45:48,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1122400.0, ans=0.125
2023-11-20 15:46:12,514 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 50, loss[loss=0.1085, simple_loss=0.1376, pruned_loss=0.02546, audio_tagging_loss=0.0142, over 16483.00 frames. ], tot_loss[loss=0.08778, simple_loss=0.09884, pruned_loss=0.01892, audio_tagging_loss=0.01944, over 691726.11 frames. ], batch size: 58, lr: 4.67e-03, grad_scale: 32.0
2023-11-20 15:46:34,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1122600.0, ans=0.0
2023-11-20 15:46:35,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1122600.0, ans=0.125
2023-11-20 15:46:38,102 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168400
2023-11-20 15:46:49,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1122666.6666666667, ans=0.0
2023-11-20 15:47:09,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1122800.0, ans=0.125
2023-11-20 15:47:19,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1122800.0, ans=0.1
2023-11-20 15:47:21,470 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 100, loss[loss=0.06961, simple_loss=0.08902, pruned_loss=0.01071, audio_tagging_loss=0.01439, over 14689.00 frames. ], tot_loss[loss=0.0874, simple_loss=0.09971, pruned_loss=0.01919, audio_tagging_loss=0.01836, over 1214467.72 frames. ], batch size: 55, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:47:27,793 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.479e+01 8.856e+01 9.417e+01 9.999e+01 1.434e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-20 15:47:44,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168450
2023-11-20 15:48:21,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1123133.3333333333, ans=0.125
2023-11-20 15:48:27,323 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 150, loss[loss=0.08482, simple_loss=0.1108, pruned_loss=0.01758, audio_tagging_loss=0.01186, over 15252.00 frames. ], tot_loss[loss=0.0851, simple_loss=0.09935, pruned_loss=0.01879, audio_tagging_loss=0.01664, over 1625374.93 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:48:38,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1123266.6666666667, ans=0.125
2023-11-20 15:48:50,713 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168500
2023-11-20 15:49:03,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1123333.3333333333, ans=0.04949747468305833
2023-11-20 15:49:13,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1123400.0, ans=0.0
2023-11-20 15:49:31,881 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 200, loss[loss=0.07355, simple_loss=0.09887, pruned_loss=0.01669, audio_tagging_loss=0.007423, over 15596.00 frames. ], tot_loss[loss=0.0839, simple_loss=0.1007, pruned_loss=0.01897, audio_tagging_loss=0.01456, over 1945309.20 frames. ], batch size: 57, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:49:32,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1123533.3333333333, ans=0.2
2023-11-20 15:49:32,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1123533.3333333333, ans=0.95
2023-11-20 15:49:38,090 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.900e+01 8.283e+01 8.917e+01 9.795e+01 1.407e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-20 15:49:38,825 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.18 vs. limit=15.0
2023-11-20 15:49:56,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168550
2023-11-20 15:50:15,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1123733.3333333333, ans=0.1
2023-11-20 15:50:36,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1123800.0, ans=0.125
2023-11-20 15:50:38,422 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 250, loss[loss=0.06777, simple_loss=0.07761, pruned_loss=0.01442, audio_tagging_loss=0.01454, over 13944.00 frames. ], tot_loss[loss=0.08275, simple_loss=0.1009, pruned_loss=0.0191, audio_tagging_loss=0.01318, over 2193164.28 frames. ], batch size: 55, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:50:52,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1123933.3333333333, ans=0.125
2023-11-20 15:51:00,749 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168600
2023-11-20 15:51:09,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1124000.0, ans=0.125
2023-11-20 15:51:12,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1124000.0, ans=0.0
2023-11-20 15:51:23,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1124066.6666666667, ans=0.05
2023-11-20 15:51:24,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1124066.6666666667, ans=0.125
2023-11-20 15:51:25,801 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 15:51:36,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1124133.3333333333, ans=0.125
2023-11-20 15:51:43,545 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 300, loss[loss=0.07672, simple_loss=0.1022, pruned_loss=0.01453, audio_tagging_loss=0.01109, over 15094.00 frames. ], tot_loss[loss=0.08278, simple_loss=0.1021, pruned_loss=0.01949, audio_tagging_loss=0.01225, over 2391428.32 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:51:46,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1124200.0, ans=0.125
2023-11-20 15:51:49,689 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.496e+01 9.198e+01 1.000e+02 1.340e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-20 15:52:01,655 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.42 vs. limit=15.0
2023-11-20 15:52:04,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1124266.6666666667, ans=0.2
2023-11-20 15:52:05,910 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168650
2023-11-20 15:52:36,758 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.81 vs. limit=15.0
2023-11-20 15:52:47,374 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 350, loss[loss=0.08854, simple_loss=0.1135, pruned_loss=0.02089, audio_tagging_loss=0.01088, over 16959.00 frames. ], tot_loss[loss=0.08201, simple_loss=0.1019, pruned_loss=0.01949, audio_tagging_loss=0.01157, over 2544880.09 frames. ], batch size: 61, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:53:01,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1124600.0, ans=0.1
2023-11-20 15:53:11,772 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168700
2023-11-20 15:53:18,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1124666.6666666667, ans=0.0
2023-11-20 15:53:19,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1124666.6666666667, ans=0.125
2023-11-20 15:53:50,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1124800.0, ans=0.0
2023-11-20 15:53:52,672 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 400, loss[loss=0.07295, simple_loss=0.09849, pruned_loss=0.01392, audio_tagging_loss=0.009782, over 16494.00 frames. ], tot_loss[loss=0.0813, simple_loss=0.1012, pruned_loss=0.01953, audio_tagging_loss=0.01115, over 2661705.94 frames. ], batch size: 61, lr: 4.67e-03, grad_scale: 32.0
2023-11-20 15:53:59,451 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.749e+01 8.311e+01 9.076e+01 1.027e+02 1.296e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-20 15:54:15,670 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168750
2023-11-20 15:54:23,342 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 15:54:57,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0
2023-11-20 15:54:57,582 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 450, loss[loss=0.0789, simple_loss=0.09415, pruned_loss=0.01804, audio_tagging_loss=0.01378, over 15964.00 frames. ], tot_loss[loss=0.08055, simple_loss=0.1007, pruned_loss=0.01937, audio_tagging_loss=0.01085, over 2745526.05 frames. ], batch size: 61, lr: 4.67e-03, grad_scale: 32.0
2023-11-20 15:54:57,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1125200.0, ans=0.0
2023-11-20 15:55:14,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1125266.6666666667, ans=0.125
2023-11-20 15:55:20,004 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168800
2023-11-20 15:55:38,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1125400.0, ans=0.1
2023-11-20 15:56:03,124 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 500, loss[loss=0.08953, simple_loss=0.1132, pruned_loss=0.02011, audio_tagging_loss=0.01282, over 15242.00 frames. ], tot_loss[loss=0.07967, simple_loss=0.09995, pruned_loss=0.01913, audio_tagging_loss=0.01057, over 2806822.14 frames. ], batch size: 57, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:56:10,664 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.084e+01 8.651e+01 9.631e+01 1.244e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-20 15:56:27,750 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168850
2023-11-20 15:56:29,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1125666.6666666667, ans=15.0
2023-11-20 15:56:36,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1125666.6666666667, ans=0.125
2023-11-20 15:56:40,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1125666.6666666667, ans=0.125
2023-11-20 15:56:41,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1125666.6666666667, ans=0.95
2023-11-20 15:56:41,237 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 15:56:49,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1125733.3333333333, ans=0.125
2023-11-20 15:56:57,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1125800.0, ans=10.0
2023-11-20 15:57:00,119 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 15:57:03,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1125800.0, ans=0.125
2023-11-20 15:57:10,349 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 550, loss[loss=0.07529, simple_loss=0.09682, pruned_loss=0.01765, audio_tagging_loss=0.009224, over 14874.00 frames. ], tot_loss[loss=0.07935, simple_loss=0.09979, pruned_loss=0.01901, audio_tagging_loss=0.01044, over 2866528.86 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:57:14,782 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.14 vs. limit=15.0
2023-11-20 15:57:26,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1125933.3333333333, ans=0.1
2023-11-20 15:57:27,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1125933.3333333333, ans=0.125
2023-11-20 15:57:31,844 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 15:57:34,025 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168900
2023-11-20 15:57:38,522 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.15 vs. limit=15.0
2023-11-20 15:58:03,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1126133.3333333333, ans=0.125
2023-11-20 15:58:16,251 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 600, loss[loss=0.07042, simple_loss=0.08604, pruned_loss=0.01597, audio_tagging_loss=0.01142, over 15177.00 frames. ], tot_loss[loss=0.07904, simple_loss=0.09918, pruned_loss=0.01905, audio_tagging_loss=0.0104, over 2898670.60 frames. ], batch size: 58, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:58:23,543 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 7.985e+01 8.751e+01 9.752e+01 1.592e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-20 15:58:37,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1126266.6666666667, ans=0.5
2023-11-20 15:58:38,534 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 168950
2023-11-20 15:58:51,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1126333.3333333333, ans=0.0
2023-11-20 15:58:53,054 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.19 vs. limit=15.0
2023-11-20 15:58:57,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1126400.0, ans=0.2
2023-11-20 15:59:01,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1126400.0, ans=0.05
2023-11-20 15:59:02,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1126400.0, ans=0.125
2023-11-20 15:59:20,622 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 650, loss[loss=0.05887, simple_loss=0.07356, pruned_loss=0.01306, audio_tagging_loss=0.009025, over 14087.00 frames. ], tot_loss[loss=0.0781, simple_loss=0.09791, pruned_loss=0.0187, audio_tagging_loss=0.01044, over 2934648.03 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 15:59:26,154 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.27 vs. limit=15.0
2023-11-20 15:59:27,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1126533.3333333333, ans=0.2
2023-11-20 15:59:39,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1126600.0, ans=0.125
2023-11-20 15:59:44,705 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169000
2023-11-20 15:59:48,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.30 vs. limit=22.5
2023-11-20 15:59:56,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1126666.6666666667, ans=0.2
2023-11-20 16:00:00,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.24 vs. limit=22.5
2023-11-20 16:00:00,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1126733.3333333333, ans=0.125
2023-11-20 16:00:07,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1126733.3333333333, ans=0.2
2023-11-20 16:00:07,713 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.84 vs. limit=22.5
2023-11-20 16:00:14,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1126800.0, ans=0.125
2023-11-20 16:00:26,807 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 700, loss[loss=0.09316, simple_loss=0.1124, pruned_loss=0.02464, audio_tagging_loss=0.01234, over 14749.00 frames. ], tot_loss[loss=0.07804, simple_loss=0.09816, pruned_loss=0.01871, audio_tagging_loss=0.01025, over 2962630.85 frames. ], batch size: 55, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 16:00:34,823 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.436e+01 8.027e+01 8.656e+01 9.309e+01 1.974e+02, threshold=1.731e+02, percent-clipped=1.0
2023-11-20 16:00:50,222 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169050
2023-11-20 16:00:57,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1127000.0, ans=0.0
2023-11-20 16:01:00,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1127000.0, ans=0.0
2023-11-20 16:01:12,138 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.28 vs. limit=15.0
2023-11-20 16:01:16,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1127066.6666666667, ans=0.05
2023-11-20 16:01:21,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1127133.3333333333, ans=0.125
2023-11-20 16:01:21,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1127133.3333333333, ans=0.0
2023-11-20 16:01:31,594 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 750, loss[loss=0.05944, simple_loss=0.06728, pruned_loss=0.01256, audio_tagging_loss=0.01324, over 14736.00 frames. ], tot_loss[loss=0.07854, simple_loss=0.09907, pruned_loss=0.01886, audio_tagging_loss=0.01015, over 2986206.27 frames. ], batch size: 56, lr: 4.67e-03, grad_scale: 16.0
2023-11-20 16:01:35,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1127200.0, ans=0.05
2023-11-20 16:01:52,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1127266.6666666667, ans=0.125
2023-11-20 16:01:53,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1127266.6666666667, ans=0.0
2023-11-20 16:01:54,454 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169100
2023-11-20 16:02:03,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1127333.3333333333, ans=0.125
2023-11-20 16:02:12,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1127400.0, ans=0.125
2023-11-20 16:02:17,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1127400.0, ans=0.5
2023-11-20 16:02:34,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=1127466.6666666667, ans=10.0
2023-11-20 16:02:36,482 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 800, loss[loss=0.1037, simple_loss=0.126, pruned_loss=0.02963, audio_tagging_loss=0.01111, over 14824.00 frames. ], tot_loss[loss=0.07983, simple_loss=0.1006, pruned_loss=0.01943, audio_tagging_loss=0.0101, over 2997537.22 frames. ], batch size: 56, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:02:43,909 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.081e+01 8.651e+01 9.218e+01 1.161e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-20 16:03:00,986 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169150
2023-11-20 16:03:07,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1127666.6666666667, ans=0.125
2023-11-20 16:03:12,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1127666.6666666667, ans=0.09899494936611666
2023-11-20 16:03:26,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.74 vs. limit=22.5
2023-11-20 16:03:33,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1127800.0, ans=0.125
2023-11-20 16:03:41,758 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 850, loss[loss=0.09671, simple_loss=0.1201, pruned_loss=0.02681, audio_tagging_loss=0.009844, over 14869.00 frames. ], tot_loss[loss=0.07882, simple_loss=0.0989, pruned_loss=0.0191, audio_tagging_loss=0.01027, over 2997751.63 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:03:45,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1127866.6666666667, ans=0.2
2023-11-20 16:04:05,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169200
2023-11-20 16:04:24,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1128066.6666666667, ans=0.125
2023-11-20 16:04:31,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1128066.6666666667, ans=0.2
2023-11-20 16:04:32,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1128066.6666666667, ans=0.04949747468305833
2023-11-20 16:04:43,696 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.19 vs. limit=15.0
2023-11-20 16:04:48,000 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 900, loss[loss=0.08611, simple_loss=0.1036, pruned_loss=0.02239, audio_tagging_loss=0.0119, over 14195.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.09842, pruned_loss=0.01893, audio_tagging_loss=0.01034, over 3009052.82 frames. ], batch size: 55, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:04:52,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.78 vs. limit=15.0
2023-11-20 16:04:55,434 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 8.050e+01 8.717e+01 9.449e+01 1.348e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-20 16:05:02,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1128266.6666666667, ans=0.0
2023-11-20 16:05:11,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169250
2023-11-20 16:05:13,199 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.93 vs. limit=15.0
2023-11-20 16:05:31,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.44 vs. limit=5.0
2023-11-20 16:05:37,484 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.24 vs. limit=15.0
2023-11-20 16:05:41,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.40 vs. limit=6.0
2023-11-20 16:05:53,005 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 950, loss[loss=0.09961, simple_loss=0.1261, pruned_loss=0.02706, audio_tagging_loss=0.009472, over 15501.00 frames. ], tot_loss[loss=0.07945, simple_loss=0.1, pruned_loss=0.01925, audio_tagging_loss=0.01018, over 3021471.07 frames. ], batch size: 56, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:05:56,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1128533.3333333333, ans=0.0
2023-11-20 16:06:17,457 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169300
2023-11-20 16:06:26,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=1128666.6666666667, ans=0.1
2023-11-20 16:06:29,416 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:06:48,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1128800.0, ans=0.125
2023-11-20 16:06:57,907 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1000, loss[loss=0.06476, simple_loss=0.07971, pruned_loss=0.01321, audio_tagging_loss=0.01169, over 15585.00 frames. ], tot_loss[loss=0.0781, simple_loss=0.09821, pruned_loss=0.01877, audio_tagging_loss=0.01022, over 3033094.26 frames. ], batch size: 58, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:07:02,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1128866.6666666667, ans=0.125
2023-11-20 16:07:06,530 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.366e+01 7.987e+01 8.758e+01 9.641e+01 1.276e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-20 16:07:22,260 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169350
2023-11-20 16:07:22,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1128933.3333333333, ans=0.0
2023-11-20 16:07:25,408 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.74 vs. limit=15.0
2023-11-20 16:07:25,951 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:07:37,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.85 vs. limit=10.0
2023-11-20 16:07:45,692 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.38 vs. limit=6.0
2023-11-20 16:07:45,848 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.55 vs. limit=15.0
2023-11-20 16:07:45,891 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.13 vs. limit=15.0
2023-11-20 16:08:01,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1129133.3333333333, ans=0.95
2023-11-20 16:08:04,568 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1050, loss[loss=0.08153, simple_loss=0.1075, pruned_loss=0.01847, audio_tagging_loss=0.009312, over 14294.00 frames. ], tot_loss[loss=0.07802, simple_loss=0.0985, pruned_loss=0.01869, audio_tagging_loss=0.01008, over 3032103.61 frames. ], batch size: 54, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:08:06,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1129200.0, ans=0.0
2023-11-20 16:08:17,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1129266.6666666667, ans=0.05
2023-11-20 16:08:27,021 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169400
2023-11-20 16:08:31,907 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.58 vs. limit=15.0
2023-11-20 16:09:00,056 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.89 vs. limit=12.0
2023-11-20 16:09:09,317 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1100, loss[loss=0.06146, simple_loss=0.07724, pruned_loss=0.01322, audio_tagging_loss=0.009619, over 15398.00 frames. ], tot_loss[loss=0.0768, simple_loss=0.09695, pruned_loss=0.01831, audio_tagging_loss=0.01002, over 3033310.96 frames. ], batch size: 59, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:09:11,872 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:09:12,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1129533.3333333333, ans=0.125
2023-11-20 16:09:16,781 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.560e+01 8.137e+01 8.767e+01 9.510e+01 1.319e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-20 16:09:17,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1129533.3333333333, ans=0.0
2023-11-20 16:09:18,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1129533.3333333333, ans=0.2
2023-11-20 16:09:22,451 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.33 vs. limit=10.0
2023-11-20 16:09:28,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1129600.0, ans=0.1
2023-11-20 16:09:33,007 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169450
2023-11-20 16:10:04,615 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.45 vs. limit=22.5
2023-11-20 16:10:05,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1129800.0, ans=0.2
2023-11-20 16:10:13,842 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1150, loss[loss=0.1062, simple_loss=0.147, pruned_loss=0.02752, audio_tagging_loss=0.00521, over 16132.00 frames. ], tot_loss[loss=0.07687, simple_loss=0.09711, pruned_loss=0.01832, audio_tagging_loss=0.01, over 3033401.46 frames. ], batch size: 56, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:10:17,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1129866.6666666667, ans=0.0
2023-11-20 16:10:25,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1129866.6666666667, ans=0.125
2023-11-20 16:10:38,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169500
2023-11-20 16:10:52,022 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.99 vs. limit=15.0
2023-11-20 16:11:00,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1130066.6666666667, ans=0.0
2023-11-20 16:11:05,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1130133.3333333333, ans=0.2
2023-11-20 16:11:16,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1130133.3333333333, ans=0.125
2023-11-20 16:11:21,495 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1200, loss[loss=0.08016, simple_loss=0.1119, pruned_loss=0.01885, audio_tagging_loss=0.005365, over 14763.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09641, pruned_loss=0.01815, audio_tagging_loss=0.01002, over 3023746.21 frames. ], batch size: 55, lr: 4.66e-03, grad_scale: 32.0
2023-11-20 16:11:30,092 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.644e+01 8.141e+01 8.800e+01 9.449e+01 1.223e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-20 16:11:42,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1130266.6666666667, ans=0.1
2023-11-20 16:11:43,985 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169550
2023-11-20 16:12:16,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1130466.6666666667, ans=0.2
2023-11-20 16:12:26,141 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1250, loss[loss=0.06074, simple_loss=0.0756, pruned_loss=0.01554, audio_tagging_loss=0.0074, over 14122.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09655, pruned_loss=0.01841, audio_tagging_loss=0.009878, over 3022979.11 frames. ], batch size: 53, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:12:31,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1130533.3333333333, ans=0.0
2023-11-20 16:12:45,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1130600.0, ans=0.04949747468305833
2023-11-20 16:12:49,061 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169600
2023-11-20 16:12:51,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1130666.6666666667, ans=0.125
2023-11-20 16:12:58,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1130666.6666666667, ans=0.125
2023-11-20 16:13:06,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1130733.3333333333, ans=0.125
2023-11-20 16:13:15,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1130733.3333333333, ans=0.0
2023-11-20 16:13:17,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1130800.0, ans=0.125
2023-11-20 16:13:17,731 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.74 vs. limit=22.5
2023-11-20 16:13:28,578 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.91 vs. limit=22.5
2023-11-20 16:13:30,642 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1300, loss[loss=0.0594, simple_loss=0.06205, pruned_loss=0.01178, audio_tagging_loss=0.0166, over 14869.00 frames. ], tot_loss[loss=0.07602, simple_loss=0.09577, pruned_loss=0.01816, audio_tagging_loss=0.009978, over 3029009.01 frames. ], batch size: 59, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:13:41,677 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.174e+01 8.827e+01 9.605e+01 1.804e+02, threshold=1.765e+02, percent-clipped=1.0
2023-11-20 16:13:45,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1130933.3333333333, ans=0.0
2023-11-20 16:13:53,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1130933.3333333333, ans=0.0
2023-11-20 16:13:55,586 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169650
2023-11-20 16:14:25,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1131133.3333333333, ans=0.125
2023-11-20 16:14:36,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1131200.0, ans=0.125
2023-11-20 16:14:37,402 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1350, loss[loss=0.1, simple_loss=0.1359, pruned_loss=0.02561, audio_tagging_loss=0.006478, over 15859.00 frames. ], tot_loss[loss=0.07671, simple_loss=0.09646, pruned_loss=0.01847, audio_tagging_loss=0.01001, over 3029417.33 frames. ], batch size: 59, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:14:38,024 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.21 vs. limit=6.0
2023-11-20 16:14:45,090 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.73 vs. limit=10.0
2023-11-20 16:15:00,490 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169700
2023-11-20 16:15:14,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1131400.0, ans=0.125
2023-11-20 16:15:23,989 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:15:28,542 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:15:34,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1131466.6666666667, ans=0.125
2023-11-20 16:15:36,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1131466.6666666667, ans=0.125
2023-11-20 16:15:39,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1131466.6666666667, ans=0.0
2023-11-20 16:15:42,465 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1400, loss[loss=0.06668, simple_loss=0.08026, pruned_loss=0.01653, audio_tagging_loss=0.01001, over 14929.00 frames. ], tot_loss[loss=0.07683, simple_loss=0.09646, pruned_loss=0.01861, audio_tagging_loss=0.009994, over 3035139.10 frames. ], batch size: 59, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:15:52,436 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 7.986e+01 8.672e+01 9.769e+01 1.289e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-20 16:16:05,052 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169750
2023-11-20 16:16:11,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1131666.6666666667, ans=0.0
2023-11-20 16:16:47,122 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1450, loss[loss=0.06086, simple_loss=0.07958, pruned_loss=0.01104, audio_tagging_loss=0.01003, over 15130.00 frames. ], tot_loss[loss=0.07771, simple_loss=0.09768, pruned_loss=0.01887, audio_tagging_loss=0.009995, over 3039511.19 frames. ], batch size: 57, lr: 4.66e-03, grad_scale: 16.0
2023-11-20 16:16:48,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1131866.6666666667, ans=0.125
2023-11-20 16:17:11,308 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169800
2023-11-20 16:17:25,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1132066.6666666667, ans=0.125
2023-11-20 16:17:52,590 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1500, loss[loss=0.07237, simple_loss=0.08348, pruned_loss=0.01932, audio_tagging_loss=0.01132, over 15132.00 frames. ], tot_loss[loss=0.07835, simple_loss=0.09855, pruned_loss=0.01917, audio_tagging_loss=0.009906, over 3040413.25 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 8.0
2023-11-20 16:17:54,349 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.20 vs. limit=22.5
2023-11-20 16:18:01,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1132200.0, ans=0.2
2023-11-20 16:18:01,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1132200.0, ans=0.07
2023-11-20 16:18:04,964 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.236e+01 9.090e+01 9.607e+01 1.235e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-20 16:18:16,391 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169850
2023-11-20 16:18:23,208 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.72 vs. limit=15.0
2023-11-20 16:18:32,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1132400.0, ans=0.125
2023-11-20 16:18:55,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1132466.6666666667, ans=0.125
2023-11-20 16:18:58,742 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1550, loss[loss=0.06798, simple_loss=0.08991, pruned_loss=0.01411, audio_tagging_loss=0.008914, over 14420.00 frames. ], tot_loss[loss=0.0792, simple_loss=0.09961, pruned_loss=0.01935, audio_tagging_loss=0.01004, over 3040878.46 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 8.0
2023-11-20 16:19:07,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1132533.3333333333, ans=0.125
2023-11-20 16:19:11,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1132600.0, ans=0.1
2023-11-20 16:19:20,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1132600.0, ans=0.5
2023-11-20 16:19:21,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169900
2023-11-20 16:20:03,533 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1600, loss[loss=0.06119, simple_loss=0.0707, pruned_loss=0.01386, audio_tagging_loss=0.01198, over 15355.00 frames. ], tot_loss[loss=0.07918, simple_loss=0.09953, pruned_loss=0.0192, audio_tagging_loss=0.01021, over 3046475.32 frames. ], batch size: 62, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:20:14,715 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.371e+01 8.069e+01 8.651e+01 9.390e+01 1.565e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-20 16:20:21,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1132933.3333333333, ans=0.1
2023-11-20 16:20:27,752 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 169950
2023-11-20 16:20:43,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1133066.6666666667, ans=0.2
2023-11-20 16:20:44,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.05 vs. limit=15.0
2023-11-20 16:20:45,541 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.26 vs. limit=10.0
2023-11-20 16:20:47,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1133066.6666666667, ans=0.05
2023-11-20 16:21:10,157 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1650, loss[loss=0.0872, simple_loss=0.1122, pruned_loss=0.02099, audio_tagging_loss=0.01013, over 15871.00 frames. ], tot_loss[loss=0.0791, simple_loss=0.09942, pruned_loss=0.01909, audio_tagging_loss=0.0103, over 3051129.59 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:21:33,749 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170000
2023-11-20 16:21:33,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1133266.6666666667, ans=0.015
2023-11-20 16:22:01,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1133466.6666666667, ans=0.125
2023-11-20 16:22:02,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1133466.6666666667, ans=0.0
2023-11-20 16:22:13,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=1133466.6666666667, ans=0.05
2023-11-20 16:22:15,871 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1700, loss[loss=0.08439, simple_loss=0.1044, pruned_loss=0.0217, audio_tagging_loss=0.01047, over 16625.00 frames. ], tot_loss[loss=0.07877, simple_loss=0.09917, pruned_loss=0.01887, audio_tagging_loss=0.01032, over 3055316.50 frames. ], batch size: 59, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:22:27,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.14 vs. limit=12.0
2023-11-20 16:22:27,602 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.030e+01 8.019e+01 8.780e+01 9.587e+01 1.403e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-20 16:22:35,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1133600.0, ans=0.125
2023-11-20 16:22:38,982 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170050
2023-11-20 16:22:41,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1133666.6666666667, ans=0.125
2023-11-20 16:22:43,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1133666.6666666667, ans=0.125
2023-11-20 16:22:48,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1133666.6666666667, ans=0.125
2023-11-20 16:23:05,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1133733.3333333333, ans=0.0
2023-11-20 16:23:07,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1133800.0, ans=0.125
2023-11-20 16:23:21,157 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1750, loss[loss=0.06137, simple_loss=0.06911, pruned_loss=0.0127, audio_tagging_loss=0.01413, over 15752.00 frames. ], tot_loss[loss=0.07813, simple_loss=0.09854, pruned_loss=0.01866, audio_tagging_loss=0.01021, over 3059599.64 frames. ], batch size: 60, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:23:25,588 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.14 vs. limit=15.0
2023-11-20 16:23:44,877 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170100
2023-11-20 16:23:56,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1134000.0, ans=0.125
2023-11-20 16:24:07,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1134066.6666666667, ans=0.1
2023-11-20 16:24:11,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1134066.6666666667, ans=0.0
2023-11-20 16:24:13,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1134133.3333333333, ans=0.125
2023-11-20 16:24:26,742 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1800, loss[loss=0.1005, simple_loss=0.136, pruned_loss=0.02396, audio_tagging_loss=0.008522, over 15277.00 frames. ], tot_loss[loss=0.07823, simple_loss=0.09867, pruned_loss=0.01875, audio_tagging_loss=0.01015, over 3053941.99 frames. ], batch size: 55, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:24:30,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1134200.0, ans=0.125
2023-11-20 16:24:38,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1134200.0, ans=0.125
2023-11-20 16:24:39,076 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.001e+01 8.710e+01 9.346e+01 1.351e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-20 16:24:39,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1134266.6666666667, ans=0.125
2023-11-20 16:24:51,060 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170150
2023-11-20 16:24:58,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1134333.3333333333, ans=0.1
2023-11-20 16:25:02,796 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=15.0
2023-11-20 16:25:03,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1134333.3333333333, ans=0.0
2023-11-20 16:25:27,959 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.76 vs. limit=15.0
2023-11-20 16:25:32,398 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1850, loss[loss=0.08852, simple_loss=0.1079, pruned_loss=0.02076, audio_tagging_loss=0.01382, over 15585.00 frames. ], tot_loss[loss=0.07818, simple_loss=0.0985, pruned_loss=0.01875, audio_tagging_loss=0.01018, over 3053926.43 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:25:32,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1134533.3333333333, ans=0.0
2023-11-20 16:25:33,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1134533.3333333333, ans=0.125
2023-11-20 16:25:43,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1134533.3333333333, ans=0.125
2023-11-20 16:25:48,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1134600.0, ans=0.125
2023-11-20 16:25:52,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.42 vs. limit=15.0
2023-11-20 16:25:55,425 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170200
2023-11-20 16:26:16,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1134733.3333333333, ans=0.0
2023-11-20 16:26:32,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1134800.0, ans=0.09899494936611666
2023-11-20 16:26:34,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1134800.0, ans=0.0
2023-11-20 16:26:38,618 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1900, loss[loss=0.05773, simple_loss=0.07289, pruned_loss=0.007922, audio_tagging_loss=0.01336, over 14789.00 frames. ], tot_loss[loss=0.07766, simple_loss=0.09822, pruned_loss=0.01851, audio_tagging_loss=0.01004, over 3051944.74 frames. ], batch size: 56, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:26:46,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1134866.6666666667, ans=0.025
2023-11-20 16:26:49,716 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.240e+01 7.901e+01 8.698e+01 9.524e+01 1.257e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-20 16:27:02,109 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170250
2023-11-20 16:27:15,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1135000.0, ans=0.125
2023-11-20 16:27:19,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1135066.6666666667, ans=0.0
2023-11-20 16:27:31,561 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=15.10 vs. limit=15.0
2023-11-20 16:27:35,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1135133.3333333333, ans=0.125
2023-11-20 16:27:38,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1135133.3333333333, ans=0.1
2023-11-20 16:27:43,330 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 1950, loss[loss=0.07038, simple_loss=0.09319, pruned_loss=0.01254, audio_tagging_loss=0.01125, over 15954.00 frames. ], tot_loss[loss=0.07787, simple_loss=0.09863, pruned_loss=0.01858, audio_tagging_loss=0.009978, over 3051474.24 frames. ], batch size: 59, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:27:47,202 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.34 vs. limit=15.0
2023-11-20 16:27:53,539 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:28:07,851 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170300
2023-11-20 16:28:17,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1135333.3333333333, ans=0.125
2023-11-20 16:28:19,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1135333.3333333333, ans=0.125
2023-11-20 16:28:41,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1135466.6666666667, ans=0.07
2023-11-20 16:28:49,882 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2000, loss[loss=0.04527, simple_loss=0.04178, pruned_loss=0.005724, audio_tagging_loss=0.01866, over 14854.00 frames. ], tot_loss[loss=0.07796, simple_loss=0.09857, pruned_loss=0.01865, audio_tagging_loss=0.01002, over 3049455.54 frames. ], batch size: 59, lr: 4.65e-03, grad_scale: 32.0
2023-11-20 16:29:00,828 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 7.997e+01 8.751e+01 9.649e+01 1.208e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-20 16:29:04,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.19 vs. limit=15.0
2023-11-20 16:29:12,906 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170350
2023-11-20 16:29:35,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.41 vs. limit=15.0
2023-11-20 16:29:54,324 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2050, loss[loss=0.06597, simple_loss=0.08112, pruned_loss=0.01362, audio_tagging_loss=0.01179, over 17114.00 frames. ], tot_loss[loss=0.07735, simple_loss=0.09783, pruned_loss=0.01844, audio_tagging_loss=0.01, over 3053245.97 frames. ], batch size: 62, lr: 4.65e-03, grad_scale: 32.0
2023-11-20 16:30:09,034 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:30:18,799 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170400
2023-11-20 16:30:20,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1136000.0, ans=0.125
2023-11-20 16:30:22,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1136000.0, ans=0.0
2023-11-20 16:30:30,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1136000.0, ans=0.0
2023-11-20 16:31:00,396 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2100, loss[loss=0.07742, simple_loss=0.1002, pruned_loss=0.0168, audio_tagging_loss=0.01054, over 15505.00 frames. ], tot_loss[loss=0.07773, simple_loss=0.09862, pruned_loss=0.01855, audio_tagging_loss=0.009869, over 3053225.39 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:31:14,452 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.060e+01 8.301e+01 9.019e+01 9.806e+01 1.324e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-20 16:31:18,381 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.23 vs. limit=10.0
2023-11-20 16:31:18,499 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.43 vs. limit=15.0
2023-11-20 16:31:25,473 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170450
2023-11-20 16:31:52,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1136466.6666666667, ans=0.0
2023-11-20 16:32:03,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1136466.6666666667, ans=0.125
2023-11-20 16:32:07,314 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2150, loss[loss=0.0848, simple_loss=0.1052, pruned_loss=0.0208, audio_tagging_loss=0.01138, over 15640.00 frames. ], tot_loss[loss=0.0782, simple_loss=0.09909, pruned_loss=0.01875, audio_tagging_loss=0.009913, over 3044134.60 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:32:30,349 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170500
2023-11-20 16:32:45,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1136733.3333333333, ans=0.0
2023-11-20 16:32:46,618 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:33:12,830 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2200, loss[loss=0.08599, simple_loss=0.112, pruned_loss=0.02192, audio_tagging_loss=0.008083, over 14655.00 frames. ], tot_loss[loss=0.07887, simple_loss=0.1001, pruned_loss=0.01902, audio_tagging_loss=0.0098, over 3046203.22 frames. ], batch size: 58, lr: 4.65e-03, grad_scale: 16.0
2023-11-20 16:33:13,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1136866.6666666667, ans=0.125
2023-11-20 16:33:15,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1136866.6666666667, ans=0.125
2023-11-20 16:33:23,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1136866.6666666667, ans=0.125
2023-11-20 16:33:25,933 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.367e+01 8.465e+01 9.006e+01 9.663e+01 1.304e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-20 16:33:30,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1136933.3333333333, ans=0.2
2023-11-20 16:33:36,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170550
2023-11-20 16:33:39,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1137000.0, ans=0.0
2023-11-20 16:33:43,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1137000.0, ans=0.2
2023-11-20 16:33:49,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1137000.0, ans=0.0
2023-11-20 16:34:00,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=1137066.6666666667, ans=0.05
2023-11-20 16:34:02,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1137066.6666666667, ans=0.1
2023-11-20 16:34:04,813 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.88 vs. limit=22.5
2023-11-20 16:34:17,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1137200.0, ans=0.2
2023-11-20 16:34:18,614 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2250, loss[loss=0.09677, simple_loss=0.1247, pruned_loss=0.02651, audio_tagging_loss=0.007908, over 15513.00 frames. ], tot_loss[loss=0.08011, simple_loss=0.1017, pruned_loss=0.01944, audio_tagging_loss=0.009801, over 3048332.62 frames. ], batch size: 56, lr: 4.64e-03, grad_scale: 16.0
2023-11-20 16:34:23,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1137200.0, ans=0.0
2023-11-20 16:34:33,150 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.97 vs. limit=15.0
2023-11-20 16:34:43,195 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170600
2023-11-20 16:35:00,481 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0
2023-11-20 16:35:17,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1137466.6666666667, ans=0.0
2023-11-20 16:35:25,861 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2300, loss[loss=0.06174, simple_loss=0.07595, pruned_loss=0.0115, audio_tagging_loss=0.01227, over 16150.00 frames. ], tot_loss[loss=0.07992, simple_loss=0.1011, pruned_loss=0.0194, audio_tagging_loss=0.009967, over 3056137.45 frames. ], batch size: 61, lr: 4.64e-03, grad_scale: 16.0
2023-11-20 16:35:38,745 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.000e+01 8.643e+01 9.323e+01 1.269e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-20 16:35:48,908 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170650
2023-11-20 16:36:05,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1137733.3333333333, ans=0.2
2023-11-20 16:36:23,772 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:36:29,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1137800.0, ans=0.125
2023-11-20 16:36:29,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1137800.0, ans=0.125
2023-11-20 16:36:30,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1137866.6666666667, ans=0.5
2023-11-20 16:36:31,355 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2350, loss[loss=0.07418, simple_loss=0.09257, pruned_loss=0.01875, audio_tagging_loss=0.00914, over 14523.00 frames. ], tot_loss[loss=0.0791, simple_loss=0.09991, pruned_loss=0.01908, audio_tagging_loss=0.01006, over 3053186.37 frames. ], batch size: 55, lr: 4.64e-03, grad_scale: 16.0
2023-11-20 16:36:31,963 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.33 vs. limit=22.5
2023-11-20 16:36:40,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1137866.6666666667, ans=0.0
2023-11-20 16:36:52,642 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.91 vs. limit=15.0
2023-11-20 16:36:53,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1137933.3333333333, ans=0.2
2023-11-20 16:36:54,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170700
2023-11-20 16:37:14,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1138066.6666666667, ans=0.125
2023-11-20 16:37:16,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1138066.6666666667, ans=0.1
2023-11-20 16:37:18,647 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.93 vs. limit=10.0
2023-11-20 16:37:23,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1138133.3333333333, ans=0.1
2023-11-20 16:37:36,652 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2400, loss[loss=0.06259, simple_loss=0.07389, pruned_loss=0.01334, audio_tagging_loss=0.01231, over 14377.00 frames. ], tot_loss[loss=0.07944, simple_loss=0.1002, pruned_loss=0.01919, audio_tagging_loss=0.01016, over 3053059.55 frames. ], batch size: 53, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:37:43,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1138200.0, ans=0.2
2023-11-20 16:37:50,905 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.992e+01 8.259e+01 8.944e+01 9.702e+01 1.216e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-20 16:38:00,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1138266.6666666667, ans=0.95
2023-11-20 16:38:01,528 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170750
2023-11-20 16:38:02,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1138333.3333333333, ans=0.125
2023-11-20 16:38:19,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1138400.0, ans=0.0
2023-11-20 16:38:20,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1138400.0, ans=0.125
2023-11-20 16:38:37,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1138466.6666666667, ans=0.0
2023-11-20 16:38:42,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1138533.3333333333, ans=0.125
2023-11-20 16:38:43,365 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2450, loss[loss=0.08046, simple_loss=0.09934, pruned_loss=0.01932, audio_tagging_loss=0.01146, over 15085.00 frames. ], tot_loss[loss=0.07937, simple_loss=0.1003, pruned_loss=0.01912, audio_tagging_loss=0.0101, over 3054167.56 frames. ], batch size: 57, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:38:51,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1138533.3333333333, ans=0.95
2023-11-20 16:38:52,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1138533.3333333333, ans=0.0
2023-11-20 16:39:06,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170800
2023-11-20 16:39:16,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1138666.6666666667, ans=0.2
2023-11-20 16:39:29,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1138733.3333333333, ans=0.0
2023-11-20 16:39:37,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1138800.0, ans=0.125
2023-11-20 16:39:42,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1138800.0, ans=0.0
2023-11-20 16:39:48,730 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2500, loss[loss=0.07175, simple_loss=0.09579, pruned_loss=0.01639, audio_tagging_loss=0.007465, over 14661.00 frames. ], tot_loss[loss=0.07886, simple_loss=0.09948, pruned_loss=0.01897, audio_tagging_loss=0.01016, over 3053582.36 frames. ], batch size: 55, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:39:50,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1138866.6666666667, ans=0.125
2023-11-20 16:39:51,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1138866.6666666667, ans=0.125
2023-11-20 16:39:53,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1138866.6666666667, ans=0.125
2023-11-20 16:39:55,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.18 vs. limit=15.0
2023-11-20 16:40:01,219 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 8.113e+01 8.897e+01 9.641e+01 1.351e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-20 16:40:07,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1138933.3333333333, ans=0.0
2023-11-20 16:40:11,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170850
2023-11-20 16:40:27,418 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.33 vs. limit=22.5
2023-11-20 16:40:53,582 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2550, loss[loss=0.05989, simple_loss=0.07528, pruned_loss=0.0105, audio_tagging_loss=0.01175, over 14857.00 frames. ], tot_loss[loss=0.07757, simple_loss=0.09781, pruned_loss=0.01857, audio_tagging_loss=0.01009, over 3049744.61 frames. ], batch size: 58, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:41:10,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1139266.6666666667, ans=0.125
2023-11-20 16:41:17,531 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170900
2023-11-20 16:41:31,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1139400.0, ans=0.125
2023-11-20 16:41:34,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1139400.0, ans=0.2
2023-11-20 16:41:51,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1139466.6666666667, ans=0.1
2023-11-20 16:41:53,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1139466.6666666667, ans=0.125
2023-11-20 16:41:54,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1139466.6666666667, ans=0.125
2023-11-20 16:41:58,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1139533.3333333333, ans=0.125
2023-11-20 16:41:59,386 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2600, loss[loss=0.06377, simple_loss=0.07439, pruned_loss=0.01576, audio_tagging_loss=0.01081, over 14531.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09819, pruned_loss=0.01872, audio_tagging_loss=0.009941, over 3042502.80 frames. ], batch size: 59, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:42:05,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1139533.3333333333, ans=0.125
2023-11-20 16:42:10,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1139533.3333333333, ans=0.125
2023-11-20 16:42:13,173 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.583e+01 8.062e+01 8.877e+01 9.575e+01 1.292e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-20 16:42:13,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1139600.0, ans=0.2
2023-11-20 16:42:19,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1139600.0, ans=0.125
2023-11-20 16:42:20,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1139600.0, ans=0.0
2023-11-20 16:42:23,185 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 170950
2023-11-20 16:42:27,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1139666.6666666667, ans=0.0
2023-11-20 16:42:47,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1139733.3333333333, ans=0.2
2023-11-20 16:43:01,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1139800.0, ans=0.0
2023-11-20 16:43:05,759 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2650, loss[loss=0.05949, simple_loss=0.07561, pruned_loss=0.01312, audio_tagging_loss=0.008565, over 15445.00 frames. ], tot_loss[loss=0.07749, simple_loss=0.09775, pruned_loss=0.01875, audio_tagging_loss=0.009874, over 3047386.29 frames. ], batch size: 57, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:43:28,360 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171000
2023-11-20 16:43:30,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1140000.0, ans=0.09899494936611666
2023-11-20 16:43:48,150 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.46 vs. limit=12.0
2023-11-20 16:44:01,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.51 vs. limit=15.0
2023-11-20 16:44:03,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1140133.3333333333, ans=0.125
2023-11-20 16:44:03,420 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0
2023-11-20 16:44:08,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1140133.3333333333, ans=0.0
2023-11-20 16:44:11,717 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2700, loss[loss=0.09283, simple_loss=0.1238, pruned_loss=0.02333, audio_tagging_loss=0.007616, over 15967.00 frames. ], tot_loss[loss=0.07706, simple_loss=0.09714, pruned_loss=0.01861, audio_tagging_loss=0.009883, over 3048216.53 frames. ], batch size: 56, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:44:24,012 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.274e+01 8.361e+01 8.930e+01 9.610e+01 1.277e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-20 16:44:26,020 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.05 vs. limit=15.0
2023-11-20 16:44:34,473 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.55 vs. limit=15.0
2023-11-20 16:44:35,166 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171050
2023-11-20 16:44:36,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1140266.6666666667, ans=0.125
2023-11-20 16:44:42,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1140333.3333333333, ans=0.2
2023-11-20 16:44:53,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1140400.0, ans=10.0
2023-11-20 16:44:57,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1140400.0, ans=0.0
2023-11-20 16:45:11,333 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.65 vs. limit=22.5
2023-11-20 16:45:15,530 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2750, loss[loss=0.09708, simple_loss=0.1105, pruned_loss=0.03362, audio_tagging_loss=0.008182, over 15016.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09718, pruned_loss=0.01874, audio_tagging_loss=0.009806, over 3056314.79 frames. ], batch size: 56, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:45:20,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1140533.3333333333, ans=0.0
2023-11-20 16:45:31,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1140600.0, ans=0.125
2023-11-20 16:45:40,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171100
2023-11-20 16:45:47,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.38 vs. limit=15.0
2023-11-20 16:46:03,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.37 vs. limit=6.0
2023-11-20 16:46:12,573 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:46:21,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1140866.6666666667, ans=0.125
2023-11-20 16:46:22,422 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2800, loss[loss=0.07996, simple_loss=0.09838, pruned_loss=0.02173, audio_tagging_loss=0.009042, over 15505.00 frames. ], tot_loss[loss=0.07787, simple_loss=0.09836, pruned_loss=0.01892, audio_tagging_loss=0.009765, over 3048080.66 frames. ], batch size: 57, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:46:24,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.13 vs. limit=22.5
2023-11-20 16:46:34,710 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.215e+01 7.973e+01 8.545e+01 9.454e+01 1.250e+02, threshold=1.709e+02, percent-clipped=0.0
2023-11-20 16:46:44,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171150
2023-11-20 16:46:46,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1141000.0, ans=0.1
2023-11-20 16:46:57,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.87 vs. limit=6.0
2023-11-20 16:46:58,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1141000.0, ans=0.0
2023-11-20 16:46:58,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1141000.0, ans=0.125
2023-11-20 16:47:13,808 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.88 vs. limit=15.0
2023-11-20 16:47:17,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1141133.3333333333, ans=0.125
2023-11-20 16:47:19,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1141133.3333333333, ans=0.125
2023-11-20 16:47:26,754 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2850, loss[loss=0.06065, simple_loss=0.0709, pruned_loss=0.01558, audio_tagging_loss=0.009621, over 14331.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09805, pruned_loss=0.01884, audio_tagging_loss=0.009727, over 3044209.30 frames. ], batch size: 57, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:47:35,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.47 vs. limit=15.0
2023-11-20 16:47:37,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.53 vs. limit=6.0
2023-11-20 16:47:50,318 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171200
2023-11-20 16:48:27,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1141466.6666666667, ans=0.1
2023-11-20 16:48:31,826 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2900, loss[loss=0.07807, simple_loss=0.1062, pruned_loss=0.01595, audio_tagging_loss=0.009032, over 15381.00 frames. ], tot_loss[loss=0.07845, simple_loss=0.09938, pruned_loss=0.01902, audio_tagging_loss=0.009737, over 3045154.21 frames. ], batch size: 55, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:48:34,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1141533.3333333333, ans=0.0
2023-11-20 16:48:45,417 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.270e+01 7.954e+01 8.776e+01 9.382e+01 1.409e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-20 16:48:51,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1141600.0, ans=0.0
2023-11-20 16:48:56,669 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171250
2023-11-20 16:48:58,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1141666.6666666667, ans=0.0
2023-11-20 16:49:37,943 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 2950, loss[loss=0.06398, simple_loss=0.08004, pruned_loss=0.01401, audio_tagging_loss=0.009949, over 15629.00 frames. ], tot_loss[loss=0.07894, simple_loss=0.1002, pruned_loss=0.01921, audio_tagging_loss=0.009649, over 3043040.72 frames. ], batch size: 60, lr: 4.64e-03, grad_scale: 32.0
2023-11-20 16:49:38,464 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.11 vs. limit=12.0
2023-11-20 16:50:01,357 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171300
2023-11-20 16:50:13,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1142000.0, ans=0.04949747468305833
2023-11-20 16:50:14,335 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.15 vs. limit=15.0
2023-11-20 16:50:37,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1142133.3333333333, ans=0.125
2023-11-20 16:50:43,609 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3000, loss[loss=0.06688, simple_loss=0.09004, pruned_loss=0.01303, audio_tagging_loss=0.008831, over 15297.00 frames. ], tot_loss[loss=0.07889, simple_loss=0.09985, pruned_loss=0.01914, audio_tagging_loss=0.009825, over 3043632.09 frames. ], batch size: 57, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:50:43,612 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-20 16:51:00,253 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3940, 5.0919, 4.7692, 4.7185], device='cuda:0')
2023-11-20 16:51:23,428 INFO [train_asr.py:1253] (0/4) Epoch 15, validation: loss=0.06163, simple_loss=0.05329, pruned_loss=0.005569, audio_tagging_loss=0.02942, over 4681554.00 frames. 
2023-11-20 16:51:23,429 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-20 16:51:27,358 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:51:36,341 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.037e+01 8.379e+01 9.112e+01 1.041e+02 3.133e+02, threshold=1.822e+02, percent-clipped=1.0
2023-11-20 16:51:44,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1142266.6666666667, ans=0.125
2023-11-20 16:51:47,254 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171350
2023-11-20 16:52:13,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1142400.0, ans=0.125
2023-11-20 16:52:18,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1142466.6666666667, ans=0.2
2023-11-20 16:52:28,807 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3050, loss[loss=0.07953, simple_loss=0.1044, pruned_loss=0.0211, audio_tagging_loss=0.006237, over 14624.00 frames. ], tot_loss[loss=0.07907, simple_loss=0.0999, pruned_loss=0.01928, audio_tagging_loss=0.009843, over 3040865.62 frames. ], batch size: 55, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:52:30,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1142533.3333333333, ans=0.0
2023-11-20 16:52:51,565 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171400
2023-11-20 16:52:54,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1142666.6666666667, ans=0.2
2023-11-20 16:52:54,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1142666.6666666667, ans=0.125
2023-11-20 16:52:57,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1142666.6666666667, ans=0.125
2023-11-20 16:52:58,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1142666.6666666667, ans=0.125
2023-11-20 16:53:07,714 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 16:53:17,904 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.78 vs. limit=12.0
2023-11-20 16:53:23,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1142800.0, ans=0.95
2023-11-20 16:53:24,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1142800.0, ans=0.2
2023-11-20 16:53:28,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1142800.0, ans=0.125
2023-11-20 16:53:34,121 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3100, loss[loss=0.06945, simple_loss=0.07985, pruned_loss=0.01811, audio_tagging_loss=0.01141, over 16758.00 frames. ], tot_loss[loss=0.07928, simple_loss=0.1004, pruned_loss=0.01923, audio_tagging_loss=0.009863, over 3040884.80 frames. ], batch size: 63, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:53:46,516 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.278e+01 8.696e+01 9.346e+01 1.301e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-20 16:53:46,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1142933.3333333333, ans=0.125
2023-11-20 16:53:57,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.23 vs. limit=6.0
2023-11-20 16:53:57,645 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171450
2023-11-20 16:54:00,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1143000.0, ans=0.1
2023-11-20 16:54:23,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1143066.6666666667, ans=0.0
2023-11-20 16:54:32,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1143133.3333333333, ans=0.125
2023-11-20 16:54:39,677 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3150, loss[loss=0.09245, simple_loss=0.1244, pruned_loss=0.02129, audio_tagging_loss=0.008961, over 15359.00 frames. ], tot_loss[loss=0.08043, simple_loss=0.1018, pruned_loss=0.01969, audio_tagging_loss=0.009837, over 3051381.74 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:54:41,138 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 16:54:47,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1143200.0, ans=0.125
2023-11-20 16:54:48,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1143200.0, ans=0.125
2023-11-20 16:54:49,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1143200.0, ans=0.1
2023-11-20 16:55:03,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171500
2023-11-20 16:55:16,694 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.77 vs. limit=15.0
2023-11-20 16:55:18,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1143400.0, ans=0.125
2023-11-20 16:55:34,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1143466.6666666667, ans=0.125
2023-11-20 16:55:45,990 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3200, loss[loss=0.06449, simple_loss=0.07764, pruned_loss=0.01548, audio_tagging_loss=0.01019, over 14017.00 frames. ], tot_loss[loss=0.07991, simple_loss=0.1011, pruned_loss=0.01938, audio_tagging_loss=0.009972, over 3049458.00 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:55:46,581 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.66 vs. limit=15.0
2023-11-20 16:55:58,484 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.865e+01 8.146e+01 8.709e+01 9.815e+01 1.794e+02, threshold=1.742e+02, percent-clipped=1.0
2023-11-20 16:56:09,273 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171550
2023-11-20 16:56:16,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1143666.6666666667, ans=0.0
2023-11-20 16:56:36,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1143800.0, ans=0.125
2023-11-20 16:56:39,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1143800.0, ans=0.0
2023-11-20 16:56:44,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1143800.0, ans=0.125
2023-11-20 16:56:50,357 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3250, loss[loss=0.09696, simple_loss=0.1222, pruned_loss=0.02638, audio_tagging_loss=0.009498, over 16179.00 frames. ], tot_loss[loss=0.07961, simple_loss=0.1005, pruned_loss=0.01923, audio_tagging_loss=0.01015, over 3048666.61 frames. ], batch size: 61, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:57:04,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1143933.3333333333, ans=0.0
2023-11-20 16:57:05,133 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.95 vs. limit=15.0
2023-11-20 16:57:14,014 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171600
2023-11-20 16:57:21,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1144000.0, ans=0.125
2023-11-20 16:57:23,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1144000.0, ans=0.125
2023-11-20 16:57:37,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1144066.6666666667, ans=0.0
2023-11-20 16:57:55,238 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3300, loss[loss=0.07023, simple_loss=0.09487, pruned_loss=0.01327, audio_tagging_loss=0.009523, over 15053.00 frames. ], tot_loss[loss=0.07879, simple_loss=0.09914, pruned_loss=0.01886, audio_tagging_loss=0.01036, over 3041039.38 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:57:58,604 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.59 vs. limit=15.0
2023-11-20 16:58:05,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1144200.0, ans=0.0
2023-11-20 16:58:09,266 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.004e+01 8.604e+01 9.340e+01 1.280e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-20 16:58:09,972 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.33 vs. limit=15.0
2023-11-20 16:58:20,055 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171650
2023-11-20 16:58:27,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1144333.3333333333, ans=0.125
2023-11-20 16:58:41,111 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.95 vs. limit=12.0
2023-11-20 16:58:47,314 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.72 vs. limit=12.0
2023-11-20 16:59:02,016 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3350, loss[loss=0.09402, simple_loss=0.1073, pruned_loss=0.03108, audio_tagging_loss=0.009284, over 14758.00 frames. ], tot_loss[loss=0.07875, simple_loss=0.09938, pruned_loss=0.01883, audio_tagging_loss=0.01023, over 3043443.48 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 16:59:02,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1144533.3333333333, ans=0.125
2023-11-20 16:59:24,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171700
2023-11-20 16:59:31,068 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.04 vs. limit=22.5
2023-11-20 16:59:34,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1144666.6666666667, ans=0.5
2023-11-20 16:59:43,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1144733.3333333333, ans=0.125
2023-11-20 17:00:06,498 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3400, loss[loss=0.09411, simple_loss=0.1262, pruned_loss=0.0224, audio_tagging_loss=0.008589, over 15493.00 frames. ], tot_loss[loss=0.07837, simple_loss=0.09917, pruned_loss=0.01872, audio_tagging_loss=0.01006, over 3042491.78 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 16.0
2023-11-20 17:00:07,055 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.79 vs. limit=12.0
2023-11-20 17:00:20,869 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.042e+01 8.596e+01 9.172e+01 1.141e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-20 17:00:21,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1144933.3333333333, ans=0.0
2023-11-20 17:00:27,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1144933.3333333333, ans=0.0
2023-11-20 17:00:30,301 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171750
2023-11-20 17:00:42,878 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:00:53,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1145066.6666666667, ans=0.5
2023-11-20 17:00:57,261 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.77 vs. limit=6.0
2023-11-20 17:01:12,078 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3450, loss[loss=0.08159, simple_loss=0.1054, pruned_loss=0.01999, audio_tagging_loss=0.008881, over 15673.00 frames. ], tot_loss[loss=0.07863, simple_loss=0.09992, pruned_loss=0.01883, audio_tagging_loss=0.009846, over 3045372.74 frames. ], batch size: 59, lr: 4.63e-03, grad_scale: 16.0
2023-11-20 17:01:27,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1145266.6666666667, ans=0.0
2023-11-20 17:01:36,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171800
2023-11-20 17:01:44,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1145333.3333333333, ans=0.2
2023-11-20 17:01:49,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1145333.3333333333, ans=0.1
2023-11-20 17:02:02,911 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.739e-01
2023-11-20 17:02:06,925 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.41 vs. limit=12.0
2023-11-20 17:02:09,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1145466.6666666667, ans=0.0
2023-11-20 17:02:18,791 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3500, loss[loss=0.07189, simple_loss=0.08821, pruned_loss=0.01576, audio_tagging_loss=0.01203, over 15286.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.09979, pruned_loss=0.01894, audio_tagging_loss=0.009785, over 3048198.46 frames. ], batch size: 58, lr: 4.63e-03, grad_scale: 16.0
2023-11-20 17:02:22,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1145533.3333333333, ans=0.0
2023-11-20 17:02:33,126 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.582e+01 8.137e+01 8.586e+01 9.265e+01 1.254e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-20 17:02:39,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1145600.0, ans=0.1
2023-11-20 17:02:41,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171850
2023-11-20 17:02:42,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1145600.0, ans=0.125
2023-11-20 17:02:51,760 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:02:56,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1145733.3333333333, ans=0.1
2023-11-20 17:02:56,704 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.89 vs. limit=6.0
2023-11-20 17:03:03,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1145733.3333333333, ans=0.95
2023-11-20 17:03:08,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1145733.3333333333, ans=0.07
2023-11-20 17:03:08,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1145733.3333333333, ans=0.0
2023-11-20 17:03:17,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1145800.0, ans=0.1
2023-11-20 17:03:24,256 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3550, loss[loss=0.07595, simple_loss=0.1032, pruned_loss=0.01406, audio_tagging_loss=0.01031, over 14086.00 frames. ], tot_loss[loss=0.07764, simple_loss=0.09837, pruned_loss=0.01863, audio_tagging_loss=0.009828, over 3043230.89 frames. ], batch size: 54, lr: 4.63e-03, grad_scale: 16.0
2023-11-20 17:03:47,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171900
2023-11-20 17:03:55,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=1146000.0, ans=15.0
2023-11-20 17:04:15,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1146133.3333333333, ans=0.125
2023-11-20 17:04:29,143 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3600, loss[loss=0.07283, simple_loss=0.09053, pruned_loss=0.01627, audio_tagging_loss=0.01129, over 15163.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09843, pruned_loss=0.01871, audio_tagging_loss=0.009798, over 3040683.37 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 17:04:37,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1146200.0, ans=0.0
2023-11-20 17:04:44,614 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.710e+01 8.052e+01 8.613e+01 9.290e+01 1.173e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-20 17:04:50,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.10 vs. limit=15.0
2023-11-20 17:04:53,929 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 171950
2023-11-20 17:05:19,213 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=15.0
2023-11-20 17:05:20,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1146466.6666666667, ans=0.0
2023-11-20 17:05:34,600 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3650, loss[loss=0.0657, simple_loss=0.08737, pruned_loss=0.01454, audio_tagging_loss=0.007472, over 15185.00 frames. ], tot_loss[loss=0.07781, simple_loss=0.09875, pruned_loss=0.01876, audio_tagging_loss=0.009677, over 3041483.44 frames. ], batch size: 56, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 17:05:46,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1146600.0, ans=0.0
2023-11-20 17:05:50,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1146600.0, ans=0.0
2023-11-20 17:05:57,537 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172000
2023-11-20 17:05:59,281 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-172000.pt
2023-11-20 17:06:06,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1146666.6666666667, ans=0.1
2023-11-20 17:06:22,190 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.32 vs. limit=6.0
2023-11-20 17:06:42,353 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3700, loss[loss=0.1083, simple_loss=0.1396, pruned_loss=0.03149, audio_tagging_loss=0.007026, over 15455.00 frames. ], tot_loss[loss=0.07846, simple_loss=0.0995, pruned_loss=0.01898, audio_tagging_loss=0.00972, over 3042527.86 frames. ], batch size: 55, lr: 4.63e-03, grad_scale: 32.0
2023-11-20 17:06:52,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1146866.6666666667, ans=0.125
2023-11-20 17:06:56,084 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.121e+01 8.690e+01 9.493e+01 1.302e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 17:06:59,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1146933.3333333333, ans=0.1
2023-11-20 17:07:04,491 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.44 vs. limit=15.0
2023-11-20 17:07:05,261 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172050
2023-11-20 17:07:05,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1146933.3333333333, ans=0.1
2023-11-20 17:07:07,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1147000.0, ans=0.1
2023-11-20 17:07:08,454 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.27 vs. limit=12.0
2023-11-20 17:07:16,924 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.34 vs. limit=22.5
2023-11-20 17:07:28,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1147066.6666666667, ans=0.125
2023-11-20 17:07:46,863 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3750, loss[loss=0.08037, simple_loss=0.09692, pruned_loss=0.02401, audio_tagging_loss=0.007895, over 14815.00 frames. ], tot_loss[loss=0.07872, simple_loss=0.09988, pruned_loss=0.01904, audio_tagging_loss=0.009742, over 3037984.96 frames. ], batch size: 55, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:08:00,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1147266.6666666667, ans=0.125
2023-11-20 17:08:00,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1147266.6666666667, ans=0.125
2023-11-20 17:08:04,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1147266.6666666667, ans=0.125
2023-11-20 17:08:06,255 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.39 vs. limit=22.5
2023-11-20 17:08:11,573 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172100
2023-11-20 17:08:32,674 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:08:35,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1147400.0, ans=0.125
2023-11-20 17:08:39,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1147466.6666666667, ans=0.2
2023-11-20 17:08:41,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1147466.6666666667, ans=0.125
2023-11-20 17:08:52,194 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3800, loss[loss=0.08086, simple_loss=0.1091, pruned_loss=0.01697, audio_tagging_loss=0.009344, over 15637.00 frames. ], tot_loss[loss=0.07855, simple_loss=0.09951, pruned_loss=0.01893, audio_tagging_loss=0.009864, over 3045784.95 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:08:54,890 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.60 vs. limit=15.0
2023-11-20 17:09:08,178 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.643e+01 8.118e+01 8.690e+01 9.561e+01 1.262e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 17:09:15,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172150
2023-11-20 17:09:26,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1147666.6666666667, ans=0.0
2023-11-20 17:09:27,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1147666.6666666667, ans=0.0
2023-11-20 17:09:56,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1147800.0, ans=0.0
2023-11-20 17:09:58,570 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3850, loss[loss=0.07742, simple_loss=0.09345, pruned_loss=0.01568, audio_tagging_loss=0.01501, over 15911.00 frames. ], tot_loss[loss=0.07871, simple_loss=0.09971, pruned_loss=0.01886, audio_tagging_loss=0.00999, over 3040831.36 frames. ], batch size: 59, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:10:08,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1147866.6666666667, ans=0.2
2023-11-20 17:10:13,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1147933.3333333333, ans=0.125
2023-11-20 17:10:21,396 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172200
2023-11-20 17:10:26,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1148000.0, ans=0.0
2023-11-20 17:10:34,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1148000.0, ans=0.0
2023-11-20 17:10:36,702 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.40 vs. limit=15.0
2023-11-20 17:11:04,036 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3900, loss[loss=0.07521, simple_loss=0.09357, pruned_loss=0.01777, audio_tagging_loss=0.01065, over 14404.00 frames. ], tot_loss[loss=0.07786, simple_loss=0.09829, pruned_loss=0.01857, audio_tagging_loss=0.01015, over 3041100.07 frames. ], batch size: 56, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:11:19,571 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.094e+01 8.942e+01 9.624e+01 1.235e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-20 17:11:22,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1148266.6666666667, ans=0.2
2023-11-20 17:11:28,393 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172250
2023-11-20 17:11:31,117 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.98 vs. limit=15.0
2023-11-20 17:11:34,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1148333.3333333333, ans=0.015
2023-11-20 17:11:47,364 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.66 vs. limit=22.5
2023-11-20 17:11:55,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1148466.6666666667, ans=0.125
2023-11-20 17:11:55,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1148466.6666666667, ans=0.2
2023-11-20 17:12:08,870 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 3950, loss[loss=0.1005, simple_loss=0.1458, pruned_loss=0.01893, audio_tagging_loss=0.008652, over 15050.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09807, pruned_loss=0.01858, audio_tagging_loss=0.01015, over 3037112.46 frames. ], batch size: 54, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:12:32,718 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172300
2023-11-20 17:12:32,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1148600.0, ans=0.1
2023-11-20 17:12:39,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1148666.6666666667, ans=0.1
2023-11-20 17:12:43,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1148666.6666666667, ans=0.04949747468305833
2023-11-20 17:12:48,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1148733.3333333333, ans=0.125
2023-11-20 17:12:56,619 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:13:04,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.64 vs. limit=15.0
2023-11-20 17:13:05,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1148800.0, ans=0.125
2023-11-20 17:13:12,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1148800.0, ans=0.125
2023-11-20 17:13:14,347 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4000, loss[loss=0.06988, simple_loss=0.08109, pruned_loss=0.01781, audio_tagging_loss=0.01152, over 14028.00 frames. ], tot_loss[loss=0.07854, simple_loss=0.09891, pruned_loss=0.01888, audio_tagging_loss=0.01021, over 3036405.30 frames. ], batch size: 53, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:13:14,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1148866.6666666667, ans=0.125
2023-11-20 17:13:14,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1148866.6666666667, ans=0.1
2023-11-20 17:13:20,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1148866.6666666667, ans=0.1
2023-11-20 17:13:29,458 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.881e+01 7.998e+01 8.708e+01 9.550e+01 1.131e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-20 17:13:37,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172350
2023-11-20 17:13:55,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1149066.6666666667, ans=0.0
2023-11-20 17:13:56,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1149066.6666666667, ans=0.0
2023-11-20 17:14:04,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1149066.6666666667, ans=0.125
2023-11-20 17:14:16,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1149133.3333333333, ans=0.0
2023-11-20 17:14:18,664 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4050, loss[loss=0.08194, simple_loss=0.1023, pruned_loss=0.02051, audio_tagging_loss=0.01027, over 15561.00 frames. ], tot_loss[loss=0.07891, simple_loss=0.09959, pruned_loss=0.01889, audio_tagging_loss=0.01023, over 3033592.54 frames. ], batch size: 58, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:14:18,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1149200.0, ans=0.125
2023-11-20 17:14:22,411 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:14:40,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1149266.6666666667, ans=0.0
2023-11-20 17:14:42,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172400
2023-11-20 17:14:44,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1149333.3333333333, ans=0.125
2023-11-20 17:14:56,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1149333.3333333333, ans=0.0
2023-11-20 17:15:24,190 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4100, loss[loss=0.05895, simple_loss=0.06939, pruned_loss=0.01381, audio_tagging_loss=0.01045, over 14525.00 frames. ], tot_loss[loss=0.07883, simple_loss=0.09949, pruned_loss=0.01896, audio_tagging_loss=0.01013, over 3042278.31 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:15:25,976 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.93 vs. limit=15.0
2023-11-20 17:15:27,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1149533.3333333333, ans=0.0
2023-11-20 17:15:38,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1149600.0, ans=0.125
2023-11-20 17:15:42,058 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.166e+01 8.151e+01 8.717e+01 9.713e+01 1.333e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-20 17:15:44,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1149600.0, ans=0.0
2023-11-20 17:15:48,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172450
2023-11-20 17:16:01,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1149666.6666666667, ans=0.125
2023-11-20 17:16:10,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1149733.3333333333, ans=0.1
2023-11-20 17:16:11,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1149733.3333333333, ans=0.0
2023-11-20 17:16:30,793 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4150, loss[loss=0.09855, simple_loss=0.1278, pruned_loss=0.02626, audio_tagging_loss=0.008384, over 16020.00 frames. ], tot_loss[loss=0.07898, simple_loss=0.1, pruned_loss=0.01899, audio_tagging_loss=0.009985, over 3045428.79 frames. ], batch size: 55, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:16:51,565 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=2.504e-03
2023-11-20 17:16:53,814 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172500
2023-11-20 17:16:55,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.whiten.whitening_limit, batch_count=1150000.0, ans=12.0
2023-11-20 17:17:04,375 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.45 vs. limit=15.0
2023-11-20 17:17:16,589 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:17:18,188 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:17:22,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1150133.3333333333, ans=0.2
2023-11-20 17:17:36,223 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4200, loss[loss=0.06626, simple_loss=0.08678, pruned_loss=0.01459, audio_tagging_loss=0.008277, over 14487.00 frames. ], tot_loss[loss=0.07827, simple_loss=0.09933, pruned_loss=0.01873, audio_tagging_loss=0.009876, over 3047204.63 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:17:52,269 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.720e+01 8.093e+01 8.811e+01 9.474e+01 1.183e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-20 17:17:56,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1150266.6666666667, ans=0.0
2023-11-20 17:17:59,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172550
2023-11-20 17:18:02,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1150333.3333333333, ans=0.125
2023-11-20 17:18:08,143 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:18:10,940 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.72 vs. limit=12.0
2023-11-20 17:18:25,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.93 vs. limit=22.5
2023-11-20 17:18:29,433 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.92 vs. limit=22.5
2023-11-20 17:18:40,996 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4250, loss[loss=0.09552, simple_loss=0.1248, pruned_loss=0.02651, audio_tagging_loss=0.006622, over 16209.00 frames. ], tot_loss[loss=0.07855, simple_loss=0.09974, pruned_loss=0.01887, audio_tagging_loss=0.009808, over 3044696.62 frames. ], batch size: 56, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:18:50,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=4.06 vs. limit=15.0
2023-11-20 17:18:57,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1150600.0, ans=0.125
2023-11-20 17:19:02,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1150600.0, ans=0.125
2023-11-20 17:19:05,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172600
2023-11-20 17:19:05,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1150600.0, ans=0.125
2023-11-20 17:19:13,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1150666.6666666667, ans=0.125
2023-11-20 17:19:17,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1150666.6666666667, ans=0.125
2023-11-20 17:19:30,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1150733.3333333333, ans=0.1
2023-11-20 17:19:47,504 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4300, loss[loss=0.08096, simple_loss=0.09959, pruned_loss=0.01855, audio_tagging_loss=0.01262, over 14986.00 frames. ], tot_loss[loss=0.07926, simple_loss=0.1008, pruned_loss=0.01913, audio_tagging_loss=0.00974, over 3043665.13 frames. ], batch size: 55, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:19:53,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=10.45 vs. limit=15.0
2023-11-20 17:20:01,248 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.54 vs. limit=15.0
2023-11-20 17:20:04,271 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.964e+01 7.979e+01 8.573e+01 9.478e+01 1.236e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-20 17:20:10,718 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172650
2023-11-20 17:20:38,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1151133.3333333333, ans=0.0
2023-11-20 17:20:47,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1151133.3333333333, ans=0.1
2023-11-20 17:20:52,676 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4350, loss[loss=0.08423, simple_loss=0.1116, pruned_loss=0.02082, audio_tagging_loss=0.007614, over 15680.00 frames. ], tot_loss[loss=0.07894, simple_loss=0.1005, pruned_loss=0.019, audio_tagging_loss=0.009694, over 3041064.44 frames. ], batch size: 57, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:21:00,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1151200.0, ans=0.125
2023-11-20 17:21:05,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1151266.6666666667, ans=0.2
2023-11-20 17:21:05,682 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.69 vs. limit=15.0
2023-11-20 17:21:15,710 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172700
2023-11-20 17:21:42,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1151400.0, ans=0.125
2023-11-20 17:21:57,242 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4400, loss[loss=0.05214, simple_loss=0.06322, pruned_loss=0.01026, audio_tagging_loss=0.01026, over 15865.00 frames. ], tot_loss[loss=0.07868, simple_loss=0.09999, pruned_loss=0.01896, audio_tagging_loss=0.00972, over 3038990.03 frames. ], batch size: 60, lr: 4.62e-03, grad_scale: 32.0
2023-11-20 17:22:16,313 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.328e+01 8.168e+01 8.779e+01 9.674e+01 1.363e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-20 17:22:20,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1151600.0, ans=0.025
2023-11-20 17:22:21,355 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172750
2023-11-20 17:22:58,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1151800.0, ans=0.125
2023-11-20 17:22:59,731 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.96 vs. limit=15.0
2023-11-20 17:23:02,746 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4450, loss[loss=0.06171, simple_loss=0.08199, pruned_loss=0.01257, audio_tagging_loss=0.008146, over 13520.00 frames. ], tot_loss[loss=0.07877, simple_loss=0.1003, pruned_loss=0.01899, audio_tagging_loss=0.009632, over 3033836.53 frames. ], batch size: 52, lr: 4.62e-03, grad_scale: 16.0
2023-11-20 17:23:03,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1151866.6666666667, ans=0.0
2023-11-20 17:23:16,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1151933.3333333333, ans=0.1
2023-11-20 17:23:26,476 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172800
2023-11-20 17:23:32,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.70 vs. limit=15.0
2023-11-20 17:24:08,398 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4500, loss[loss=0.06478, simple_loss=0.07543, pruned_loss=0.01446, audio_tagging_loss=0.0126, over 15268.00 frames. ], tot_loss[loss=0.07882, simple_loss=0.1005, pruned_loss=0.01895, audio_tagging_loss=0.009596, over 3038617.55 frames. ], batch size: 60, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:24:11,498 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.80 vs. limit=15.0
2023-11-20 17:24:20,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1152266.6666666667, ans=0.0
2023-11-20 17:24:24,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1152266.6666666667, ans=0.125
2023-11-20 17:24:26,180 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.191e+01 8.215e+01 8.732e+01 9.444e+01 1.886e+02, threshold=1.746e+02, percent-clipped=1.0
2023-11-20 17:24:26,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1152266.6666666667, ans=0.1
2023-11-20 17:24:29,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1152266.6666666667, ans=0.0
2023-11-20 17:24:31,390 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172850
2023-11-20 17:24:42,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1152333.3333333333, ans=0.0
2023-11-20 17:25:13,139 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4550, loss[loss=0.04906, simple_loss=0.05913, pruned_loss=0.006804, audio_tagging_loss=0.01269, over 16290.00 frames. ], tot_loss[loss=0.07883, simple_loss=0.1006, pruned_loss=0.01886, audio_tagging_loss=0.009678, over 3042674.76 frames. ], batch size: 63, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:25:20,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1152533.3333333333, ans=0.125
2023-11-20 17:25:26,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1152600.0, ans=0.0
2023-11-20 17:25:36,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1152600.0, ans=0.125
2023-11-20 17:25:37,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172900
2023-11-20 17:26:02,828 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:26:16,049 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.58 vs. limit=15.0
2023-11-20 17:26:19,006 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4600, loss[loss=0.07203, simple_loss=0.09407, pruned_loss=0.01372, audio_tagging_loss=0.01127, over 14786.00 frames. ], tot_loss[loss=0.07834, simple_loss=0.0998, pruned_loss=0.01865, audio_tagging_loss=0.009787, over 3046730.17 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:26:19,632 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.30 vs. limit=15.0
2023-11-20 17:26:26,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1152866.6666666667, ans=0.125
2023-11-20 17:26:36,935 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 8.346e+01 9.151e+01 1.010e+02 1.307e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-20 17:26:39,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1152933.3333333333, ans=0.2
2023-11-20 17:26:42,034 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 172950
2023-11-20 17:26:56,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1153066.6666666667, ans=0.125
2023-11-20 17:26:58,286 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.33 vs. limit=12.0
2023-11-20 17:27:10,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1153133.3333333333, ans=0.0
2023-11-20 17:27:24,115 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4650, loss[loss=0.08421, simple_loss=0.1006, pruned_loss=0.02286, audio_tagging_loss=0.01104, over 14581.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.09973, pruned_loss=0.0188, audio_tagging_loss=0.009801, over 3047730.61 frames. ], batch size: 57, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:27:37,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1153266.6666666667, ans=0.125
2023-11-20 17:27:43,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1153266.6666666667, ans=0.0
2023-11-20 17:27:46,893 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173000
2023-11-20 17:27:46,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1153266.6666666667, ans=0.125
2023-11-20 17:27:47,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1153266.6666666667, ans=0.0
2023-11-20 17:28:08,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1153400.0, ans=0.0
2023-11-20 17:28:09,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.20 vs. limit=15.0
2023-11-20 17:28:29,540 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4700, loss[loss=0.06416, simple_loss=0.08075, pruned_loss=0.014, audio_tagging_loss=0.009791, over 16506.00 frames. ], tot_loss[loss=0.07814, simple_loss=0.09924, pruned_loss=0.01858, audio_tagging_loss=0.009942, over 3045941.48 frames. ], batch size: 63, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:28:49,104 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 7.947e+01 8.690e+01 9.286e+01 1.416e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 17:28:54,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173050
2023-11-20 17:28:56,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1153666.6666666667, ans=0.07
2023-11-20 17:28:59,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.54 vs. limit=15.0
2023-11-20 17:29:06,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1153666.6666666667, ans=0.1
2023-11-20 17:29:15,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1153733.3333333333, ans=0.125
2023-11-20 17:29:33,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1153800.0, ans=0.2
2023-11-20 17:29:36,594 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4750, loss[loss=0.07655, simple_loss=0.09848, pruned_loss=0.01708, audio_tagging_loss=0.01023, over 15645.00 frames. ], tot_loss[loss=0.07819, simple_loss=0.09914, pruned_loss=0.01862, audio_tagging_loss=0.009996, over 3038477.27 frames. ], batch size: 59, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:29:59,563 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173100
2023-11-20 17:30:14,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1154066.6666666667, ans=0.125
2023-11-20 17:30:42,298 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4800, loss[loss=0.06283, simple_loss=0.07073, pruned_loss=0.01479, audio_tagging_loss=0.01267, over 13708.00 frames. ], tot_loss[loss=0.0787, simple_loss=0.09957, pruned_loss=0.01882, audio_tagging_loss=0.0101, over 3040041.50 frames. ], batch size: 55, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:30:47,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1154200.0, ans=0.125
2023-11-20 17:30:49,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.18 vs. limit=15.0
2023-11-20 17:30:49,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1154200.0, ans=0.0
2023-11-20 17:30:59,603 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.494e+01 7.777e+01 8.468e+01 9.403e+01 1.279e+02, threshold=1.694e+02, percent-clipped=0.0
2023-11-20 17:31:04,479 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173150
2023-11-20 17:31:24,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1154400.0, ans=0.0
2023-11-20 17:31:47,317 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4850, loss[loss=0.0849, simple_loss=0.1141, pruned_loss=0.02094, audio_tagging_loss=0.006901, over 15737.00 frames. ], tot_loss[loss=0.07813, simple_loss=0.09855, pruned_loss=0.01855, audio_tagging_loss=0.01031, over 3038276.32 frames. ], batch size: 58, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:31:55,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1154533.3333333333, ans=0.1
2023-11-20 17:32:03,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1154600.0, ans=0.1
2023-11-20 17:32:08,552 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.97 vs. limit=15.0
2023-11-20 17:32:11,841 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173200
2023-11-20 17:32:16,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1154666.6666666667, ans=0.2
2023-11-20 17:32:30,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1154733.3333333333, ans=0.0
2023-11-20 17:32:51,426 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4900, loss[loss=0.07228, simple_loss=0.08658, pruned_loss=0.02053, audio_tagging_loss=0.008458, over 15036.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.09802, pruned_loss=0.01858, audio_tagging_loss=0.01031, over 3038722.79 frames. ], batch size: 57, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:32:57,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1154866.6666666667, ans=0.0
2023-11-20 17:33:09,827 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.588e+01 8.105e+01 8.801e+01 9.548e+01 1.347e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-20 17:33:13,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1154933.3333333333, ans=0.125
2023-11-20 17:33:14,816 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173250
2023-11-20 17:33:29,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1155066.6666666667, ans=0.125
2023-11-20 17:33:42,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1155133.3333333333, ans=0.125
2023-11-20 17:33:53,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1155133.3333333333, ans=0.0
2023-11-20 17:33:55,029 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 4950, loss[loss=0.08413, simple_loss=0.1032, pruned_loss=0.02275, audio_tagging_loss=0.009758, over 16059.00 frames. ], tot_loss[loss=0.07843, simple_loss=0.09871, pruned_loss=0.01886, audio_tagging_loss=0.01021, over 3036499.22 frames. ], batch size: 60, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:33:56,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1155200.0, ans=0.125
2023-11-20 17:33:57,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1155200.0, ans=0.125
2023-11-20 17:33:57,906 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.12 vs. limit=15.0
2023-11-20 17:34:00,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1155200.0, ans=0.0
2023-11-20 17:34:12,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1155266.6666666667, ans=0.125
2023-11-20 17:34:16,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173300
2023-11-20 17:34:20,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1155333.3333333333, ans=0.125
2023-11-20 17:34:31,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1155400.0, ans=0.0
2023-11-20 17:34:53,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1155466.6666666667, ans=0.1
2023-11-20 17:34:57,670 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5000, loss[loss=0.1112, simple_loss=0.1431, pruned_loss=0.03157, audio_tagging_loss=0.00804, over 14543.00 frames. ], tot_loss[loss=0.07881, simple_loss=0.09964, pruned_loss=0.01903, audio_tagging_loss=0.009962, over 3036431.26 frames. ], batch size: 53, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:35:05,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1155533.3333333333, ans=0.1
2023-11-20 17:35:16,435 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.798e+01 8.227e+01 8.824e+01 9.880e+01 1.350e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-20 17:35:20,297 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173350
2023-11-20 17:35:30,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1155666.6666666667, ans=0.125
2023-11-20 17:35:33,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1155666.6666666667, ans=0.125
2023-11-20 17:35:39,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1155733.3333333333, ans=0.0
2023-11-20 17:35:52,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1155800.0, ans=0.0
2023-11-20 17:35:59,957 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5050, loss[loss=0.06346, simple_loss=0.07854, pruned_loss=0.01625, audio_tagging_loss=0.007936, over 15573.00 frames. ], tot_loss[loss=0.07819, simple_loss=0.09905, pruned_loss=0.01883, audio_tagging_loss=0.009839, over 3040615.88 frames. ], batch size: 57, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:36:08,638 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.50 vs. limit=15.0
2023-11-20 17:36:10,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1155866.6666666667, ans=0.125
2023-11-20 17:36:23,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173400
2023-11-20 17:37:04,832 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5100, loss[loss=0.07305, simple_loss=0.09654, pruned_loss=0.01566, audio_tagging_loss=0.009121, over 15178.00 frames. ], tot_loss[loss=0.07782, simple_loss=0.09857, pruned_loss=0.01866, audio_tagging_loss=0.00988, over 3042018.96 frames. ], batch size: 57, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:37:12,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1156200.0, ans=0.05
2023-11-20 17:37:19,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1156266.6666666667, ans=0.07
2023-11-20 17:37:23,041 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.611e+01 8.188e+01 8.649e+01 9.367e+01 2.807e+02, threshold=1.730e+02, percent-clipped=1.0
2023-11-20 17:37:23,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1156266.6666666667, ans=0.125
2023-11-20 17:37:26,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173450
2023-11-20 17:37:36,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1156333.3333333333, ans=0.125
2023-11-20 17:37:49,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1156400.0, ans=0.1
2023-11-20 17:38:04,048 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.09 vs. limit=15.0
2023-11-20 17:38:07,899 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5150, loss[loss=0.04603, simple_loss=0.05592, pruned_loss=0.009325, audio_tagging_loss=0.008751, over 14418.00 frames. ], tot_loss[loss=0.07817, simple_loss=0.09898, pruned_loss=0.01884, audio_tagging_loss=0.009845, over 3038985.80 frames. ], batch size: 55, lr: 4.61e-03, grad_scale: 16.0
2023-11-20 17:38:11,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1156533.3333333333, ans=0.0
2023-11-20 17:38:27,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1156600.0, ans=0.0
2023-11-20 17:38:27,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1156600.0, ans=0.125
2023-11-20 17:38:30,356 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173500
2023-11-20 17:38:35,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1156666.6666666667, ans=0.2
2023-11-20 17:38:45,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1156733.3333333333, ans=0.125
2023-11-20 17:39:10,957 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5200, loss[loss=0.08223, simple_loss=0.1074, pruned_loss=0.02067, audio_tagging_loss=0.007882, over 15099.00 frames. ], tot_loss[loss=0.07815, simple_loss=0.09897, pruned_loss=0.01881, audio_tagging_loss=0.009859, over 3039851.71 frames. ], batch size: 56, lr: 4.61e-03, grad_scale: 32.0
2023-11-20 17:39:22,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1156866.6666666667, ans=0.1
2023-11-20 17:39:22,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1156866.6666666667, ans=0.125
2023-11-20 17:39:30,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1156933.3333333333, ans=0.0
2023-11-20 17:39:31,481 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.205e+01 8.715e+01 9.461e+01 1.258e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-20 17:39:35,269 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173550
2023-11-20 17:39:36,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1157000.0, ans=0.0
2023-11-20 17:39:42,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1157000.0, ans=0.125
2023-11-20 17:39:57,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1157066.6666666667, ans=0.125
2023-11-20 17:40:03,722 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.56 vs. limit=15.0
2023-11-20 17:40:05,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1157133.3333333333, ans=22.5
2023-11-20 17:40:15,232 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5250, loss[loss=0.07093, simple_loss=0.09166, pruned_loss=0.01604, audio_tagging_loss=0.00906, over 14301.00 frames. ], tot_loss[loss=0.07819, simple_loss=0.09899, pruned_loss=0.01893, audio_tagging_loss=0.009769, over 3036950.62 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:40:17,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1157200.0, ans=0.1
2023-11-20 17:40:22,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1157200.0, ans=0.0
2023-11-20 17:40:36,752 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.01 vs. limit=15.0
2023-11-20 17:40:37,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1157266.6666666667, ans=0.0
2023-11-20 17:40:38,374 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173600
2023-11-20 17:40:47,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1157333.3333333333, ans=0.125
2023-11-20 17:40:55,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1157400.0, ans=0.025
2023-11-20 17:41:19,888 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5300, loss[loss=0.07561, simple_loss=0.09712, pruned_loss=0.01348, audio_tagging_loss=0.01357, over 13966.00 frames. ], tot_loss[loss=0.078, simple_loss=0.09897, pruned_loss=0.01872, audio_tagging_loss=0.009792, over 3036538.03 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:41:30,284 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.45 vs. limit=15.0
2023-11-20 17:41:32,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1157600.0, ans=0.125
2023-11-20 17:41:37,922 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.138e+01 8.869e+01 9.897e+01 2.566e+02, threshold=1.774e+02, percent-clipped=1.0
2023-11-20 17:41:41,711 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173650
2023-11-20 17:41:58,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1157733.3333333333, ans=0.0
2023-11-20 17:42:22,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1157866.6666666667, ans=0.125
2023-11-20 17:42:22,920 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5350, loss[loss=0.04255, simple_loss=0.048, pruned_loss=0.00869, audio_tagging_loss=0.009859, over 14253.00 frames. ], tot_loss[loss=0.07801, simple_loss=0.09911, pruned_loss=0.01871, audio_tagging_loss=0.009746, over 3038953.15 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:42:30,884 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.66 vs. limit=6.0
2023-11-20 17:42:46,552 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173700
2023-11-20 17:42:49,448 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.70 vs. limit=10.0
2023-11-20 17:43:25,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1158200.0, ans=0.125
2023-11-20 17:43:26,685 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5400, loss[loss=0.1116, simple_loss=0.1486, pruned_loss=0.02895, audio_tagging_loss=0.008383, over 16448.00 frames. ], tot_loss[loss=0.07829, simple_loss=0.09904, pruned_loss=0.0189, audio_tagging_loss=0.009877, over 3040569.74 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:43:38,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1158266.6666666667, ans=0.125
2023-11-20 17:43:44,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1158266.6666666667, ans=0.2
2023-11-20 17:43:45,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.16 vs. limit=15.0
2023-11-20 17:43:46,150 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.350e+01 7.982e+01 8.687e+01 9.392e+01 1.840e+02, threshold=1.737e+02, percent-clipped=1.0
2023-11-20 17:43:49,993 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173750
2023-11-20 17:44:23,811 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.99 vs. limit=15.0
2023-11-20 17:44:24,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1158466.6666666667, ans=0.2
2023-11-20 17:44:28,242 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:44:30,519 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5450, loss[loss=0.0816, simple_loss=0.1064, pruned_loss=0.01762, audio_tagging_loss=0.01077, over 14758.00 frames. ], tot_loss[loss=0.07887, simple_loss=0.09954, pruned_loss=0.01922, audio_tagging_loss=0.009885, over 3039667.47 frames. ], batch size: 53, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:44:35,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1158533.3333333333, ans=0.125
2023-11-20 17:44:53,212 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173800
2023-11-20 17:44:54,004 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.89 vs. limit=15.0
2023-11-20 17:45:04,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.62 vs. limit=22.5
2023-11-20 17:45:08,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1158733.3333333333, ans=0.0
2023-11-20 17:45:13,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1158733.3333333333, ans=0.125
2023-11-20 17:45:20,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1158800.0, ans=0.125
2023-11-20 17:45:23,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1158800.0, ans=0.125
2023-11-20 17:45:32,434 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.37 vs. limit=15.0
2023-11-20 17:45:34,260 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5500, loss[loss=0.1012, simple_loss=0.1352, pruned_loss=0.02626, audio_tagging_loss=0.007316, over 15976.00 frames. ], tot_loss[loss=0.07909, simple_loss=0.1002, pruned_loss=0.0192, audio_tagging_loss=0.009791, over 3041158.20 frames. ], batch size: 58, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:45:54,014 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.646e+01 8.155e+01 8.582e+01 9.481e+01 1.342e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-20 17:45:57,166 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173850
2023-11-20 17:46:08,955 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.24 vs. limit=10.0
2023-11-20 17:46:25,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1159133.3333333333, ans=0.0
2023-11-20 17:46:37,046 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5550, loss[loss=0.08031, simple_loss=0.1049, pruned_loss=0.02043, audio_tagging_loss=0.007409, over 15091.00 frames. ], tot_loss[loss=0.07869, simple_loss=0.09967, pruned_loss=0.019, audio_tagging_loss=0.009854, over 3045111.86 frames. ], batch size: 55, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:46:42,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1159200.0, ans=0.1
2023-11-20 17:46:59,574 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173900
2023-11-20 17:47:06,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1159333.3333333333, ans=0.125
2023-11-20 17:47:09,138 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.89 vs. limit=22.5
2023-11-20 17:47:11,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1159333.3333333333, ans=0.125
2023-11-20 17:47:17,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1159400.0, ans=0.125
2023-11-20 17:47:18,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1159400.0, ans=0.125
2023-11-20 17:47:27,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1159466.6666666667, ans=0.125
2023-11-20 17:47:33,404 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.08 vs. limit=15.0
2023-11-20 17:47:40,158 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5600, loss[loss=0.07583, simple_loss=0.09612, pruned_loss=0.01787, audio_tagging_loss=0.009899, over 14993.00 frames. ], tot_loss[loss=0.07827, simple_loss=0.09933, pruned_loss=0.01867, audio_tagging_loss=0.009939, over 3047955.90 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 32.0
2023-11-20 17:47:41,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1159533.3333333333, ans=0.1
2023-11-20 17:47:50,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1159533.3333333333, ans=0.125
2023-11-20 17:48:00,193 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 7.988e+01 8.538e+01 9.609e+01 1.592e+02, threshold=1.708e+02, percent-clipped=0.0
2023-11-20 17:48:02,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 173950
2023-11-20 17:48:03,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1159600.0, ans=0.125
2023-11-20 17:48:12,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1159666.6666666667, ans=0.125
2023-11-20 17:48:24,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1159733.3333333333, ans=0.125
2023-11-20 17:48:26,231 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:48:44,016 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5650, loss[loss=0.07128, simple_loss=0.0978, pruned_loss=0.01243, audio_tagging_loss=0.009957, over 14559.00 frames. ], tot_loss[loss=0.07818, simple_loss=0.0992, pruned_loss=0.01856, audio_tagging_loss=0.01002, over 3048123.36 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:49:07,151 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174000
2023-11-20 17:49:29,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.18 vs. limit=15.0
2023-11-20 17:49:41,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1160133.3333333333, ans=0.125
2023-11-20 17:49:48,565 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5700, loss[loss=0.071, simple_loss=0.09475, pruned_loss=0.01308, audio_tagging_loss=0.01054, over 15209.00 frames. ], tot_loss[loss=0.07837, simple_loss=0.09952, pruned_loss=0.01861, audio_tagging_loss=0.01001, over 3046988.83 frames. ], batch size: 58, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:49:55,897 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.43 vs. limit=15.0
2023-11-20 17:49:58,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1160200.0, ans=0.0
2023-11-20 17:50:09,196 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.601e-02
2023-11-20 17:50:10,025 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.368e+01 9.255e+01 1.006e+02 1.511e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-20 17:50:11,501 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174050
2023-11-20 17:50:13,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1160333.3333333333, ans=0.0
2023-11-20 17:50:14,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1160333.3333333333, ans=0.2
2023-11-20 17:50:52,585 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5750, loss[loss=0.09308, simple_loss=0.1254, pruned_loss=0.02419, audio_tagging_loss=0.00619, over 15348.00 frames. ], tot_loss[loss=0.07706, simple_loss=0.0978, pruned_loss=0.0182, audio_tagging_loss=0.009965, over 3037709.45 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:51:06,466 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.14 vs. limit=6.0
2023-11-20 17:51:06,469 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.48 vs. limit=12.0
2023-11-20 17:51:12,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1160600.0, ans=0.125
2023-11-20 17:51:12,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1160600.0, ans=0.0
2023-11-20 17:51:15,173 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174100
2023-11-20 17:51:19,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1160666.6666666667, ans=0.125
2023-11-20 17:51:25,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1160666.6666666667, ans=0.2
2023-11-20 17:51:55,659 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5800, loss[loss=0.08431, simple_loss=0.1044, pruned_loss=0.02584, audio_tagging_loss=0.006285, over 15027.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09682, pruned_loss=0.01806, audio_tagging_loss=0.009903, over 3037378.71 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:51:55,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1160866.6666666667, ans=0.125
2023-11-20 17:52:17,730 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.484e+01 8.088e+01 8.646e+01 9.359e+01 1.315e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-20 17:52:19,114 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174150
2023-11-20 17:52:19,663 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.04 vs. limit=15.0
2023-11-20 17:52:23,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1161000.0, ans=0.0
2023-11-20 17:52:50,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1161133.3333333333, ans=0.2
2023-11-20 17:52:51,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1161133.3333333333, ans=0.125
2023-11-20 17:52:59,010 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5850, loss[loss=0.08547, simple_loss=0.1075, pruned_loss=0.0195, audio_tagging_loss=0.01221, over 15169.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.09625, pruned_loss=0.01809, audio_tagging_loss=0.009942, over 3033692.95 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:52:59,730 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.59 vs. limit=6.0
2023-11-20 17:53:05,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1161200.0, ans=0.1
2023-11-20 17:53:07,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1161200.0, ans=0.125
2023-11-20 17:53:22,401 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174200
2023-11-20 17:53:41,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1161400.0, ans=0.1
2023-11-20 17:53:44,721 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.80 vs. limit=10.0
2023-11-20 17:53:54,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1161466.6666666667, ans=0.1
2023-11-20 17:54:03,802 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5900, loss[loss=0.06931, simple_loss=0.09668, pruned_loss=0.01328, audio_tagging_loss=0.007696, over 15227.00 frames. ], tot_loss[loss=0.07679, simple_loss=0.09726, pruned_loss=0.01832, audio_tagging_loss=0.009832, over 3037121.88 frames. ], batch size: 57, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:54:11,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1161533.3333333333, ans=0.04949747468305833
2023-11-20 17:54:23,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1161600.0, ans=0.0
2023-11-20 17:54:23,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1161600.0, ans=0.2
2023-11-20 17:54:24,234 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.636e+01 8.417e+01 8.980e+01 9.978e+01 1.286e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-20 17:54:25,539 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174250
2023-11-20 17:54:50,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1161733.3333333333, ans=0.125
2023-11-20 17:54:53,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1161800.0, ans=0.125
2023-11-20 17:55:06,754 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 5950, loss[loss=0.1022, simple_loss=0.1241, pruned_loss=0.03068, audio_tagging_loss=0.009507, over 15476.00 frames. ], tot_loss[loss=0.07688, simple_loss=0.09717, pruned_loss=0.01841, audio_tagging_loss=0.009879, over 3038684.13 frames. ], batch size: 56, lr: 4.60e-03, grad_scale: 16.0
2023-11-20 17:55:07,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1161866.6666666667, ans=0.125
2023-11-20 17:55:08,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1161866.6666666667, ans=0.125
2023-11-20 17:55:30,077 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.80 vs. limit=22.5
2023-11-20 17:55:30,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174300
2023-11-20 17:55:41,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1162000.0, ans=0.0
2023-11-20 17:55:46,330 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 17:55:53,946 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.69 vs. limit=12.0
2023-11-20 17:56:03,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1162133.3333333333, ans=0.125
2023-11-20 17:56:09,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1162200.0, ans=0.125
2023-11-20 17:56:10,511 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6000, loss[loss=0.07117, simple_loss=0.07759, pruned_loss=0.01765, audio_tagging_loss=0.01473, over 15265.00 frames. ], tot_loss[loss=0.0767, simple_loss=0.09702, pruned_loss=0.01831, audio_tagging_loss=0.009878, over 3036182.31 frames. ], batch size: 60, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 17:56:10,514 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-20 17:56:38,497 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.3500, 0.5617, 3.3497, 3.1434, 2.7372, 3.0412, 2.7420, 2.8842],
       device='cuda:0')
2023-11-20 17:56:51,557 INFO [train_asr.py:1253] (0/4) Epoch 15, validation: loss=0.06114, simple_loss=0.05327, pruned_loss=0.005599, audio_tagging_loss=0.02891, over 4681554.00 frames. 
2023-11-20 17:56:51,558 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-20 17:56:51,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1162200.0, ans=0.2
2023-11-20 17:57:12,188 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.279e+01 8.029e+01 8.706e+01 9.735e+01 1.152e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-20 17:57:13,581 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174350
2023-11-20 17:57:38,301 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 17:57:53,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.22 vs. limit=15.0
2023-11-20 17:57:55,661 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6050, loss[loss=0.08885, simple_loss=0.1124, pruned_loss=0.02542, audio_tagging_loss=0.007224, over 14915.00 frames. ], tot_loss[loss=0.0774, simple_loss=0.09822, pruned_loss=0.01851, audio_tagging_loss=0.009776, over 3044651.93 frames. ], batch size: 54, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 17:57:57,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=15.0
2023-11-20 17:58:15,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1162600.0, ans=0.125
2023-11-20 17:58:19,549 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174400
2023-11-20 17:58:28,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1162666.6666666667, ans=0.04949747468305833
2023-11-20 17:58:28,555 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0
2023-11-20 17:58:54,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1162800.0, ans=0.0
2023-11-20 17:58:54,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1162800.0, ans=0.0
2023-11-20 17:58:57,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1162800.0, ans=0.125
2023-11-20 17:58:59,450 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6100, loss[loss=0.05884, simple_loss=0.07195, pruned_loss=0.01303, audio_tagging_loss=0.009839, over 15352.00 frames. ], tot_loss[loss=0.0781, simple_loss=0.09904, pruned_loss=0.01876, audio_tagging_loss=0.009824, over 3043660.51 frames. ], batch size: 59, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 17:59:16,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1162933.3333333333, ans=0.0
2023-11-20 17:59:22,145 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 7.952e+01 8.424e+01 9.114e+01 1.496e+02, threshold=1.685e+02, percent-clipped=0.0
2023-11-20 17:59:23,503 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174450
2023-11-20 18:00:04,775 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6150, loss[loss=0.08275, simple_loss=0.1112, pruned_loss=0.01867, audio_tagging_loss=0.008472, over 14681.00 frames. ], tot_loss[loss=0.07805, simple_loss=0.09894, pruned_loss=0.01871, audio_tagging_loss=0.009867, over 3045095.25 frames. ], batch size: 57, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:00:27,094 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174500
2023-11-20 18:00:38,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1163333.3333333333, ans=0.125
2023-11-20 18:00:44,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1163400.0, ans=0.0
2023-11-20 18:01:08,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1163533.3333333333, ans=0.0
2023-11-20 18:01:09,265 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6200, loss[loss=0.07421, simple_loss=0.08773, pruned_loss=0.01923, audio_tagging_loss=0.01111, over 14681.00 frames. ], tot_loss[loss=0.07764, simple_loss=0.09828, pruned_loss=0.01849, audio_tagging_loss=0.01002, over 3043570.22 frames. ], batch size: 56, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:01:10,085 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.79 vs. limit=12.0
2023-11-20 18:01:11,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1163533.3333333333, ans=0.1
2023-11-20 18:01:18,481 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.30 vs. limit=12.0
2023-11-20 18:01:22,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1163600.0, ans=0.1
2023-11-20 18:01:33,401 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.483e+01 8.002e+01 8.659e+01 9.317e+01 2.710e+02, threshold=1.732e+02, percent-clipped=1.0
2023-11-20 18:01:33,541 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174550
2023-11-20 18:01:52,947 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.88 vs. limit=22.5
2023-11-20 18:02:09,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1163800.0, ans=0.0
2023-11-20 18:02:09,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1163800.0, ans=0.0
2023-11-20 18:02:10,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1163800.0, ans=0.125
2023-11-20 18:02:13,309 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6250, loss[loss=0.08033, simple_loss=0.09345, pruned_loss=0.02095, audio_tagging_loss=0.01265, over 15286.00 frames. ], tot_loss[loss=0.07817, simple_loss=0.09868, pruned_loss=0.01874, audio_tagging_loss=0.01009, over 3034991.21 frames. ], batch size: 58, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:02:26,964 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.17 vs. limit=6.0
2023-11-20 18:02:27,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1163933.3333333333, ans=0.0
2023-11-20 18:02:37,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174600
2023-11-20 18:02:45,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1164000.0, ans=0.125
2023-11-20 18:02:47,577 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.28 vs. limit=15.0
2023-11-20 18:02:57,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1164066.6666666667, ans=0.125
2023-11-20 18:03:03,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1164133.3333333333, ans=0.0
2023-11-20 18:03:18,941 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6300, loss[loss=0.0965, simple_loss=0.1223, pruned_loss=0.02312, audio_tagging_loss=0.0122, over 14486.00 frames. ], tot_loss[loss=0.07876, simple_loss=0.09952, pruned_loss=0.01892, audio_tagging_loss=0.01008, over 3045134.23 frames. ], batch size: 54, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:03:40,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1164266.6666666667, ans=0.0
2023-11-20 18:03:41,348 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.283e+01 8.184e+01 8.938e+01 9.921e+01 1.399e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-20 18:03:41,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174650
2023-11-20 18:03:47,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1164333.3333333333, ans=0.1
2023-11-20 18:03:49,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1164333.3333333333, ans=0.0
2023-11-20 18:03:50,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1164333.3333333333, ans=0.125
2023-11-20 18:04:00,257 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.62 vs. limit=6.0
2023-11-20 18:04:12,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1164466.6666666667, ans=0.125
2023-11-20 18:04:23,034 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6350, loss[loss=0.08944, simple_loss=0.1148, pruned_loss=0.01987, audio_tagging_loss=0.01218, over 15416.00 frames. ], tot_loss[loss=0.07813, simple_loss=0.09881, pruned_loss=0.01859, audio_tagging_loss=0.01013, over 3044483.51 frames. ], batch size: 56, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:04:39,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1164600.0, ans=0.1
2023-11-20 18:04:46,338 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174700
2023-11-20 18:04:50,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1164666.6666666667, ans=0.0
2023-11-20 18:05:02,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1164733.3333333333, ans=0.1
2023-11-20 18:05:06,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1164733.3333333333, ans=0.125
2023-11-20 18:05:11,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.66 vs. limit=15.0
2023-11-20 18:05:23,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1164800.0, ans=0.1
2023-11-20 18:05:26,463 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6400, loss[loss=0.07947, simple_loss=0.103, pruned_loss=0.01919, audio_tagging_loss=0.008756, over 15665.00 frames. ], tot_loss[loss=0.0784, simple_loss=0.09898, pruned_loss=0.01869, audio_tagging_loss=0.01022, over 3046617.45 frames. ], batch size: 60, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 18:05:27,145 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.64 vs. limit=15.0
2023-11-20 18:05:34,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1164866.6666666667, ans=0.0
2023-11-20 18:05:43,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1164933.3333333333, ans=0.125
2023-11-20 18:05:50,744 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.152e+01 8.050e+01 8.682e+01 9.459e+01 1.192e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-20 18:05:50,902 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174750
2023-11-20 18:06:04,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1165066.6666666667, ans=0.125
2023-11-20 18:06:05,123 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.60 vs. limit=15.0
2023-11-20 18:06:11,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.60 vs. limit=6.0
2023-11-20 18:06:26,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1165133.3333333333, ans=0.125
2023-11-20 18:06:31,334 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6450, loss[loss=0.09086, simple_loss=0.1152, pruned_loss=0.02077, audio_tagging_loss=0.01249, over 15630.00 frames. ], tot_loss[loss=0.07857, simple_loss=0.09905, pruned_loss=0.0188, audio_tagging_loss=0.01025, over 3047362.75 frames. ], batch size: 56, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 18:06:40,839 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.57 vs. limit=15.0
2023-11-20 18:06:54,865 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174800
2023-11-20 18:06:55,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1165266.6666666667, ans=0.125
2023-11-20 18:07:19,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1165400.0, ans=0.125
2023-11-20 18:07:33,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1165466.6666666667, ans=0.1
2023-11-20 18:07:36,638 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6500, loss[loss=0.06713, simple_loss=0.08321, pruned_loss=0.01637, audio_tagging_loss=0.009151, over 15249.00 frames. ], tot_loss[loss=0.07842, simple_loss=0.09858, pruned_loss=0.01892, audio_tagging_loss=0.01021, over 3045673.98 frames. ], batch size: 57, lr: 4.59e-03, grad_scale: 32.0
2023-11-20 18:07:42,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1165533.3333333333, ans=0.0
2023-11-20 18:07:56,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1165600.0, ans=0.125
2023-11-20 18:07:58,593 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174850
2023-11-20 18:08:00,208 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.390e+01 7.879e+01 8.698e+01 9.570e+01 1.330e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-20 18:08:29,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1165800.0, ans=0.2
2023-11-20 18:08:29,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1165800.0, ans=0.0
2023-11-20 18:08:36,080 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.48 vs. limit=12.0
2023-11-20 18:08:40,138 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6550, loss[loss=0.06192, simple_loss=0.07375, pruned_loss=0.01613, audio_tagging_loss=0.008922, over 14902.00 frames. ], tot_loss[loss=0.07811, simple_loss=0.09836, pruned_loss=0.01884, audio_tagging_loss=0.0101, over 3045581.04 frames. ], batch size: 57, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:09:03,921 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174900
2023-11-20 18:09:10,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1166000.0, ans=0.125
2023-11-20 18:09:28,608 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.70 vs. limit=15.0
2023-11-20 18:09:44,718 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6600, loss[loss=0.08124, simple_loss=0.1079, pruned_loss=0.01966, audio_tagging_loss=0.007627, over 16227.00 frames. ], tot_loss[loss=0.07802, simple_loss=0.0989, pruned_loss=0.01869, audio_tagging_loss=0.009879, over 3046935.37 frames. ], batch size: 58, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:09:50,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.83 vs. limit=15.0
2023-11-20 18:09:55,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1166200.0, ans=0.1
2023-11-20 18:09:55,623 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.52 vs. limit=22.5
2023-11-20 18:09:56,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1166266.6666666667, ans=0.125
2023-11-20 18:10:01,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1166266.6666666667, ans=0.125
2023-11-20 18:10:06,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1166266.6666666667, ans=0.05
2023-11-20 18:10:08,407 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 174950
2023-11-20 18:10:09,445 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.156e+01 8.016e+01 8.893e+01 9.402e+01 1.270e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-20 18:10:11,025 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:10:38,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1166466.6666666667, ans=0.125
2023-11-20 18:10:42,030 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.61 vs. limit=22.5
2023-11-20 18:10:48,690 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6650, loss[loss=0.07445, simple_loss=0.1018, pruned_loss=0.01772, audio_tagging_loss=0.005837, over 13794.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.09815, pruned_loss=0.01834, audio_tagging_loss=0.009752, over 3045527.19 frames. ], batch size: 53, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:10:58,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.03 vs. limit=15.0
2023-11-20 18:11:10,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=1166600.0, ans=0.2
2023-11-20 18:11:11,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175000
2023-11-20 18:11:26,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.42 vs. limit=22.5
2023-11-20 18:11:31,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1166733.3333333333, ans=0.125
2023-11-20 18:11:53,102 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6700, loss[loss=0.09243, simple_loss=0.1169, pruned_loss=0.0238, audio_tagging_loss=0.01015, over 16698.00 frames. ], tot_loss[loss=0.07659, simple_loss=0.09719, pruned_loss=0.01814, audio_tagging_loss=0.009856, over 3050325.37 frames. ], batch size: 60, lr: 4.59e-03, grad_scale: 16.0
2023-11-20 18:11:56,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1166866.6666666667, ans=0.125
2023-11-20 18:11:58,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1166866.6666666667, ans=0.125
2023-11-20 18:12:15,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175050
2023-11-20 18:12:17,144 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.109e+01 8.688e+01 9.322e+01 1.481e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 18:12:23,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1167000.0, ans=0.0
2023-11-20 18:12:56,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1167200.0, ans=0.1
2023-11-20 18:12:57,125 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6750, loss[loss=0.09227, simple_loss=0.1085, pruned_loss=0.02201, audio_tagging_loss=0.01602, over 15539.00 frames. ], tot_loss[loss=0.07657, simple_loss=0.09717, pruned_loss=0.01804, audio_tagging_loss=0.009941, over 3050829.18 frames. ], batch size: 59, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:12:57,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1167200.0, ans=0.0
2023-11-20 18:13:20,225 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175100
2023-11-20 18:13:56,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1167466.6666666667, ans=0.125
2023-11-20 18:14:01,466 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6800, loss[loss=0.07369, simple_loss=0.09816, pruned_loss=0.01537, audio_tagging_loss=0.009241, over 15503.00 frames. ], tot_loss[loss=0.07712, simple_loss=0.09791, pruned_loss=0.0183, audio_tagging_loss=0.009867, over 3050991.95 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:14:20,159 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.74 vs. limit=10.0
2023-11-20 18:14:21,115 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.31 vs. limit=15.0
2023-11-20 18:14:24,207 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175150
2023-11-20 18:14:25,252 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.115e+01 8.227e+01 9.120e+01 1.003e+02 1.352e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-20 18:14:41,680 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.57 vs. limit=15.0
2023-11-20 18:14:45,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1167733.3333333333, ans=0.125
2023-11-20 18:14:46,132 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.22 vs. limit=12.0
2023-11-20 18:14:51,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1167800.0, ans=0.125
2023-11-20 18:15:02,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1167800.0, ans=0.125
2023-11-20 18:15:05,554 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6850, loss[loss=0.06101, simple_loss=0.0755, pruned_loss=0.01032, audio_tagging_loss=0.01294, over 15123.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09769, pruned_loss=0.01835, audio_tagging_loss=0.009915, over 3046277.53 frames. ], batch size: 56, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:15:22,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1167933.3333333333, ans=0.125
2023-11-20 18:15:28,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175200
2023-11-20 18:15:38,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1168000.0, ans=0.125
2023-11-20 18:15:40,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1168000.0, ans=0.0
2023-11-20 18:15:56,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1168133.3333333333, ans=0.09899494936611666
2023-11-20 18:15:58,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1168133.3333333333, ans=0.0
2023-11-20 18:16:10,193 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6900, loss[loss=0.057, simple_loss=0.06403, pruned_loss=0.01274, audio_tagging_loss=0.01225, over 15215.00 frames. ], tot_loss[loss=0.07705, simple_loss=0.09818, pruned_loss=0.01817, audio_tagging_loss=0.009794, over 3048680.07 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:16:11,145 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.56 vs. limit=15.0
2023-11-20 18:16:33,536 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175250
2023-11-20 18:16:34,532 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.834e+01 8.085e+01 8.884e+01 9.597e+01 1.266e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-20 18:16:39,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1168333.3333333333, ans=0.1
2023-11-20 18:16:59,604 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.12 vs. limit=15.0
2023-11-20 18:17:00,873 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 18:17:02,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1168466.6666666667, ans=0.04949747468305833
2023-11-20 18:17:06,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1168466.6666666667, ans=0.125
2023-11-20 18:17:14,806 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 6950, loss[loss=0.06952, simple_loss=0.08933, pruned_loss=0.01352, audio_tagging_loss=0.01133, over 14988.00 frames. ], tot_loss[loss=0.07719, simple_loss=0.09832, pruned_loss=0.01822, audio_tagging_loss=0.009805, over 3043326.60 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:17:37,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175300
2023-11-20 18:17:55,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1168733.3333333333, ans=0.125
2023-11-20 18:17:55,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1168733.3333333333, ans=0.0
2023-11-20 18:18:18,363 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7000, loss[loss=0.09243, simple_loss=0.1209, pruned_loss=0.02262, audio_tagging_loss=0.009338, over 15773.00 frames. ], tot_loss[loss=0.07765, simple_loss=0.09857, pruned_loss=0.01846, audio_tagging_loss=0.009907, over 3033741.86 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:18:29,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1168866.6666666667, ans=0.04949747468305833
2023-11-20 18:18:41,601 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175350
2023-11-20 18:18:42,700 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.669e+01 7.873e+01 8.691e+01 9.185e+01 1.165e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 18:18:45,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1169000.0, ans=0.0
2023-11-20 18:18:46,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1169000.0, ans=0.125
2023-11-20 18:19:05,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1169066.6666666667, ans=0.1
2023-11-20 18:19:14,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1169133.3333333333, ans=0.1
2023-11-20 18:19:16,567 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.46 vs. limit=15.0
2023-11-20 18:19:21,839 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7050, loss[loss=0.05662, simple_loss=0.0705, pruned_loss=0.01181, audio_tagging_loss=0.009557, over 15604.00 frames. ], tot_loss[loss=0.07778, simple_loss=0.09864, pruned_loss=0.01853, audio_tagging_loss=0.009925, over 3036830.08 frames. ], batch size: 59, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:19:45,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175400
2023-11-20 18:19:51,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1169333.3333333333, ans=0.95
2023-11-20 18:20:18,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1169466.6666666667, ans=0.125
2023-11-20 18:20:18,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1169466.6666666667, ans=0.2
2023-11-20 18:20:26,371 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7100, loss[loss=0.0659, simple_loss=0.07882, pruned_loss=0.01492, audio_tagging_loss=0.01157, over 15169.00 frames. ], tot_loss[loss=0.07792, simple_loss=0.09882, pruned_loss=0.01852, audio_tagging_loss=0.009989, over 3046322.02 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:20:36,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1169533.3333333333, ans=0.125
2023-11-20 18:20:40,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1169600.0, ans=0.0
2023-11-20 18:20:46,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1169600.0, ans=0.1
2023-11-20 18:20:48,707 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175450
2023-11-20 18:20:49,780 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.811e+01 8.001e+01 8.747e+01 9.712e+01 1.225e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-20 18:21:15,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1169733.3333333333, ans=0.1
2023-11-20 18:21:29,407 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7150, loss[loss=0.06829, simple_loss=0.09068, pruned_loss=0.01562, audio_tagging_loss=0.007323, over 14836.00 frames. ], tot_loss[loss=0.07799, simple_loss=0.09871, pruned_loss=0.01859, audio_tagging_loss=0.01005, over 3043405.13 frames. ], batch size: 56, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:21:49,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1169933.3333333333, ans=0.0
2023-11-20 18:21:53,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175500
2023-11-20 18:21:54,837 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0
2023-11-20 18:21:58,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1170000.0, ans=0.125
2023-11-20 18:22:15,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1170066.6666666667, ans=0.2
2023-11-20 18:22:32,647 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7200, loss[loss=0.05504, simple_loss=0.07024, pruned_loss=0.008478, audio_tagging_loss=0.01144, over 15927.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09806, pruned_loss=0.01847, audio_tagging_loss=0.01012, over 3038081.51 frames. ], batch size: 61, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:22:38,876 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.90 vs. limit=15.0
2023-11-20 18:22:56,698 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175550
2023-11-20 18:22:57,775 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.535e+01 8.191e+01 8.742e+01 9.688e+01 2.740e+02, threshold=1.748e+02, percent-clipped=1.0
2023-11-20 18:23:01,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1170333.3333333333, ans=0.125
2023-11-20 18:23:03,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1170333.3333333333, ans=0.125
2023-11-20 18:23:16,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1170400.0, ans=0.125
2023-11-20 18:23:31,403 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:23:37,156 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7250, loss[loss=0.07148, simple_loss=0.09302, pruned_loss=0.01805, audio_tagging_loss=0.006915, over 14618.00 frames. ], tot_loss[loss=0.07742, simple_loss=0.09769, pruned_loss=0.01838, audio_tagging_loss=0.01019, over 3045594.88 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 32.0
2023-11-20 18:23:40,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.22 vs. limit=15.0
2023-11-20 18:23:46,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1170533.3333333333, ans=0.125
2023-11-20 18:23:47,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1170533.3333333333, ans=0.0
2023-11-20 18:23:59,858 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175600
2023-11-20 18:24:02,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1170666.6666666667, ans=0.125
2023-11-20 18:24:05,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1170666.6666666667, ans=0.125
2023-11-20 18:24:07,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1170666.6666666667, ans=0.07
2023-11-20 18:24:10,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1170666.6666666667, ans=0.1
2023-11-20 18:24:15,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1170733.3333333333, ans=0.125
2023-11-20 18:24:16,326 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.71 vs. limit=15.0
2023-11-20 18:24:34,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1170800.0, ans=0.125
2023-11-20 18:24:35,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1170800.0, ans=0.1
2023-11-20 18:24:38,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1170800.0, ans=0.05
2023-11-20 18:24:41,088 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7300, loss[loss=0.08745, simple_loss=0.1108, pruned_loss=0.02196, audio_tagging_loss=0.01007, over 15523.00 frames. ], tot_loss[loss=0.07811, simple_loss=0.09887, pruned_loss=0.01864, audio_tagging_loss=0.01003, over 3043430.48 frames. ], batch size: 57, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:24:44,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1170866.6666666667, ans=0.5
2023-11-20 18:24:52,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1170933.3333333333, ans=0.125
2023-11-20 18:24:53,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1170933.3333333333, ans=0.125
2023-11-20 18:25:01,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1170933.3333333333, ans=0.125
2023-11-20 18:25:03,895 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175650
2023-11-20 18:25:06,283 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 7.796e+01 8.638e+01 9.366e+01 1.171e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-20 18:25:43,790 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7350, loss[loss=0.085, simple_loss=0.1056, pruned_loss=0.02354, audio_tagging_loss=0.008648, over 13595.00 frames. ], tot_loss[loss=0.07829, simple_loss=0.09927, pruned_loss=0.01881, audio_tagging_loss=0.009836, over 3039128.09 frames. ], batch size: 54, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:25:45,673 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.64 vs. limit=12.0
2023-11-20 18:25:53,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1171200.0, ans=0.125
2023-11-20 18:26:07,568 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175700
2023-11-20 18:26:17,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1171333.3333333333, ans=0.125
2023-11-20 18:26:41,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1171466.6666666667, ans=0.125
2023-11-20 18:26:47,852 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7400, loss[loss=0.08238, simple_loss=0.1062, pruned_loss=0.01651, audio_tagging_loss=0.01277, over 16034.00 frames. ], tot_loss[loss=0.0785, simple_loss=0.09969, pruned_loss=0.01887, audio_tagging_loss=0.009783, over 3045476.40 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:27:10,351 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175750
2023-11-20 18:27:12,676 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.603e+01 8.125e+01 8.808e+01 9.818e+01 1.259e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-20 18:27:20,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1171666.6666666667, ans=0.125
2023-11-20 18:27:34,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.72 vs. limit=22.5
2023-11-20 18:27:36,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn1.whiten.whitening_limit, batch_count=1171733.3333333333, ans=22.5
2023-11-20 18:27:40,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1171800.0, ans=0.125
2023-11-20 18:27:43,238 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.24 vs. limit=15.0
2023-11-20 18:27:51,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.48 vs. limit=15.0
2023-11-20 18:27:51,329 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7450, loss[loss=0.07805, simple_loss=0.09137, pruned_loss=0.02029, audio_tagging_loss=0.01208, over 14772.00 frames. ], tot_loss[loss=0.07848, simple_loss=0.09971, pruned_loss=0.01882, audio_tagging_loss=0.00981, over 3043941.37 frames. ], batch size: 58, lr: 4.58e-03, grad_scale: 16.0
2023-11-20 18:28:01,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1171866.6666666667, ans=0.125
2023-11-20 18:28:13,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175800
2023-11-20 18:28:14,375 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.15 vs. limit=6.0
2023-11-20 18:28:22,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1172000.0, ans=0.1
2023-11-20 18:28:47,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1172133.3333333333, ans=0.0
2023-11-20 18:28:48,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1172133.3333333333, ans=0.125
2023-11-20 18:28:54,514 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7500, loss[loss=0.06334, simple_loss=0.08231, pruned_loss=0.01272, audio_tagging_loss=0.009472, over 15247.00 frames. ], tot_loss[loss=0.07802, simple_loss=0.09909, pruned_loss=0.01864, audio_tagging_loss=0.009831, over 3041385.94 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:28:58,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1172200.0, ans=0.125
2023-11-20 18:29:05,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1172200.0, ans=0.0
2023-11-20 18:29:09,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.16 vs. limit=22.5
2023-11-20 18:29:16,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1172266.6666666667, ans=0.0
2023-11-20 18:29:19,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175850
2023-11-20 18:29:21,657 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.057e+01 8.747e+01 9.687e+01 1.264e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-20 18:29:59,022 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7550, loss[loss=0.06822, simple_loss=0.08766, pruned_loss=0.015, audio_tagging_loss=0.00939, over 15275.00 frames. ], tot_loss[loss=0.07826, simple_loss=0.09928, pruned_loss=0.01888, audio_tagging_loss=0.00974, over 3038013.00 frames. ], batch size: 54, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:30:02,307 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.35 vs. limit=10.0
2023-11-20 18:30:12,608 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=12.0
2023-11-20 18:30:22,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175900
2023-11-20 18:30:37,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.01 vs. limit=15.0
2023-11-20 18:30:44,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.18 vs. limit=12.0
2023-11-20 18:30:45,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1172733.3333333333, ans=0.125
2023-11-20 18:30:51,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1172800.0, ans=0.5
2023-11-20 18:30:56,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1172800.0, ans=0.125
2023-11-20 18:31:03,477 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7600, loss[loss=0.05813, simple_loss=0.07156, pruned_loss=0.01316, audio_tagging_loss=0.009189, over 14962.00 frames. ], tot_loss[loss=0.07836, simple_loss=0.09925, pruned_loss=0.01899, audio_tagging_loss=0.009742, over 3037247.92 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:31:07,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1172866.6666666667, ans=0.0
2023-11-20 18:31:23,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1172933.3333333333, ans=0.125
2023-11-20 18:31:25,636 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 175950
2023-11-20 18:31:27,954 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.999e+01 7.835e+01 8.443e+01 9.077e+01 1.271e+02, threshold=1.689e+02, percent-clipped=0.0
2023-11-20 18:31:37,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1173000.0, ans=0.125
2023-11-20 18:31:49,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1173066.6666666667, ans=0.125
2023-11-20 18:32:06,910 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7650, loss[loss=0.06989, simple_loss=0.09687, pruned_loss=0.01134, audio_tagging_loss=0.01012, over 15672.00 frames. ], tot_loss[loss=0.07792, simple_loss=0.09876, pruned_loss=0.01877, audio_tagging_loss=0.009766, over 3038002.86 frames. ], batch size: 60, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:32:20,173 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.87 vs. limit=15.0
2023-11-20 18:32:28,369 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.47 vs. limit=12.0
2023-11-20 18:32:30,229 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176000
2023-11-20 18:32:31,673 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-176000.pt
2023-11-20 18:32:37,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1173333.3333333333, ans=0.125
2023-11-20 18:32:56,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1173400.0, ans=0.2
2023-11-20 18:33:14,112 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7700, loss[loss=0.06805, simple_loss=0.08643, pruned_loss=0.01388, audio_tagging_loss=0.01096, over 15010.00 frames. ], tot_loss[loss=0.07787, simple_loss=0.09903, pruned_loss=0.01869, audio_tagging_loss=0.00967, over 3041429.47 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:33:14,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1173533.3333333333, ans=0.0
2023-11-20 18:33:14,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1173533.3333333333, ans=0.1
2023-11-20 18:33:22,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1173533.3333333333, ans=0.125
2023-11-20 18:33:30,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1173600.0, ans=0.0
2023-11-20 18:33:30,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1173600.0, ans=0.2
2023-11-20 18:33:37,326 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176050
2023-11-20 18:33:39,695 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.684e+01 8.040e+01 8.573e+01 9.268e+01 1.195e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-20 18:33:47,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1173666.6666666667, ans=0.1
2023-11-20 18:34:18,329 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7750, loss[loss=0.08842, simple_loss=0.1077, pruned_loss=0.02266, audio_tagging_loss=0.01189, over 15067.00 frames. ], tot_loss[loss=0.07806, simple_loss=0.09929, pruned_loss=0.01867, audio_tagging_loss=0.009752, over 3043297.99 frames. ], batch size: 59, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:34:39,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.61 vs. limit=15.0
2023-11-20 18:34:41,084 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176100
2023-11-20 18:34:41,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1173933.3333333333, ans=0.125
2023-11-20 18:35:02,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1174066.6666666667, ans=0.125
2023-11-20 18:35:15,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1174133.3333333333, ans=0.125
2023-11-20 18:35:22,307 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7800, loss[loss=0.07124, simple_loss=0.09778, pruned_loss=0.01339, audio_tagging_loss=0.008953, over 15051.00 frames. ], tot_loss[loss=0.07835, simple_loss=0.09972, pruned_loss=0.01866, audio_tagging_loss=0.009834, over 3048809.51 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:35:28,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1174200.0, ans=0.05
2023-11-20 18:35:44,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1174266.6666666667, ans=0.125
2023-11-20 18:35:45,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176150
2023-11-20 18:35:48,943 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.331e+01 9.045e+01 1.006e+02 1.273e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-20 18:36:02,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1174400.0, ans=0.125
2023-11-20 18:36:09,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1174400.0, ans=0.0
2023-11-20 18:36:17,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1174466.6666666667, ans=0.2
2023-11-20 18:36:22,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff2.min_abs, batch_count=1174466.6666666667, ans=0.1
2023-11-20 18:36:23,913 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.58 vs. limit=15.0
2023-11-20 18:36:26,129 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7850, loss[loss=0.06587, simple_loss=0.07582, pruned_loss=0.0173, audio_tagging_loss=0.01066, over 16299.00 frames. ], tot_loss[loss=0.07882, simple_loss=0.1003, pruned_loss=0.01881, audio_tagging_loss=0.009876, over 3048008.07 frames. ], batch size: 65, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:36:30,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1174533.3333333333, ans=0.125
2023-11-20 18:36:30,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1174533.3333333333, ans=0.0
2023-11-20 18:36:50,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176200
2023-11-20 18:36:58,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1174666.6666666667, ans=0.2
2023-11-20 18:37:04,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1174733.3333333333, ans=0.5
2023-11-20 18:37:10,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1174733.3333333333, ans=0.125
2023-11-20 18:37:19,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1174800.0, ans=0.125
2023-11-20 18:37:21,528 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:37:25,156 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:37:26,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1174800.0, ans=0.0
2023-11-20 18:37:31,668 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7900, loss[loss=0.06948, simple_loss=0.09503, pruned_loss=0.01292, audio_tagging_loss=0.009047, over 14679.00 frames. ], tot_loss[loss=0.07868, simple_loss=0.09988, pruned_loss=0.01883, audio_tagging_loss=0.009915, over 3043083.07 frames. ], batch size: 55, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:37:54,461 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176250
2023-11-20 18:37:57,519 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.91 vs. limit=15.0
2023-11-20 18:37:57,993 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.028e+01 8.743e+01 9.651e+01 2.118e+02, threshold=1.749e+02, percent-clipped=1.0
2023-11-20 18:38:01,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1175000.0, ans=0.0
2023-11-20 18:38:04,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1175000.0, ans=0.025
2023-11-20 18:38:07,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1175000.0, ans=0.1
2023-11-20 18:38:12,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1175066.6666666667, ans=0.2
2023-11-20 18:38:14,488 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.98 vs. limit=10.0
2023-11-20 18:38:16,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1175066.6666666667, ans=0.2
2023-11-20 18:38:31,648 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.64 vs. limit=6.0
2023-11-20 18:38:32,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1175133.3333333333, ans=0.125
2023-11-20 18:38:35,861 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 7950, loss[loss=0.05581, simple_loss=0.07107, pruned_loss=0.01102, audio_tagging_loss=0.009252, over 14766.00 frames. ], tot_loss[loss=0.0793, simple_loss=0.1006, pruned_loss=0.01909, audio_tagging_loss=0.00991, over 3040596.00 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 16.0
2023-11-20 18:38:37,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1175200.0, ans=0.0
2023-11-20 18:38:44,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1175200.0, ans=0.125
2023-11-20 18:38:51,799 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 18:38:55,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1175266.6666666667, ans=0.2
2023-11-20 18:38:59,249 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176300
2023-11-20 18:39:16,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1175400.0, ans=0.125
2023-11-20 18:39:27,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1175466.6666666667, ans=0.2
2023-11-20 18:39:39,806 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8000, loss[loss=0.07999, simple_loss=0.09949, pruned_loss=0.01898, audio_tagging_loss=0.01126, over 15530.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.09939, pruned_loss=0.01874, audio_tagging_loss=0.01003, over 3046392.25 frames. ], batch size: 57, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:40:01,609 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.54 vs. limit=22.5
2023-11-20 18:40:03,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176350
2023-11-20 18:40:07,203 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.314e+01 8.360e+01 9.089e+01 9.769e+01 1.308e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-20 18:40:07,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1175666.6666666667, ans=0.1
2023-11-20 18:40:07,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1175666.6666666667, ans=0.0
2023-11-20 18:40:13,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1175666.6666666667, ans=0.05
2023-11-20 18:40:15,631 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.43 vs. limit=22.5
2023-11-20 18:40:15,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.34 vs. limit=15.0
2023-11-20 18:40:25,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1175733.3333333333, ans=0.125
2023-11-20 18:40:44,424 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8050, loss[loss=0.1121, simple_loss=0.149, pruned_loss=0.03087, audio_tagging_loss=0.00676, over 15829.00 frames. ], tot_loss[loss=0.07912, simple_loss=0.1002, pruned_loss=0.01895, audio_tagging_loss=0.01007, over 3044787.01 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:41:07,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176400
2023-11-20 18:41:31,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1176066.6666666667, ans=0.125
2023-11-20 18:41:33,329 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.39 vs. limit=15.0
2023-11-20 18:41:43,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1176133.3333333333, ans=0.2
2023-11-20 18:41:49,108 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8100, loss[loss=0.05735, simple_loss=0.06864, pruned_loss=0.01207, audio_tagging_loss=0.01096, over 13845.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.09963, pruned_loss=0.01868, audio_tagging_loss=0.009969, over 3045930.88 frames. ], batch size: 56, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:41:51,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1176200.0, ans=0.05
2023-11-20 18:42:13,014 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176450
2023-11-20 18:42:16,560 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.737e+01 7.970e+01 8.520e+01 9.468e+01 1.287e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-20 18:42:18,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1176333.3333333333, ans=0.0
2023-11-20 18:42:47,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1176466.6666666667, ans=0.125
2023-11-20 18:42:53,295 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8150, loss[loss=0.09401, simple_loss=0.1291, pruned_loss=0.0218, audio_tagging_loss=0.00768, over 15600.00 frames. ], tot_loss[loss=0.07835, simple_loss=0.09927, pruned_loss=0.01884, audio_tagging_loss=0.00988, over 3043867.44 frames. ], batch size: 58, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:43:07,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1176600.0, ans=0.125
2023-11-20 18:43:17,159 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176500
2023-11-20 18:43:17,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1176600.0, ans=0.125
2023-11-20 18:43:57,833 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8200, loss[loss=0.07204, simple_loss=0.09224, pruned_loss=0.01849, audio_tagging_loss=0.00742, over 14958.00 frames. ], tot_loss[loss=0.07833, simple_loss=0.09963, pruned_loss=0.01865, audio_tagging_loss=0.009856, over 3056950.69 frames. ], batch size: 54, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:43:59,136 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 18:44:20,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176550
2023-11-20 18:44:24,660 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.959e+01 8.657e+01 9.371e+01 1.307e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-20 18:44:39,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1177066.6666666667, ans=0.0
2023-11-20 18:45:02,164 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8250, loss[loss=0.09697, simple_loss=0.1301, pruned_loss=0.02245, audio_tagging_loss=0.009485, over 15967.00 frames. ], tot_loss[loss=0.07811, simple_loss=0.0994, pruned_loss=0.0186, audio_tagging_loss=0.009807, over 3058788.45 frames. ], batch size: 58, lr: 4.57e-03, grad_scale: 32.0
2023-11-20 18:45:15,697 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.67 vs. limit=15.0
2023-11-20 18:45:17,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1177266.6666666667, ans=0.0
2023-11-20 18:45:25,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176600
2023-11-20 18:45:29,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1177333.3333333333, ans=0.125
2023-11-20 18:45:34,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1177333.3333333333, ans=0.0
2023-11-20 18:45:35,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1177333.3333333333, ans=0.125
2023-11-20 18:45:46,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1177400.0, ans=0.125
2023-11-20 18:45:46,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1177400.0, ans=0.0
2023-11-20 18:46:06,173 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8300, loss[loss=0.05436, simple_loss=0.06304, pruned_loss=0.01137, audio_tagging_loss=0.01148, over 15233.00 frames. ], tot_loss[loss=0.07866, simple_loss=0.1001, pruned_loss=0.0188, audio_tagging_loss=0.009821, over 3062106.81 frames. ], batch size: 58, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:46:21,424 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.29 vs. limit=22.5
2023-11-20 18:46:22,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1177600.0, ans=0.125
2023-11-20 18:46:29,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1177600.0, ans=0.0
2023-11-20 18:46:30,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176650
2023-11-20 18:46:33,811 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.522e+01 7.955e+01 8.600e+01 9.336e+01 1.507e+02, threshold=1.720e+02, percent-clipped=0.0
2023-11-20 18:46:56,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1177800.0, ans=0.0
2023-11-20 18:47:11,259 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8350, loss[loss=0.07171, simple_loss=0.09283, pruned_loss=0.01502, audio_tagging_loss=0.01028, over 15139.00 frames. ], tot_loss[loss=0.07785, simple_loss=0.09908, pruned_loss=0.01846, audio_tagging_loss=0.009846, over 3062473.35 frames. ], batch size: 55, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:47:13,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.60 vs. limit=22.5
2023-11-20 18:47:25,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1177933.3333333333, ans=0.09899494936611666
2023-11-20 18:47:31,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1177933.3333333333, ans=0.0
2023-11-20 18:47:33,909 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176700
2023-11-20 18:47:37,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1178000.0, ans=0.125
2023-11-20 18:47:43,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten.whitening_limit, batch_count=1178000.0, ans=15.0
2023-11-20 18:47:49,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1178066.6666666667, ans=0.125
2023-11-20 18:47:59,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1178066.6666666667, ans=0.1
2023-11-20 18:48:15,746 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8400, loss[loss=0.06954, simple_loss=0.08563, pruned_loss=0.01659, audio_tagging_loss=0.01013, over 15053.00 frames. ], tot_loss[loss=0.07768, simple_loss=0.09873, pruned_loss=0.01853, audio_tagging_loss=0.009784, over 3055449.14 frames. ], batch size: 59, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:48:26,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1178266.6666666667, ans=0.015
2023-11-20 18:48:38,704 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176750
2023-11-20 18:48:42,872 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.237e+01 7.973e+01 8.686e+01 9.472e+01 1.183e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-20 18:48:57,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1178400.0, ans=0.0
2023-11-20 18:48:58,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1178400.0, ans=0.125
2023-11-20 18:49:19,827 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8450, loss[loss=0.06746, simple_loss=0.08911, pruned_loss=0.01398, audio_tagging_loss=0.008927, over 15346.00 frames. ], tot_loss[loss=0.07742, simple_loss=0.09857, pruned_loss=0.01838, audio_tagging_loss=0.009758, over 3059799.22 frames. ], batch size: 57, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:49:25,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1178533.3333333333, ans=0.125
2023-11-20 18:49:26,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1178533.3333333333, ans=0.125
2023-11-20 18:49:27,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1178533.3333333333, ans=0.1
2023-11-20 18:49:43,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176800
2023-11-20 18:49:54,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1178666.6666666667, ans=0.125
2023-11-20 18:50:10,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1178800.0, ans=0.125
2023-11-20 18:50:25,064 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8500, loss[loss=0.1005, simple_loss=0.1337, pruned_loss=0.02713, audio_tagging_loss=0.006488, over 15256.00 frames. ], tot_loss[loss=0.07812, simple_loss=0.0997, pruned_loss=0.01855, audio_tagging_loss=0.009717, over 3054969.75 frames. ], batch size: 57, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:50:34,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1178866.6666666667, ans=0.125
2023-11-20 18:50:47,894 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176850
2023-11-20 18:50:51,460 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.113e+01 8.136e+01 8.962e+01 9.902e+01 1.387e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-20 18:50:55,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1179000.0, ans=0.125
2023-11-20 18:51:00,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1179000.0, ans=0.125
2023-11-20 18:51:17,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1179133.3333333333, ans=0.0
2023-11-20 18:51:21,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1179133.3333333333, ans=0.125
2023-11-20 18:51:24,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1179133.3333333333, ans=0.09899494936611666
2023-11-20 18:51:28,700 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8550, loss[loss=0.0948, simple_loss=0.1208, pruned_loss=0.02396, audio_tagging_loss=0.01045, over 16145.00 frames. ], tot_loss[loss=0.07869, simple_loss=0.1002, pruned_loss=0.01881, audio_tagging_loss=0.009769, over 3050067.87 frames. ], batch size: 58, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:51:51,433 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176900
2023-11-20 18:52:00,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1179333.3333333333, ans=0.125
2023-11-20 18:52:03,057 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.01 vs. limit=22.5
2023-11-20 18:52:07,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1179400.0, ans=0.1
2023-11-20 18:52:20,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1179466.6666666667, ans=10.0
2023-11-20 18:52:21,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1179466.6666666667, ans=0.125
2023-11-20 18:52:22,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1179466.6666666667, ans=0.125
2023-11-20 18:52:32,729 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8600, loss[loss=0.06656, simple_loss=0.09247, pruned_loss=0.01311, audio_tagging_loss=0.007215, over 15509.00 frames. ], tot_loss[loss=0.07816, simple_loss=0.09942, pruned_loss=0.01862, audio_tagging_loss=0.009838, over 3053194.86 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:52:56,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 176950
2023-11-20 18:53:00,852 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.645e+01 8.095e+01 8.772e+01 9.480e+01 1.996e+02, threshold=1.754e+02, percent-clipped=1.0
2023-11-20 18:53:02,459 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 18:53:05,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1179666.6666666667, ans=0.125
2023-11-20 18:53:12,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1179733.3333333333, ans=15.0
2023-11-20 18:53:33,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=1179800.0, ans=15.0
2023-11-20 18:53:37,686 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8650, loss[loss=0.08903, simple_loss=0.1124, pruned_loss=0.0197, audio_tagging_loss=0.01315, over 16125.00 frames. ], tot_loss[loss=0.07789, simple_loss=0.09902, pruned_loss=0.01843, audio_tagging_loss=0.009954, over 3046433.87 frames. ], batch size: 60, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:53:55,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1179933.3333333333, ans=0.1
2023-11-20 18:53:57,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1179933.3333333333, ans=0.125
2023-11-20 18:54:01,285 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177000
2023-11-20 18:54:10,854 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.84 vs. limit=6.0
2023-11-20 18:54:13,453 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.50 vs. limit=12.0
2023-11-20 18:54:16,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1180066.6666666667, ans=0.125
2023-11-20 18:54:43,964 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8700, loss[loss=0.1118, simple_loss=0.1446, pruned_loss=0.03089, audio_tagging_loss=0.008615, over 14776.00 frames. ], tot_loss[loss=0.07787, simple_loss=0.0988, pruned_loss=0.01837, audio_tagging_loss=0.01009, over 3046999.82 frames. ], batch size: 53, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:54:49,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1180200.0, ans=0.0
2023-11-20 18:54:53,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1180200.0, ans=0.1
2023-11-20 18:55:06,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177050
2023-11-20 18:55:07,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1180333.3333333333, ans=0.1
2023-11-20 18:55:09,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1180333.3333333333, ans=0.2
2023-11-20 18:55:10,012 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.109e+01 8.720e+01 9.588e+01 1.618e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-20 18:55:27,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1180400.0, ans=0.1
2023-11-20 18:55:31,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1180400.0, ans=0.1
2023-11-20 18:55:48,052 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8750, loss[loss=0.09468, simple_loss=0.1166, pruned_loss=0.02386, audio_tagging_loss=0.01254, over 14736.00 frames. ], tot_loss[loss=0.07786, simple_loss=0.09858, pruned_loss=0.01837, audio_tagging_loss=0.0102, over 3054350.37 frames. ], batch size: 54, lr: 4.56e-03, grad_scale: 16.0
2023-11-20 18:55:59,594 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.27 vs. limit=15.0
2023-11-20 18:56:11,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177100
2023-11-20 18:56:11,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1180600.0, ans=0.1
2023-11-20 18:56:15,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.02 vs. limit=12.0
2023-11-20 18:56:19,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.77 vs. limit=22.5
2023-11-20 18:56:25,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.53 vs. limit=15.0
2023-11-20 18:56:34,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1180733.3333333333, ans=0.125
2023-11-20 18:56:35,016 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.97 vs. limit=15.0
2023-11-20 18:56:42,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1180800.0, ans=0.0
2023-11-20 18:56:52,266 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8800, loss[loss=0.09335, simple_loss=0.1205, pruned_loss=0.02392, audio_tagging_loss=0.009198, over 15976.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.09907, pruned_loss=0.01851, audio_tagging_loss=0.01017, over 3054448.49 frames. ], batch size: 60, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:56:53,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1180866.6666666667, ans=0.125
2023-11-20 18:57:05,611 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.26 vs. limit=6.0
2023-11-20 18:57:16,478 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177150
2023-11-20 18:57:21,355 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.135e+01 8.370e+01 9.044e+01 9.709e+01 1.257e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-20 18:57:24,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1181000.0, ans=0.125
2023-11-20 18:57:39,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1181066.6666666667, ans=0.125
2023-11-20 18:57:39,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1181066.6666666667, ans=0.5
2023-11-20 18:57:39,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.13 vs. limit=15.0
2023-11-20 18:57:45,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1181133.3333333333, ans=0.2
2023-11-20 18:57:55,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1181133.3333333333, ans=0.2
2023-11-20 18:57:58,612 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8850, loss[loss=0.07983, simple_loss=0.1074, pruned_loss=0.01594, audio_tagging_loss=0.01022, over 15293.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.0999, pruned_loss=0.01857, audio_tagging_loss=0.01009, over 3062975.83 frames. ], batch size: 57, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:58:02,956 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.92 vs. limit=15.0
2023-11-20 18:58:11,000 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 18:58:11,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1181266.6666666667, ans=0.125
2023-11-20 18:58:19,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1181266.6666666667, ans=0.09899494936611666
2023-11-20 18:58:20,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177200
2023-11-20 18:58:53,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1181466.6666666667, ans=0.125
2023-11-20 18:59:01,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1181466.6666666667, ans=0.125
2023-11-20 18:59:03,549 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8900, loss[loss=0.08152, simple_loss=0.09504, pruned_loss=0.023, audio_tagging_loss=0.011, over 14495.00 frames. ], tot_loss[loss=0.07805, simple_loss=0.09938, pruned_loss=0.01847, audio_tagging_loss=0.009896, over 3060957.23 frames. ], batch size: 57, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 18:59:03,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1181533.3333333333, ans=0.125
2023-11-20 18:59:27,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177250
2023-11-20 18:59:32,821 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.317e+01 8.861e+01 9.855e+01 1.345e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-20 18:59:35,011 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.20 vs. limit=15.0
2023-11-20 18:59:55,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1181800.0, ans=0.1
2023-11-20 18:59:59,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1181800.0, ans=0.125
2023-11-20 19:00:08,817 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 8950, loss[loss=0.06892, simple_loss=0.08937, pruned_loss=0.01425, audio_tagging_loss=0.009983, over 14534.00 frames. ], tot_loss[loss=0.0786, simple_loss=0.1002, pruned_loss=0.01882, audio_tagging_loss=0.009683, over 3053618.66 frames. ], batch size: 56, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 19:00:27,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1181933.3333333333, ans=0.1
2023-11-20 19:00:33,300 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177300
2023-11-20 19:01:00,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1182133.3333333333, ans=0.2
2023-11-20 19:01:13,866 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9000, loss[loss=0.04471, simple_loss=0.04374, pruned_loss=0.009403, audio_tagging_loss=0.01344, over 16478.00 frames. ], tot_loss[loss=0.07742, simple_loss=0.09864, pruned_loss=0.01831, audio_tagging_loss=0.009792, over 3054589.60 frames. ], batch size: 67, lr: 4.56e-03, grad_scale: 32.0
2023-11-20 19:01:13,869 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-20 19:01:56,964 INFO [train_asr.py:1253] (0/4) Epoch 15, validation: loss=0.0619, simple_loss=0.05318, pruned_loss=0.005552, audio_tagging_loss=0.02975, over 4681554.00 frames. 
2023-11-20 19:01:56,964 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-20 19:02:18,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1182266.6666666667, ans=0.0
2023-11-20 19:02:20,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177350
2023-11-20 19:02:25,240 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.224e+01 8.889e+01 9.368e+01 1.234e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-20 19:02:31,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1182333.3333333333, ans=0.125
2023-11-20 19:02:35,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1182400.0, ans=0.5
2023-11-20 19:02:40,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1182400.0, ans=0.07
2023-11-20 19:02:47,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1182466.6666666667, ans=0.2
2023-11-20 19:02:48,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1182466.6666666667, ans=0.0
2023-11-20 19:02:57,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1182466.6666666667, ans=0.1
2023-11-20 19:03:02,032 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9050, loss[loss=0.07236, simple_loss=0.097, pruned_loss=0.01729, audio_tagging_loss=0.006569, over 15025.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09824, pruned_loss=0.0183, audio_tagging_loss=0.009791, over 3052657.61 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:03:03,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1182533.3333333333, ans=0.1
2023-11-20 19:03:25,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177400
2023-11-20 19:04:07,297 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9100, loss[loss=0.05546, simple_loss=0.07018, pruned_loss=0.01104, audio_tagging_loss=0.00933, over 15209.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09892, pruned_loss=0.01853, audio_tagging_loss=0.009716, over 3054509.75 frames. ], batch size: 58, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:04:30,230 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177450
2023-11-20 19:04:34,929 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 8.170e+01 8.840e+01 9.659e+01 1.289e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 19:04:52,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1183066.6666666667, ans=0.1
2023-11-20 19:05:12,057 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9150, loss[loss=0.0684, simple_loss=0.08806, pruned_loss=0.01273, audio_tagging_loss=0.01164, over 14648.00 frames. ], tot_loss[loss=0.07809, simple_loss=0.09959, pruned_loss=0.01867, audio_tagging_loss=0.009628, over 3056566.06 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:05:16,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1183200.0, ans=0.125
2023-11-20 19:05:20,421 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.79 vs. limit=8.0
2023-11-20 19:05:35,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177500
2023-11-20 19:05:37,372 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.93 vs. limit=15.0
2023-11-20 19:05:46,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1183333.3333333333, ans=0.0
2023-11-20 19:06:03,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1183466.6666666667, ans=0.125
2023-11-20 19:06:15,581 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9200, loss[loss=0.05851, simple_loss=0.07589, pruned_loss=0.01291, audio_tagging_loss=0.007657, over 14600.00 frames. ], tot_loss[loss=0.07753, simple_loss=0.09899, pruned_loss=0.01849, audio_tagging_loss=0.009552, over 3050825.26 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:06:39,239 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177550
2023-11-20 19:06:44,026 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.006e+01 8.354e+01 8.939e+01 9.701e+01 1.171e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-20 19:06:57,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1183733.3333333333, ans=0.125
2023-11-20 19:07:06,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1183800.0, ans=0.1
2023-11-20 19:07:12,307 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.84 vs. limit=15.0
2023-11-20 19:07:18,342 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.28 vs. limit=22.5
2023-11-20 19:07:18,543 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.63 vs. limit=10.0
2023-11-20 19:07:20,268 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9250, loss[loss=0.07288, simple_loss=0.08316, pruned_loss=0.01971, audio_tagging_loss=0.01159, over 15897.00 frames. ], tot_loss[loss=0.07693, simple_loss=0.09798, pruned_loss=0.01828, audio_tagging_loss=0.009657, over 3046715.38 frames. ], batch size: 61, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:07:23,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1183866.6666666667, ans=0.1
2023-11-20 19:07:36,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1183933.3333333333, ans=0.1
2023-11-20 19:07:43,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177600
2023-11-20 19:07:43,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1183933.3333333333, ans=0.125
2023-11-20 19:07:57,680 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:08:02,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1184066.6666666667, ans=0.0
2023-11-20 19:08:07,720 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.58 vs. limit=6.0
2023-11-20 19:08:16,837 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.47 vs. limit=15.0
2023-11-20 19:08:24,601 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9300, loss[loss=0.07547, simple_loss=0.09361, pruned_loss=0.01787, audio_tagging_loss=0.0108, over 16777.00 frames. ], tot_loss[loss=0.07652, simple_loss=0.09729, pruned_loss=0.01817, audio_tagging_loss=0.009704, over 3055764.64 frames. ], batch size: 63, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:08:32,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1184200.0, ans=22.5
2023-11-20 19:08:36,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1184266.6666666667, ans=0.2
2023-11-20 19:08:40,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.52 vs. limit=15.0
2023-11-20 19:08:48,437 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177650
2023-11-20 19:08:53,148 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.542e+01 8.090e+01 8.777e+01 9.713e+01 1.132e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-20 19:09:12,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1184400.0, ans=0.125
2023-11-20 19:09:27,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1184533.3333333333, ans=0.0
2023-11-20 19:09:28,671 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9350, loss[loss=0.08676, simple_loss=0.1122, pruned_loss=0.02142, audio_tagging_loss=0.009262, over 15642.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.0969, pruned_loss=0.01823, audio_tagging_loss=0.009744, over 3053787.24 frames. ], batch size: 59, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:09:41,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1184600.0, ans=0.125
2023-11-20 19:09:46,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1184600.0, ans=0.125
2023-11-20 19:09:52,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177700
2023-11-20 19:10:03,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1184666.6666666667, ans=0.125
2023-11-20 19:10:06,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1184733.3333333333, ans=0.125
2023-11-20 19:10:12,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1184733.3333333333, ans=0.0
2023-11-20 19:10:33,266 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9400, loss[loss=0.07453, simple_loss=0.08104, pruned_loss=0.02078, audio_tagging_loss=0.01323, over 13543.00 frames. ], tot_loss[loss=0.0769, simple_loss=0.09731, pruned_loss=0.01844, audio_tagging_loss=0.009805, over 3048277.27 frames. ], batch size: 55, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:10:43,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1184866.6666666667, ans=0.125
2023-11-20 19:10:45,607 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.16 vs. limit=15.0
2023-11-20 19:10:56,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177750
2023-11-20 19:10:57,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1185000.0, ans=0.125
2023-11-20 19:11:01,384 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.184e+01 8.847e+01 9.642e+01 1.225e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-20 19:11:07,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1185000.0, ans=0.07
2023-11-20 19:11:20,176 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.37 vs. limit=15.0
2023-11-20 19:11:25,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1185133.3333333333, ans=0.125
2023-11-20 19:11:28,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1185133.3333333333, ans=0.2
2023-11-20 19:11:31,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1185133.3333333333, ans=0.125
2023-11-20 19:11:31,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1185133.3333333333, ans=0.1
2023-11-20 19:11:35,923 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:11:37,129 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9450, loss[loss=0.0773, simple_loss=0.1019, pruned_loss=0.01689, audio_tagging_loss=0.009452, over 14991.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.09732, pruned_loss=0.01823, audio_tagging_loss=0.00988, over 3050761.68 frames. ], batch size: 57, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:11:42,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1185200.0, ans=0.0
2023-11-20 19:11:54,210 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.93 vs. limit=22.5
2023-11-20 19:12:00,837 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177800
2023-11-20 19:12:07,048 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=15.31 vs. limit=15.0
2023-11-20 19:12:21,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1185400.0, ans=0.125
2023-11-20 19:12:36,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1185466.6666666667, ans=0.1
2023-11-20 19:12:41,817 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9500, loss[loss=0.07161, simple_loss=0.09321, pruned_loss=0.01445, audio_tagging_loss=0.01056, over 14247.00 frames. ], tot_loss[loss=0.07752, simple_loss=0.09807, pruned_loss=0.01858, audio_tagging_loss=0.009898, over 3052066.68 frames. ], batch size: 54, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:12:51,030 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.52 vs. limit=15.0
2023-11-20 19:12:56,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1185600.0, ans=0.125
2023-11-20 19:13:01,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1185600.0, ans=0.1
2023-11-20 19:13:05,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.90 vs. limit=12.0
2023-11-20 19:13:05,878 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177850
2023-11-20 19:13:10,714 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.422e+01 8.141e+01 8.754e+01 9.457e+01 1.227e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-20 19:13:12,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1185666.6666666667, ans=0.125
2023-11-20 19:13:23,729 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.63 vs. limit=15.0
2023-11-20 19:13:43,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1185800.0, ans=0.0
2023-11-20 19:13:47,315 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9550, loss[loss=0.09386, simple_loss=0.1207, pruned_loss=0.02266, audio_tagging_loss=0.01083, over 14740.00 frames. ], tot_loss[loss=0.07781, simple_loss=0.09839, pruned_loss=0.0186, audio_tagging_loss=0.01001, over 3053038.87 frames. ], batch size: 53, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:13:55,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1185866.6666666667, ans=0.0
2023-11-20 19:14:10,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177900
2023-11-20 19:14:16,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1186000.0, ans=0.0
2023-11-20 19:14:23,988 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.79 vs. limit=15.0
2023-11-20 19:14:29,660 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:14:35,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1186066.6666666667, ans=0.0
2023-11-20 19:14:51,328 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.15 vs. limit=12.0
2023-11-20 19:14:51,989 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9600, loss[loss=0.06631, simple_loss=0.0783, pruned_loss=0.0143, audio_tagging_loss=0.01286, over 15849.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.0981, pruned_loss=0.01851, audio_tagging_loss=0.01018, over 3055536.86 frames. ], batch size: 59, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:15:08,841 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.59 vs. limit=12.0
2023-11-20 19:15:15,739 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 177950
2023-11-20 19:15:20,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1186333.3333333333, ans=0.125
2023-11-20 19:15:22,424 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.101e+01 8.290e+01 9.064e+01 1.000e+02 1.775e+02, threshold=1.813e+02, percent-clipped=1.0
2023-11-20 19:15:40,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1186400.0, ans=0.1
2023-11-20 19:15:56,298 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9650, loss[loss=0.05992, simple_loss=0.075, pruned_loss=0.01335, audio_tagging_loss=0.009064, over 15128.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09841, pruned_loss=0.01856, audio_tagging_loss=0.009959, over 3046168.05 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:15:56,919 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=15.0
2023-11-20 19:16:20,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178000
2023-11-20 19:17:02,201 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9700, loss[loss=0.06771, simple_loss=0.08609, pruned_loss=0.01416, audio_tagging_loss=0.0105, over 14894.00 frames. ], tot_loss[loss=0.07815, simple_loss=0.09922, pruned_loss=0.01875, audio_tagging_loss=0.009788, over 3047125.57 frames. ], batch size: 56, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:17:08,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1186866.6666666667, ans=0.0
2023-11-20 19:17:11,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1186866.6666666667, ans=0.125
2023-11-20 19:17:12,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1186866.6666666667, ans=0.125
2023-11-20 19:17:25,277 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178050
2023-11-20 19:17:31,281 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.180e+01 8.975e+01 9.665e+01 1.279e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-20 19:17:37,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1187000.0, ans=0.125
2023-11-20 19:17:43,701 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.85 vs. limit=6.0
2023-11-20 19:17:50,433 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.00 vs. limit=22.5
2023-11-20 19:17:52,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1187133.3333333333, ans=0.125
2023-11-20 19:17:59,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1187133.3333333333, ans=0.0
2023-11-20 19:18:06,473 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9750, loss[loss=0.06199, simple_loss=0.07961, pruned_loss=0.01231, audio_tagging_loss=0.00987, over 15342.00 frames. ], tot_loss[loss=0.07777, simple_loss=0.09894, pruned_loss=0.01854, audio_tagging_loss=0.009764, over 3044040.67 frames. ], batch size: 59, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:18:15,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1187200.0, ans=0.125
2023-11-20 19:18:28,251 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178100
2023-11-20 19:18:38,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1187333.3333333333, ans=0.025
2023-11-20 19:18:41,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1187333.3333333333, ans=0.125
2023-11-20 19:18:57,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1187466.6666666667, ans=0.1
2023-11-20 19:18:58,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1187466.6666666667, ans=0.125
2023-11-20 19:19:09,562 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9800, loss[loss=0.07358, simple_loss=0.09471, pruned_loss=0.01757, audio_tagging_loss=0.00866, over 14858.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.09931, pruned_loss=0.01853, audio_tagging_loss=0.009619, over 3047285.15 frames. ], batch size: 55, lr: 4.55e-03, grad_scale: 32.0
2023-11-20 19:19:14,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1187533.3333333333, ans=0.0
2023-11-20 19:19:33,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178150
2023-11-20 19:19:39,611 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.101e+01 8.620e+01 9.316e+01 1.225e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-20 19:20:05,631 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:20:14,305 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9850, loss[loss=0.09198, simple_loss=0.1241, pruned_loss=0.02002, audio_tagging_loss=0.00991, over 15847.00 frames. ], tot_loss[loss=0.07815, simple_loss=0.09986, pruned_loss=0.01865, audio_tagging_loss=0.00957, over 3053684.82 frames. ], batch size: 56, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:20:18,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1187866.6666666667, ans=0.1
2023-11-20 19:20:37,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178200
2023-11-20 19:20:40,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1188000.0, ans=0.95
2023-11-20 19:20:53,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1188066.6666666667, ans=0.125
2023-11-20 19:21:06,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1188133.3333333333, ans=0.125
2023-11-20 19:21:19,007 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9900, loss[loss=0.08109, simple_loss=0.1024, pruned_loss=0.01823, audio_tagging_loss=0.01164, over 15465.00 frames. ], tot_loss[loss=0.07826, simple_loss=0.0998, pruned_loss=0.0187, audio_tagging_loss=0.009655, over 3049029.70 frames. ], batch size: 61, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:21:29,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1188200.0, ans=0.125
2023-11-20 19:21:33,308 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.76 vs. limit=15.0
2023-11-20 19:21:41,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178250
2023-11-20 19:21:47,811 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.012e+01 8.821e+01 9.655e+01 1.193e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-20 19:21:52,750 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.27 vs. limit=15.0
2023-11-20 19:22:15,026 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.34 vs. limit=22.5
2023-11-20 19:22:22,901 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 9950, loss[loss=0.09198, simple_loss=0.1152, pruned_loss=0.02499, audio_tagging_loss=0.009387, over 15227.00 frames. ], tot_loss[loss=0.07848, simple_loss=0.1001, pruned_loss=0.01879, audio_tagging_loss=0.009637, over 3047708.11 frames. ], batch size: 54, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:22:45,965 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178300
2023-11-20 19:22:58,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1188666.6666666667, ans=0.2
2023-11-20 19:23:04,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1188733.3333333333, ans=0.0
2023-11-20 19:23:09,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1188733.3333333333, ans=0.125
2023-11-20 19:23:27,675 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10000, loss[loss=0.07721, simple_loss=0.09754, pruned_loss=0.01866, audio_tagging_loss=0.009784, over 15016.00 frames. ], tot_loss[loss=0.07784, simple_loss=0.09923, pruned_loss=0.01855, audio_tagging_loss=0.00968, over 3044103.54 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:23:38,420 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.29 vs. limit=10.0
2023-11-20 19:23:38,808 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.05 vs. limit=15.0
2023-11-20 19:23:50,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178350
2023-11-20 19:23:51,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1189000.0, ans=0.125
2023-11-20 19:23:52,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1189000.0, ans=0.0
2023-11-20 19:23:56,457 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 7.984e+01 8.693e+01 9.319e+01 1.323e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-20 19:24:03,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn1.whiten.whitening_limit, batch_count=1189000.0, ans=22.5
2023-11-20 19:24:12,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1189066.6666666667, ans=0.1
2023-11-20 19:24:17,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1189133.3333333333, ans=0.125
2023-11-20 19:24:21,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1189133.3333333333, ans=0.0
2023-11-20 19:24:22,844 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.84 vs. limit=10.0
2023-11-20 19:24:31,285 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10050, loss[loss=0.08645, simple_loss=0.1088, pruned_loss=0.024, audio_tagging_loss=0.008027, over 15677.00 frames. ], tot_loss[loss=0.07787, simple_loss=0.09905, pruned_loss=0.01859, audio_tagging_loss=0.009755, over 3048698.28 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:24:37,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1189200.0, ans=0.125
2023-11-20 19:24:45,269 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.85 vs. limit=22.5
2023-11-20 19:24:48,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1189266.6666666667, ans=0.125
2023-11-20 19:24:53,365 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178400
2023-11-20 19:24:53,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1189266.6666666667, ans=0.2
2023-11-20 19:25:11,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1189400.0, ans=0.125
2023-11-20 19:25:11,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1189400.0, ans=0.125
2023-11-20 19:25:12,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1189400.0, ans=0.015
2023-11-20 19:25:22,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1189466.6666666667, ans=0.1
2023-11-20 19:25:35,272 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10100, loss[loss=0.05532, simple_loss=0.05999, pruned_loss=0.01189, audio_tagging_loss=0.01344, over 15133.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.09882, pruned_loss=0.0186, audio_tagging_loss=0.009871, over 3050683.34 frames. ], batch size: 59, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:25:37,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1189533.3333333333, ans=0.0
2023-11-20 19:25:57,374 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:25:58,383 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178450
2023-11-20 19:26:04,395 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.521e+01 8.112e+01 8.895e+01 9.698e+01 1.490e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-20 19:26:08,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1189666.6666666667, ans=0.125
2023-11-20 19:26:08,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1189666.6666666667, ans=0.125
2023-11-20 19:26:19,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1189733.3333333333, ans=0.125
2023-11-20 19:26:26,341 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:26:35,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1189800.0, ans=0.05
2023-11-20 19:26:38,694 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10150, loss[loss=0.06949, simple_loss=0.0834, pruned_loss=0.01844, audio_tagging_loss=0.009349, over 15139.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09831, pruned_loss=0.01853, audio_tagging_loss=0.009901, over 3048885.28 frames. ], batch size: 56, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:26:47,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1189866.6666666667, ans=0.125
2023-11-20 19:27:03,193 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178500
2023-11-20 19:27:03,330 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:27:09,275 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:27:20,934 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.19 vs. limit=15.0
2023-11-20 19:27:40,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=1190133.3333333333, ans=10.0
2023-11-20 19:27:43,571 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10200, loss[loss=0.07934, simple_loss=0.09675, pruned_loss=0.01989, audio_tagging_loss=0.01108, over 15353.00 frames. ], tot_loss[loss=0.07741, simple_loss=0.09796, pruned_loss=0.01847, audio_tagging_loss=0.009959, over 3054112.58 frames. ], batch size: 59, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:28:02,689 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.55 vs. limit=6.0
2023-11-20 19:28:07,053 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178550
2023-11-20 19:28:08,205 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:28:13,018 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.453e+01 8.265e+01 8.993e+01 9.730e+01 1.182e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-20 19:28:37,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1190466.6666666667, ans=0.125
2023-11-20 19:28:41,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1190466.6666666667, ans=0.05
2023-11-20 19:28:43,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1190466.6666666667, ans=0.0
2023-11-20 19:28:48,086 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10250, loss[loss=0.07133, simple_loss=0.0869, pruned_loss=0.01655, audio_tagging_loss=0.01133, over 15146.00 frames. ], tot_loss[loss=0.07744, simple_loss=0.09809, pruned_loss=0.01836, audio_tagging_loss=0.01004, over 3053422.66 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:29:00,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1190600.0, ans=0.125
2023-11-20 19:29:11,272 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178600
2023-11-20 19:29:15,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1190666.6666666667, ans=0.025
2023-11-20 19:29:20,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1190666.6666666667, ans=0.0
2023-11-20 19:29:27,289 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.03 vs. limit=15.0
2023-11-20 19:29:33,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1190733.3333333333, ans=0.125
2023-11-20 19:29:49,875 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.67 vs. limit=6.0
2023-11-20 19:29:51,527 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10300, loss[loss=0.08779, simple_loss=0.1244, pruned_loss=0.01946, audio_tagging_loss=0.006125, over 15553.00 frames. ], tot_loss[loss=0.07802, simple_loss=0.09888, pruned_loss=0.01857, audio_tagging_loss=0.01001, over 3053415.80 frames. ], batch size: 55, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:30:09,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1190933.3333333333, ans=0.2
2023-11-20 19:30:11,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1190933.3333333333, ans=0.125
2023-11-20 19:30:15,389 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178650
2023-11-20 19:30:21,239 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 7.948e+01 8.606e+01 9.117e+01 1.225e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-20 19:30:42,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1191133.3333333333, ans=0.1
2023-11-20 19:30:51,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1191133.3333333333, ans=0.125
2023-11-20 19:30:55,999 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10350, loss[loss=0.08324, simple_loss=0.09526, pruned_loss=0.02069, audio_tagging_loss=0.01492, over 15583.00 frames. ], tot_loss[loss=0.07845, simple_loss=0.09918, pruned_loss=0.01867, audio_tagging_loss=0.01019, over 3047292.28 frames. ], batch size: 57, lr: 4.54e-03, grad_scale: 16.0
2023-11-20 19:31:13,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1191266.6666666667, ans=0.125
2023-11-20 19:31:13,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1191266.6666666667, ans=0.125
2023-11-20 19:31:19,284 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178700
2023-11-20 19:31:20,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1191333.3333333333, ans=0.125
2023-11-20 19:31:23,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1191333.3333333333, ans=0.0
2023-11-20 19:31:46,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1191466.6666666667, ans=0.125
2023-11-20 19:31:52,887 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.35 vs. limit=6.0
2023-11-20 19:31:54,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1191466.6666666667, ans=0.125
2023-11-20 19:31:58,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1191533.3333333333, ans=0.2
2023-11-20 19:31:59,607 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10400, loss[loss=0.07499, simple_loss=0.09115, pruned_loss=0.01504, audio_tagging_loss=0.01437, over 16020.00 frames. ], tot_loss[loss=0.07848, simple_loss=0.09907, pruned_loss=0.01871, audio_tagging_loss=0.01024, over 3054778.65 frames. ], batch size: 59, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:32:06,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1191533.3333333333, ans=0.2
2023-11-20 19:32:19,719 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:32:23,145 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178750
2023-11-20 19:32:30,367 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.343e+01 9.051e+01 9.727e+01 1.230e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-20 19:32:44,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1191733.3333333333, ans=0.125
2023-11-20 19:32:45,253 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.59 vs. limit=15.0
2023-11-20 19:32:49,002 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.80 vs. limit=15.0
2023-11-20 19:32:51,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1191800.0, ans=0.025
2023-11-20 19:33:03,409 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10450, loss[loss=0.09489, simple_loss=0.1269, pruned_loss=0.02667, audio_tagging_loss=0.00479, over 14911.00 frames. ], tot_loss[loss=0.07791, simple_loss=0.09829, pruned_loss=0.01863, audio_tagging_loss=0.01014, over 3046159.48 frames. ], batch size: 54, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:33:04,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1191866.6666666667, ans=0.1
2023-11-20 19:33:27,068 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178800
2023-11-20 19:33:27,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1191933.3333333333, ans=0.1
2023-11-20 19:34:08,053 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10500, loss[loss=0.04538, simple_loss=0.06275, pruned_loss=0.005669, audio_tagging_loss=0.008332, over 14978.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.09758, pruned_loss=0.01832, audio_tagging_loss=0.009968, over 3051304.20 frames. ], batch size: 61, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:34:08,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1192200.0, ans=0.125
2023-11-20 19:34:19,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1192266.6666666667, ans=0.125
2023-11-20 19:34:30,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178850
2023-11-20 19:34:30,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1192266.6666666667, ans=0.0
2023-11-20 19:34:37,990 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.449e+01 8.858e+01 9.853e+01 1.185e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-20 19:34:45,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1192400.0, ans=0.125
2023-11-20 19:34:47,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1192400.0, ans=0.0
2023-11-20 19:34:50,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1192400.0, ans=0.025
2023-11-20 19:34:52,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1192400.0, ans=0.0
2023-11-20 19:34:55,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1192400.0, ans=0.09899494936611666
2023-11-20 19:35:01,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1192466.6666666667, ans=0.1
2023-11-20 19:35:11,109 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10550, loss[loss=0.07955, simple_loss=0.1106, pruned_loss=0.01684, audio_tagging_loss=0.007433, over 15699.00 frames. ], tot_loss[loss=0.07725, simple_loss=0.09796, pruned_loss=0.01846, audio_tagging_loss=0.009804, over 3052091.43 frames. ], batch size: 58, lr: 4.54e-03, grad_scale: 32.0
2023-11-20 19:35:15,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1192533.3333333333, ans=0.1
2023-11-20 19:35:33,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178900
2023-11-20 19:35:45,952 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 19:36:14,462 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10600, loss[loss=0.06961, simple_loss=0.08524, pruned_loss=0.01505, audio_tagging_loss=0.01194, over 15105.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09753, pruned_loss=0.01828, audio_tagging_loss=0.009775, over 3043978.61 frames. ], batch size: 56, lr: 4.54e-03, grad_scale: 16.0
2023-11-20 19:36:37,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1192933.3333333333, ans=0.125
2023-11-20 19:36:38,429 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 178950
2023-11-20 19:36:42,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1193000.0, ans=0.125
2023-11-20 19:36:46,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1193000.0, ans=0.125
2023-11-20 19:36:46,940 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.904e+01 8.146e+01 8.630e+01 9.635e+01 1.232e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-20 19:36:47,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1193000.0, ans=0.0
2023-11-20 19:36:49,746 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.305e-01
2023-11-20 19:37:18,817 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10650, loss[loss=0.06102, simple_loss=0.07908, pruned_loss=0.01307, audio_tagging_loss=0.008408, over 16376.00 frames. ], tot_loss[loss=0.07694, simple_loss=0.09807, pruned_loss=0.0183, audio_tagging_loss=0.009615, over 3051092.04 frames. ], batch size: 65, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:37:41,210 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179000
2023-11-20 19:37:42,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1193333.3333333333, ans=0.0
2023-11-20 19:37:46,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1193333.3333333333, ans=0.1
2023-11-20 19:38:09,166 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.91 vs. limit=15.0
2023-11-20 19:38:12,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1193466.6666666667, ans=0.0
2023-11-20 19:38:21,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=1193533.3333333333, ans=0.02
2023-11-20 19:38:22,587 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10700, loss[loss=0.05255, simple_loss=0.06293, pruned_loss=0.009626, audio_tagging_loss=0.01146, over 15038.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09884, pruned_loss=0.01857, audio_tagging_loss=0.009639, over 3046906.81 frames. ], batch size: 58, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:38:23,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1193533.3333333333, ans=0.2
2023-11-20 19:38:25,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1193533.3333333333, ans=0.125
2023-11-20 19:38:27,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1193533.3333333333, ans=0.2
2023-11-20 19:38:29,328 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.02 vs. limit=22.5
2023-11-20 19:38:43,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1193600.0, ans=0.125
2023-11-20 19:38:45,349 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179050
2023-11-20 19:38:54,230 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.026e+01 8.604e+01 9.471e+01 1.335e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-20 19:38:54,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1193666.6666666667, ans=0.1
2023-11-20 19:39:25,052 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10750, loss[loss=0.08222, simple_loss=0.1007, pruned_loss=0.02107, audio_tagging_loss=0.01079, over 15307.00 frames. ], tot_loss[loss=0.07821, simple_loss=0.09944, pruned_loss=0.01881, audio_tagging_loss=0.009679, over 3051337.22 frames. ], batch size: 57, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:39:41,205 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.33 vs. limit=22.5
2023-11-20 19:39:47,886 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179100
2023-11-20 19:40:00,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1194000.0, ans=0.125
2023-11-20 19:40:03,491 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.62 vs. limit=15.0
2023-11-20 19:40:28,475 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10800, loss[loss=0.0933, simple_loss=0.1296, pruned_loss=0.01979, audio_tagging_loss=0.008715, over 15730.00 frames. ], tot_loss[loss=0.07919, simple_loss=0.101, pruned_loss=0.01907, audio_tagging_loss=0.009607, over 3049068.91 frames. ], batch size: 58, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:40:33,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1194200.0, ans=0.125
2023-11-20 19:40:50,827 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179150
2023-11-20 19:40:58,464 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.46 vs. limit=15.0
2023-11-20 19:41:00,343 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 8.337e+01 8.834e+01 9.923e+01 1.321e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-20 19:41:25,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1194466.6666666667, ans=0.0
2023-11-20 19:41:30,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1194533.3333333333, ans=10.0
2023-11-20 19:41:31,485 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10850, loss[loss=0.0681, simple_loss=0.08352, pruned_loss=0.01484, audio_tagging_loss=0.0115, over 17015.00 frames. ], tot_loss[loss=0.07847, simple_loss=0.09996, pruned_loss=0.01893, audio_tagging_loss=0.009566, over 3043354.61 frames. ], batch size: 66, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:41:53,639 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179200
2023-11-20 19:41:56,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1194666.6666666667, ans=0.0
2023-11-20 19:41:58,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1194666.6666666667, ans=0.0
2023-11-20 19:42:06,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1194666.6666666667, ans=0.125
2023-11-20 19:42:30,668 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:42:34,256 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10900, loss[loss=0.07389, simple_loss=0.09542, pruned_loss=0.01638, audio_tagging_loss=0.009801, over 15107.00 frames. ], tot_loss[loss=0.07818, simple_loss=0.09948, pruned_loss=0.01875, audio_tagging_loss=0.009699, over 3045432.79 frames. ], batch size: 57, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:42:34,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1194866.6666666667, ans=0.125
2023-11-20 19:42:34,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1194866.6666666667, ans=0.0
2023-11-20 19:42:40,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1194866.6666666667, ans=0.125
2023-11-20 19:42:48,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1194933.3333333333, ans=0.0
2023-11-20 19:42:58,109 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179250
2023-11-20 19:43:08,257 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.081e+01 8.823e+01 9.357e+01 1.279e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-20 19:43:19,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1195066.6666666667, ans=0.1
2023-11-20 19:43:34,119 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.39 vs. limit=15.0
2023-11-20 19:43:38,315 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 10950, loss[loss=0.062, simple_loss=0.07728, pruned_loss=0.01267, audio_tagging_loss=0.01069, over 14695.00 frames. ], tot_loss[loss=0.07745, simple_loss=0.09826, pruned_loss=0.01848, audio_tagging_loss=0.009834, over 3040341.96 frames. ], batch size: 56, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:44:01,766 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179300
2023-11-20 19:44:11,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1195333.3333333333, ans=0.0
2023-11-20 19:44:25,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1195400.0, ans=0.2
2023-11-20 19:44:27,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1195400.0, ans=0.1
2023-11-20 19:44:35,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1195466.6666666667, ans=0.125
2023-11-20 19:44:35,754 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.48 vs. limit=22.5
2023-11-20 19:44:42,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1195533.3333333333, ans=0.0
2023-11-20 19:44:42,829 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11000, loss[loss=0.06564, simple_loss=0.0825, pruned_loss=0.01599, audio_tagging_loss=0.00839, over 15986.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.09971, pruned_loss=0.01864, audio_tagging_loss=0.009723, over 3051280.78 frames. ], batch size: 62, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:44:50,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1195533.3333333333, ans=0.0
2023-11-20 19:44:51,294 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:44:55,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1195600.0, ans=0.125
2023-11-20 19:45:04,561 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179350
2023-11-20 19:45:15,404 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.295e+01 8.189e+01 8.881e+01 9.768e+01 1.271e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-20 19:45:15,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1195666.6666666667, ans=0.1
2023-11-20 19:45:23,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1195733.3333333333, ans=0.1
2023-11-20 19:45:26,174 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.89 vs. limit=15.0
2023-11-20 19:45:27,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1195733.3333333333, ans=0.125
2023-11-20 19:45:38,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1195800.0, ans=0.0
2023-11-20 19:45:45,440 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11050, loss[loss=0.08397, simple_loss=0.1132, pruned_loss=0.0196, audio_tagging_loss=0.007789, over 14880.00 frames. ], tot_loss[loss=0.07867, simple_loss=0.1002, pruned_loss=0.01886, audio_tagging_loss=0.009694, over 3045996.52 frames. ], batch size: 54, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:46:07,983 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179400
2023-11-20 19:46:09,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1196000.0, ans=0.2
2023-11-20 19:46:16,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1196000.0, ans=0.0
2023-11-20 19:46:16,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1196000.0, ans=10.0
2023-11-20 19:46:17,365 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.37 vs. limit=15.0
2023-11-20 19:46:25,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1196066.6666666667, ans=0.0
2023-11-20 19:46:47,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1196200.0, ans=0.125
2023-11-20 19:46:48,006 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11100, loss[loss=0.07742, simple_loss=0.1051, pruned_loss=0.01533, audio_tagging_loss=0.009541, over 14591.00 frames. ], tot_loss[loss=0.079, simple_loss=0.1002, pruned_loss=0.01901, audio_tagging_loss=0.009911, over 3052402.58 frames. ], batch size: 53, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:46:51,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1196200.0, ans=0.125
2023-11-20 19:46:59,332 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.39 vs. limit=6.0
2023-11-20 19:47:11,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179450
2023-11-20 19:47:21,550 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.315e+01 8.689e+01 9.550e+01 1.347e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-20 19:47:45,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1196466.6666666667, ans=0.125
2023-11-20 19:47:52,016 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11150, loss[loss=0.07985, simple_loss=0.1008, pruned_loss=0.02094, audio_tagging_loss=0.008523, over 14141.00 frames. ], tot_loss[loss=0.07845, simple_loss=0.09927, pruned_loss=0.0188, audio_tagging_loss=0.01001, over 3050062.82 frames. ], batch size: 56, lr: 4.53e-03, grad_scale: 16.0
2023-11-20 19:47:52,665 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.53 vs. limit=22.5
2023-11-20 19:48:10,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1196600.0, ans=0.125
2023-11-20 19:48:10,740 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.88 vs. limit=15.0
2023-11-20 19:48:13,506 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179500
2023-11-20 19:48:54,084 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11200, loss[loss=0.1071, simple_loss=0.1278, pruned_loss=0.03227, audio_tagging_loss=0.01092, over 15182.00 frames. ], tot_loss[loss=0.07786, simple_loss=0.09849, pruned_loss=0.01846, audio_tagging_loss=0.01015, over 3050367.00 frames. ], batch size: 56, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:49:01,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1196866.6666666667, ans=0.0
2023-11-20 19:49:11,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1196933.3333333333, ans=0.125
2023-11-20 19:49:16,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179550
2023-11-20 19:49:25,609 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.97 vs. limit=10.0
2023-11-20 19:49:26,737 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.291e+01 7.963e+01 8.549e+01 9.018e+01 1.277e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-20 19:49:31,721 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.95 vs. limit=15.0
2023-11-20 19:49:34,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1197066.6666666667, ans=0.125
2023-11-20 19:49:46,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1197133.3333333333, ans=0.125
2023-11-20 19:49:47,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1197133.3333333333, ans=0.125
2023-11-20 19:49:50,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1197133.3333333333, ans=0.125
2023-11-20 19:49:56,611 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11250, loss[loss=0.08596, simple_loss=0.1052, pruned_loss=0.02272, audio_tagging_loss=0.01063, over 15058.00 frames. ], tot_loss[loss=0.07748, simple_loss=0.09757, pruned_loss=0.01846, audio_tagging_loss=0.01023, over 3050318.74 frames. ], batch size: 56, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:50:20,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179600
2023-11-20 19:50:22,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1197333.3333333333, ans=0.0
2023-11-20 19:50:27,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1197333.3333333333, ans=0.0
2023-11-20 19:50:53,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.74 vs. limit=15.0
2023-11-20 19:50:54,412 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.04 vs. limit=15.0
2023-11-20 19:50:55,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1197466.6666666667, ans=0.0
2023-11-20 19:50:58,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1197533.3333333333, ans=0.1
2023-11-20 19:50:59,729 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11300, loss[loss=0.09295, simple_loss=0.1227, pruned_loss=0.02043, audio_tagging_loss=0.01118, over 15089.00 frames. ], tot_loss[loss=0.07804, simple_loss=0.09851, pruned_loss=0.01866, audio_tagging_loss=0.01012, over 3047088.80 frames. ], batch size: 53, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:51:11,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1197533.3333333333, ans=0.125
2023-11-20 19:51:19,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1197600.0, ans=0.125
2023-11-20 19:51:22,958 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179650
2023-11-20 19:51:29,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1197666.6666666667, ans=0.125
2023-11-20 19:51:32,420 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.320e+01 8.034e+01 8.460e+01 9.162e+01 1.233e+02, threshold=1.692e+02, percent-clipped=0.0
2023-11-20 19:51:36,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1197733.3333333333, ans=0.0
2023-11-20 19:51:53,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1197800.0, ans=0.1
2023-11-20 19:52:02,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1197866.6666666667, ans=0.2
2023-11-20 19:52:03,240 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11350, loss[loss=0.05929, simple_loss=0.07229, pruned_loss=0.01463, audio_tagging_loss=0.008519, over 14408.00 frames. ], tot_loss[loss=0.07736, simple_loss=0.09771, pruned_loss=0.01851, audio_tagging_loss=0.009993, over 3042672.82 frames. ], batch size: 57, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:52:04,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1197866.6666666667, ans=0.125
2023-11-20 19:52:25,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179700
2023-11-20 19:52:31,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1198000.0, ans=0.07
2023-11-20 19:52:50,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1198066.6666666667, ans=0.125
2023-11-20 19:53:05,831 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11400, loss[loss=0.08654, simple_loss=0.1065, pruned_loss=0.0237, audio_tagging_loss=0.009614, over 14552.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09771, pruned_loss=0.01838, audio_tagging_loss=0.009808, over 3041984.50 frames. ], batch size: 53, lr: 4.53e-03, grad_scale: 32.0
2023-11-20 19:53:07,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1198200.0, ans=0.1
2023-11-20 19:53:09,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1198200.0, ans=0.125
2023-11-20 19:53:29,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179750
2023-11-20 19:53:38,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1198333.3333333333, ans=0.125
2023-11-20 19:53:39,664 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.357e+01 7.998e+01 8.618e+01 9.401e+01 1.167e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-20 19:53:59,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.39 vs. limit=10.0
2023-11-20 19:54:00,881 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.55 vs. limit=12.0
2023-11-20 19:54:09,903 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11450, loss[loss=0.06697, simple_loss=0.08638, pruned_loss=0.01414, audio_tagging_loss=0.009635, over 15552.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09801, pruned_loss=0.01828, audio_tagging_loss=0.009729, over 3039995.60 frames. ], batch size: 58, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:54:26,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1198600.0, ans=0.07
2023-11-20 19:54:32,723 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179800
2023-11-20 19:54:46,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1198733.3333333333, ans=0.2
2023-11-20 19:54:47,015 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.24 vs. limit=15.0
2023-11-20 19:54:49,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1198733.3333333333, ans=0.0
2023-11-20 19:54:56,756 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.25 vs. limit=6.0
2023-11-20 19:54:58,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1198733.3333333333, ans=0.1
2023-11-20 19:55:13,641 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11500, loss[loss=0.08425, simple_loss=0.1073, pruned_loss=0.02182, audio_tagging_loss=0.0088, over 15527.00 frames. ], tot_loss[loss=0.07756, simple_loss=0.09917, pruned_loss=0.0183, audio_tagging_loss=0.00968, over 3047617.17 frames. ], batch size: 58, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:55:36,221 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179850
2023-11-20 19:55:43,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1199000.0, ans=0.125
2023-11-20 19:55:47,189 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.150e+01 8.270e+01 9.064e+01 9.843e+01 1.286e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-20 19:55:55,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1199066.6666666667, ans=0.125
2023-11-20 19:56:11,166 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.32 vs. limit=15.0
2023-11-20 19:56:12,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.43 vs. limit=10.0
2023-11-20 19:56:17,613 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11550, loss[loss=0.06473, simple_loss=0.07993, pruned_loss=0.01674, audio_tagging_loss=0.008021, over 15041.00 frames. ], tot_loss[loss=0.07665, simple_loss=0.09807, pruned_loss=0.01796, audio_tagging_loss=0.009656, over 3047395.31 frames. ], batch size: 57, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:56:24,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1199200.0, ans=0.125
2023-11-20 19:56:33,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1199266.6666666667, ans=0.07
2023-11-20 19:56:41,393 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179900
2023-11-20 19:56:52,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1199333.3333333333, ans=0.1
2023-11-20 19:56:52,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1199333.3333333333, ans=0.2
2023-11-20 19:56:56,103 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 19:56:59,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.39 vs. limit=15.0
2023-11-20 19:57:00,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1199400.0, ans=0.0
2023-11-20 19:57:02,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.97 vs. limit=15.0
2023-11-20 19:57:21,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1199533.3333333333, ans=0.0
2023-11-20 19:57:22,580 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11600, loss[loss=0.07738, simple_loss=0.09477, pruned_loss=0.01773, audio_tagging_loss=0.01227, over 14990.00 frames. ], tot_loss[loss=0.07728, simple_loss=0.09859, pruned_loss=0.01831, audio_tagging_loss=0.00968, over 3047099.84 frames. ], batch size: 56, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:57:22,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1199533.3333333333, ans=0.125
2023-11-20 19:57:27,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1199533.3333333333, ans=0.1
2023-11-20 19:57:44,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 179950
2023-11-20 19:57:55,303 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.413e+01 9.073e+01 1.016e+02 1.405e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-20 19:58:06,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1199733.3333333333, ans=0.2
2023-11-20 19:58:15,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1199800.0, ans=0.1
2023-11-20 19:58:20,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1199800.0, ans=0.1
2023-11-20 19:58:26,079 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11650, loss[loss=0.07579, simple_loss=0.09353, pruned_loss=0.01905, audio_tagging_loss=0.009971, over 14048.00 frames. ], tot_loss[loss=0.07668, simple_loss=0.09758, pruned_loss=0.01802, audio_tagging_loss=0.00987, over 3050539.42 frames. ], batch size: 53, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:58:48,777 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180000
2023-11-20 19:58:50,238 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-180000.pt
2023-11-20 19:58:53,627 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.13 vs. limit=15.0
2023-11-20 19:58:58,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1200000.0, ans=0.035
2023-11-20 19:59:01,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1200000.0, ans=0.2
2023-11-20 19:59:08,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1200066.6666666667, ans=0.125
2023-11-20 19:59:11,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1200066.6666666667, ans=0.0
2023-11-20 19:59:13,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1200066.6666666667, ans=0.125
2023-11-20 19:59:15,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1200066.6666666667, ans=0.0
2023-11-20 19:59:32,129 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11700, loss[loss=0.0976, simple_loss=0.1239, pruned_loss=0.0265, audio_tagging_loss=0.009146, over 14550.00 frames. ], tot_loss[loss=0.07673, simple_loss=0.09763, pruned_loss=0.01796, audio_tagging_loss=0.009963, over 3047428.16 frames. ], batch size: 55, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 19:59:36,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1200200.0, ans=0.125
2023-11-20 19:59:55,388 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180050
2023-11-20 19:59:57,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1200333.3333333333, ans=0.125
2023-11-20 20:00:06,809 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.544e+01 8.114e+01 8.755e+01 9.534e+01 1.280e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-20 20:00:10,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1200400.0, ans=0.125
2023-11-20 20:00:13,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1200400.0, ans=0.0
2023-11-20 20:00:23,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1200466.6666666667, ans=0.0
2023-11-20 20:00:35,998 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11750, loss[loss=0.08265, simple_loss=0.09473, pruned_loss=0.02539, audio_tagging_loss=0.009898, over 15131.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.0986, pruned_loss=0.01844, audio_tagging_loss=0.009885, over 3052432.96 frames. ], batch size: 59, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:00:36,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1200533.3333333333, ans=0.0
2023-11-20 20:00:44,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.90 vs. limit=10.0
2023-11-20 20:00:48,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1200600.0, ans=6.0
2023-11-20 20:00:50,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1200600.0, ans=0.125
2023-11-20 20:00:55,896 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.22 vs. limit=15.0
2023-11-20 20:00:58,972 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180100
2023-11-20 20:01:02,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1200666.6666666667, ans=0.125
2023-11-20 20:01:40,521 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11800, loss[loss=0.06574, simple_loss=0.08011, pruned_loss=0.01444, audio_tagging_loss=0.01124, over 15280.00 frames. ], tot_loss[loss=0.07784, simple_loss=0.09872, pruned_loss=0.01853, audio_tagging_loss=0.00995, over 3043734.91 frames. ], batch size: 58, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:01:59,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1200933.3333333333, ans=0.125
2023-11-20 20:02:03,384 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180150
2023-11-20 20:02:11,977 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.05 vs. limit=22.5
2023-11-20 20:02:15,390 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.889e+01 8.209e+01 9.058e+01 9.835e+01 1.362e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-20 20:02:24,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1201066.6666666667, ans=0.125
2023-11-20 20:02:29,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1201066.6666666667, ans=0.125
2023-11-20 20:02:36,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1201133.3333333333, ans=0.0
2023-11-20 20:02:43,903 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11850, loss[loss=0.05262, simple_loss=0.06493, pruned_loss=0.008741, audio_tagging_loss=0.01141, over 16297.00 frames. ], tot_loss[loss=0.07786, simple_loss=0.0988, pruned_loss=0.01844, audio_tagging_loss=0.01002, over 3047059.75 frames. ], batch size: 63, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:02:47,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1201200.0, ans=0.0
2023-11-20 20:03:00,497 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.04 vs. limit=15.0
2023-11-20 20:03:05,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1201266.6666666667, ans=0.125
2023-11-20 20:03:06,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1201266.6666666667, ans=0.0
2023-11-20 20:03:07,749 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180200
2023-11-20 20:03:09,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1201333.3333333333, ans=0.0
2023-11-20 20:03:11,316 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.57 vs. limit=10.0
2023-11-20 20:03:28,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1201400.0, ans=0.1
2023-11-20 20:03:31,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1201400.0, ans=0.1
2023-11-20 20:03:37,305 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.67 vs. limit=15.0
2023-11-20 20:03:48,566 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11900, loss[loss=0.04878, simple_loss=0.05407, pruned_loss=0.01161, audio_tagging_loss=0.01014, over 14711.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09818, pruned_loss=0.01833, audio_tagging_loss=0.01018, over 3047449.87 frames. ], batch size: 57, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:03:55,851 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.07 vs. limit=15.0
2023-11-20 20:04:09,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1201600.0, ans=0.95
2023-11-20 20:04:11,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180250
2023-11-20 20:04:15,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1201666.6666666667, ans=0.0
2023-11-20 20:04:22,706 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.137e+01 8.009e+01 8.683e+01 9.562e+01 2.232e+02, threshold=1.737e+02, percent-clipped=1.0
2023-11-20 20:04:26,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1201733.3333333333, ans=0.0
2023-11-20 20:04:27,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1201733.3333333333, ans=0.0
2023-11-20 20:04:52,320 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 11950, loss[loss=0.07403, simple_loss=0.09708, pruned_loss=0.01566, audio_tagging_loss=0.009826, over 15331.00 frames. ], tot_loss[loss=0.07796, simple_loss=0.09846, pruned_loss=0.01845, audio_tagging_loss=0.01027, over 3048846.28 frames. ], batch size: 56, lr: 4.52e-03, grad_scale: 16.0
2023-11-20 20:05:04,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1201933.3333333333, ans=0.125
2023-11-20 20:05:14,210 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180300
2023-11-20 20:05:33,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1202066.6666666667, ans=0.0
2023-11-20 20:05:52,621 INFO [train_asr.py:1221] (0/4) Epoch 15, batch 12000, loss[loss=0.09745, simple_loss=0.127, pruned_loss=0.02518, audio_tagging_loss=0.008762, over 15543.00 frames. ], tot_loss[loss=0.07892, simple_loss=0.09955, pruned_loss=0.01885, audio_tagging_loss=0.01029, over 3054262.16 frames. ], batch size: 57, lr: 4.52e-03, grad_scale: 32.0
2023-11-20 20:05:52,625 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-20 20:06:22,636 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([1.9990, 3.1324, 3.4201, 2.9459, 3.7724, 3.8357, 3.3725, 3.1446],
       device='cuda:0')
2023-11-20 20:06:33,115 INFO [train_asr.py:1253] (0/4) Epoch 15, validation: loss=0.06134, simple_loss=0.05315, pruned_loss=0.00551, audio_tagging_loss=0.02926, over 4681554.00 frames. 
2023-11-20 20:06:33,116 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-20 20:06:42,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1202200.0, ans=0.125
2023-11-20 20:06:54,555 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180350
2023-11-20 20:07:03,299 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-15.pt
2023-11-20 20:07:37,550 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 0, loss[loss=0.1061, simple_loss=0.1174, pruned_loss=0.02494, audio_tagging_loss=0.02254, over 16121.00 frames. ], tot_loss[loss=0.1061, simple_loss=0.1174, pruned_loss=0.02494, audio_tagging_loss=0.02254, over 16121.00 frames. ], batch size: 62, lr: 4.37e-03, grad_scale: 32.0
2023-11-20 20:07:37,553 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-20 20:08:13,626 INFO [train_asr.py:1253] (0/4) Epoch 16, validation: loss=0.06129, simple_loss=0.0532, pruned_loss=0.005566, audio_tagging_loss=0.02913, over 4681554.00 frames. 
2023-11-20 20:08:13,627 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-20 20:08:17,308 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.282e+01 8.389e+01 9.121e+01 1.002e+02 1.446e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-20 20:08:44,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1202493.3333333333, ans=0.2
2023-11-20 20:08:58,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1202560.0, ans=0.2
2023-11-20 20:08:59,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1202560.0, ans=0.0
2023-11-20 20:09:07,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1202626.6666666667, ans=0.125
2023-11-20 20:09:11,359 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180400
2023-11-20 20:09:19,019 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 50, loss[loss=0.07727, simple_loss=0.08195, pruned_loss=0.01663, audio_tagging_loss=0.01967, over 16364.00 frames. ], tot_loss[loss=0.08513, simple_loss=0.09515, pruned_loss=0.01796, audio_tagging_loss=0.0196, over 687023.77 frames. ], batch size: 63, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:09:58,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1202893.3333333333, ans=0.0
2023-11-20 20:10:05,367 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.22 vs. limit=6.0
2023-11-20 20:10:16,033 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180450
2023-11-20 20:10:23,390 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 100, loss[loss=0.08073, simple_loss=0.1033, pruned_loss=0.01424, audio_tagging_loss=0.01487, over 16613.00 frames. ], tot_loss[loss=0.08641, simple_loss=0.09869, pruned_loss=0.01841, audio_tagging_loss=0.01866, over 1209872.24 frames. ], batch size: 59, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:10:29,504 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.348e+01 8.922e+01 9.752e+01 1.060e+02 1.405e+02, threshold=1.950e+02, percent-clipped=0.0
2023-11-20 20:10:33,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1203026.6666666667, ans=0.2
2023-11-20 20:10:34,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1203026.6666666667, ans=0.0
2023-11-20 20:11:11,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1203226.6666666667, ans=0.0
2023-11-20 20:11:21,281 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180500
2023-11-20 20:11:29,071 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 150, loss[loss=0.09666, simple_loss=0.1289, pruned_loss=0.02503, audio_tagging_loss=0.007181, over 14263.00 frames. ], tot_loss[loss=0.08304, simple_loss=0.09673, pruned_loss=0.018, audio_tagging_loss=0.01668, over 1615960.89 frames. ], batch size: 53, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:11:29,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1203360.0, ans=0.2
2023-11-20 20:11:38,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1203360.0, ans=0.125
2023-11-20 20:11:44,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1203426.6666666667, ans=0.125
2023-11-20 20:12:03,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1203493.3333333333, ans=0.1
2023-11-20 20:12:25,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180550
2023-11-20 20:12:31,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1203626.6666666667, ans=0.125
2023-11-20 20:12:33,426 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 200, loss[loss=0.072, simple_loss=0.08278, pruned_loss=0.01857, audio_tagging_loss=0.01203, over 14402.00 frames. ], tot_loss[loss=0.08217, simple_loss=0.09803, pruned_loss=0.01848, audio_tagging_loss=0.01468, over 1934584.64 frames. ], batch size: 57, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:12:39,633 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.954e+01 8.158e+01 8.825e+01 9.398e+01 1.176e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-20 20:12:41,310 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:12:50,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.86 vs. limit=22.5
2023-11-20 20:12:59,887 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.43 vs. limit=15.0
2023-11-20 20:13:07,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.50 vs. limit=15.0
2023-11-20 20:13:12,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1203893.3333333333, ans=0.125
2023-11-20 20:13:22,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1203893.3333333333, ans=0.125
2023-11-20 20:13:29,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180600
2023-11-20 20:13:32,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1203960.0, ans=0.125
2023-11-20 20:13:36,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1204026.6666666667, ans=0.2
2023-11-20 20:13:36,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1204026.6666666667, ans=0.0
2023-11-20 20:13:37,509 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 250, loss[loss=0.09265, simple_loss=0.1195, pruned_loss=0.02268, audio_tagging_loss=0.01021, over 15424.00 frames. ], tot_loss[loss=0.08101, simple_loss=0.09862, pruned_loss=0.0185, audio_tagging_loss=0.0132, over 2182127.90 frames. ], batch size: 55, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:13:53,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff2.min_abs, batch_count=1204093.3333333333, ans=0.1
2023-11-20 20:14:03,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1204160.0, ans=0.125
2023-11-20 20:14:14,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1204160.0, ans=0.125
2023-11-20 20:14:19,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1204226.6666666667, ans=0.125
2023-11-20 20:14:23,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1204226.6666666667, ans=0.2
2023-11-20 20:14:35,068 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180650
2023-11-20 20:14:42,352 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 300, loss[loss=0.08598, simple_loss=0.1082, pruned_loss=0.02304, audio_tagging_loss=0.008822, over 15233.00 frames. ], tot_loss[loss=0.08067, simple_loss=0.09965, pruned_loss=0.01879, audio_tagging_loss=0.01205, over 2376471.58 frames. ], batch size: 54, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:14:48,970 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.454e+01 9.235e+01 1.018e+02 1.447e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-20 20:15:16,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1204493.3333333333, ans=0.2
2023-11-20 20:15:18,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1204493.3333333333, ans=0.125
2023-11-20 20:15:26,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1204560.0, ans=0.125
2023-11-20 20:15:36,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1204626.6666666667, ans=0.0
2023-11-20 20:15:39,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180700
2023-11-20 20:15:46,856 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 350, loss[loss=0.05319, simple_loss=0.06308, pruned_loss=0.01064, audio_tagging_loss=0.01101, over 15439.00 frames. ], tot_loss[loss=0.07851, simple_loss=0.09763, pruned_loss=0.0182, audio_tagging_loss=0.0115, over 2525103.49 frames. ], batch size: 59, lr: 4.37e-03, grad_scale: 8.0
2023-11-20 20:16:44,115 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180750
2023-11-20 20:16:51,461 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 400, loss[loss=0.06744, simple_loss=0.08031, pruned_loss=0.01561, audio_tagging_loss=0.01167, over 17124.00 frames. ], tot_loss[loss=0.07795, simple_loss=0.09744, pruned_loss=0.01807, audio_tagging_loss=0.01116, over 2643306.09 frames. ], batch size: 65, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:16:58,141 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.022e+01 8.670e+01 9.315e+01 1.277e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-20 20:17:11,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1205093.3333333333, ans=0.125
2023-11-20 20:17:14,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1205093.3333333333, ans=0.125
2023-11-20 20:17:23,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1205160.0, ans=0.125
2023-11-20 20:17:33,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1205226.6666666667, ans=0.05
2023-11-20 20:17:40,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1205226.6666666667, ans=0.125
2023-11-20 20:17:48,938 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180800
2023-11-20 20:17:56,396 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 450, loss[loss=0.08086, simple_loss=0.09738, pruned_loss=0.02225, audio_tagging_loss=0.009926, over 16204.00 frames. ], tot_loss[loss=0.07798, simple_loss=0.09787, pruned_loss=0.01826, audio_tagging_loss=0.01078, over 2728711.73 frames. ], batch size: 61, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:18:16,771 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.61 vs. limit=15.0
2023-11-20 20:18:21,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1205493.3333333333, ans=0.125
2023-11-20 20:18:38,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1205560.0, ans=0.0
2023-11-20 20:18:45,685 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.601e-01
2023-11-20 20:18:52,653 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180850
2023-11-20 20:18:59,881 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 500, loss[loss=0.07267, simple_loss=0.1007, pruned_loss=0.01456, audio_tagging_loss=0.007772, over 14935.00 frames. ], tot_loss[loss=0.07836, simple_loss=0.09861, pruned_loss=0.01856, audio_tagging_loss=0.01049, over 2803065.69 frames. ], batch size: 56, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:19:07,189 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.820e+01 8.156e+01 8.896e+01 9.725e+01 1.312e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-20 20:19:27,788 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:19:43,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1205893.3333333333, ans=0.125
2023-11-20 20:19:43,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1205893.3333333333, ans=0.2
2023-11-20 20:19:46,474 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.97 vs. limit=10.0
2023-11-20 20:19:53,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.68 vs. limit=15.0
2023-11-20 20:19:54,536 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:19:56,714 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180900
2023-11-20 20:19:58,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1205960.0, ans=0.125
2023-11-20 20:20:03,757 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 550, loss[loss=0.06418, simple_loss=0.08445, pruned_loss=0.01371, audio_tagging_loss=0.008247, over 15144.00 frames. ], tot_loss[loss=0.07877, simple_loss=0.09935, pruned_loss=0.0188, audio_tagging_loss=0.0103, over 2851303.26 frames. ], batch size: 58, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:20:04,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1206026.6666666667, ans=0.125
2023-11-20 20:20:26,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1206093.3333333333, ans=0.125
2023-11-20 20:20:36,012 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.49 vs. limit=22.5
2023-11-20 20:20:39,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1206160.0, ans=0.125
2023-11-20 20:20:45,528 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.62 vs. limit=22.5
2023-11-20 20:20:51,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1206226.6666666667, ans=0.125
2023-11-20 20:20:53,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1206293.3333333333, ans=0.1
2023-11-20 20:20:56,659 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.94 vs. limit=22.5
2023-11-20 20:20:58,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1206293.3333333333, ans=0.125
2023-11-20 20:20:59,467 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 180950
2023-11-20 20:21:06,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1206360.0, ans=0.125
2023-11-20 20:21:07,321 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 600, loss[loss=0.05549, simple_loss=0.06027, pruned_loss=0.01278, audio_tagging_loss=0.01258, over 15780.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09809, pruned_loss=0.01828, audio_tagging_loss=0.01026, over 2891586.38 frames. ], batch size: 61, lr: 4.37e-03, grad_scale: 16.0
2023-11-20 20:21:13,343 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.725e+01 8.009e+01 8.820e+01 9.311e+01 1.105e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-20 20:21:23,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1206426.6666666667, ans=0.0
2023-11-20 20:21:51,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1206560.0, ans=0.125
2023-11-20 20:21:58,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1206626.6666666667, ans=0.0
2023-11-20 20:22:01,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1206626.6666666667, ans=0.2
2023-11-20 20:22:02,617 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181000
2023-11-20 20:22:04,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1206626.6666666667, ans=0.125
2023-11-20 20:22:10,139 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 650, loss[loss=0.07225, simple_loss=0.08673, pruned_loss=0.01598, audio_tagging_loss=0.0129, over 15674.00 frames. ], tot_loss[loss=0.07797, simple_loss=0.09881, pruned_loss=0.01842, audio_tagging_loss=0.01015, over 2924222.49 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:22:26,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1206760.0, ans=0.0
2023-11-20 20:22:47,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1206893.3333333333, ans=0.2
2023-11-20 20:23:06,210 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181050
2023-11-20 20:23:10,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1206960.0, ans=0.125
2023-11-20 20:23:14,272 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 700, loss[loss=0.07928, simple_loss=0.1055, pruned_loss=0.01749, audio_tagging_loss=0.00903, over 14480.00 frames. ], tot_loss[loss=0.07816, simple_loss=0.0992, pruned_loss=0.01842, audio_tagging_loss=0.01013, over 2960129.27 frames. ], batch size: 53, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:23:15,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1207026.6666666667, ans=0.0
2023-11-20 20:23:15,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1207026.6666666667, ans=0.2
2023-11-20 20:23:18,651 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.64 vs. limit=15.0
2023-11-20 20:23:20,234 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.218e+01 8.797e+01 9.608e+01 1.489e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-20 20:23:36,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1207093.3333333333, ans=0.0
2023-11-20 20:23:58,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1207226.6666666667, ans=0.0
2023-11-20 20:24:01,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1207226.6666666667, ans=0.125
2023-11-20 20:24:06,482 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.17 vs. limit=15.0
2023-11-20 20:24:09,710 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181100
2023-11-20 20:24:09,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1207293.3333333333, ans=0.0
2023-11-20 20:24:10,334 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.17 vs. limit=15.0
2023-11-20 20:24:17,488 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 750, loss[loss=0.08497, simple_loss=0.1048, pruned_loss=0.02051, audio_tagging_loss=0.01206, over 14777.00 frames. ], tot_loss[loss=0.07834, simple_loss=0.09963, pruned_loss=0.01844, audio_tagging_loss=0.01008, over 2975172.49 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:24:20,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1207360.0, ans=0.125
2023-11-20 20:24:25,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1207360.0, ans=0.5
2023-11-20 20:24:41,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1207493.3333333333, ans=0.05
2023-11-20 20:24:57,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1207560.0, ans=0.0
2023-11-20 20:25:13,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181150
2023-11-20 20:25:19,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1207693.3333333333, ans=0.0
2023-11-20 20:25:20,349 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 800, loss[loss=0.08147, simple_loss=0.1056, pruned_loss=0.01988, audio_tagging_loss=0.008765, over 15033.00 frames. ], tot_loss[loss=0.07881, simple_loss=0.1001, pruned_loss=0.0186, audio_tagging_loss=0.01016, over 2989410.28 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 32.0
2023-11-20 20:25:26,966 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.116e+01 8.797e+01 9.654e+01 1.312e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-20 20:25:42,700 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.11 vs. limit=15.0
2023-11-20 20:25:46,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1207826.6666666667, ans=0.125
2023-11-20 20:26:13,002 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.25 vs. limit=15.0
2023-11-20 20:26:16,419 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181200
2023-11-20 20:26:23,966 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 850, loss[loss=0.07293, simple_loss=0.09004, pruned_loss=0.01636, audio_tagging_loss=0.01155, over 15069.00 frames. ], tot_loss[loss=0.0788, simple_loss=0.09981, pruned_loss=0.01863, audio_tagging_loss=0.01027, over 2999367.59 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 32.0
2023-11-20 20:26:39,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1208093.3333333333, ans=0.1
2023-11-20 20:26:47,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1208093.3333333333, ans=0.0
2023-11-20 20:27:20,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181250
2023-11-20 20:27:28,690 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 900, loss[loss=0.08332, simple_loss=0.1, pruned_loss=0.02256, audio_tagging_loss=0.01074, over 14968.00 frames. ], tot_loss[loss=0.07813, simple_loss=0.09867, pruned_loss=0.0184, audio_tagging_loss=0.01038, over 3001872.59 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:27:35,944 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.347e+01 8.999e+01 9.634e+01 1.332e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-20 20:27:56,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1208493.3333333333, ans=0.0
2023-11-20 20:27:57,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1208493.3333333333, ans=0.125
2023-11-20 20:28:06,436 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:28:23,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1208626.6666666667, ans=0.125
2023-11-20 20:28:23,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1208626.6666666667, ans=0.125
2023-11-20 20:28:24,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181300
2023-11-20 20:28:31,774 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 950, loss[loss=0.0918, simple_loss=0.1172, pruned_loss=0.02455, audio_tagging_loss=0.008631, over 15055.00 frames. ], tot_loss[loss=0.07826, simple_loss=0.09914, pruned_loss=0.01851, audio_tagging_loss=0.01018, over 3009754.70 frames. ], batch size: 54, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:28:33,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1208693.3333333333, ans=0.1
2023-11-20 20:28:52,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1208760.0, ans=0.125
2023-11-20 20:29:27,797 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181350
2023-11-20 20:29:35,693 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1000, loss[loss=0.08772, simple_loss=0.1174, pruned_loss=0.02171, audio_tagging_loss=0.007326, over 15981.00 frames. ], tot_loss[loss=0.07865, simple_loss=0.1002, pruned_loss=0.01866, audio_tagging_loss=0.009885, over 3022007.86 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:29:37,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1209026.6666666667, ans=0.09899494936611666
2023-11-20 20:29:42,946 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.425e+01 8.384e+01 9.027e+01 9.924e+01 1.224e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-20 20:29:57,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1209093.3333333333, ans=0.0
2023-11-20 20:30:02,979 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:30:14,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1209226.6666666667, ans=0.2
2023-11-20 20:30:20,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1209226.6666666667, ans=0.125
2023-11-20 20:30:31,936 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181400
2023-11-20 20:30:40,309 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1050, loss[loss=0.06108, simple_loss=0.07087, pruned_loss=0.01683, audio_tagging_loss=0.008822, over 15038.00 frames. ], tot_loss[loss=0.07798, simple_loss=0.09886, pruned_loss=0.01866, audio_tagging_loss=0.009892, over 3024200.22 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:30:47,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1209360.0, ans=0.0
2023-11-20 20:30:47,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1209360.0, ans=15.0
2023-11-20 20:30:48,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1209360.0, ans=0.0
2023-11-20 20:30:49,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1209360.0, ans=0.125
2023-11-20 20:30:49,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1209360.0, ans=0.2
2023-11-20 20:30:52,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1209426.6666666667, ans=0.1
2023-11-20 20:31:00,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1209426.6666666667, ans=0.0
2023-11-20 20:31:04,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1209493.3333333333, ans=0.125
2023-11-20 20:31:33,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1209626.6666666667, ans=0.125
2023-11-20 20:31:35,731 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181450
2023-11-20 20:31:42,798 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1100, loss[loss=0.06462, simple_loss=0.07595, pruned_loss=0.0131, audio_tagging_loss=0.01355, over 14224.00 frames. ], tot_loss[loss=0.07755, simple_loss=0.09858, pruned_loss=0.0184, audio_tagging_loss=0.009858, over 3036147.57 frames. ], batch size: 55, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:31:45,259 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:31:49,905 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.927e+01 7.892e+01 8.572e+01 9.142e+01 1.110e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-20 20:31:56,216 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.73 vs. limit=15.0
2023-11-20 20:32:02,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1209760.0, ans=0.1
2023-11-20 20:32:04,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1209760.0, ans=0.0
2023-11-20 20:32:19,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1209826.6666666667, ans=0.2
2023-11-20 20:32:22,026 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.14 vs. limit=15.0
2023-11-20 20:32:37,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1209960.0, ans=0.125
2023-11-20 20:32:38,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181500
2023-11-20 20:32:39,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1209960.0, ans=0.125
2023-11-20 20:32:46,234 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1150, loss[loss=0.08213, simple_loss=0.1107, pruned_loss=0.0185, audio_tagging_loss=0.00828, over 15280.00 frames. ], tot_loss[loss=0.07687, simple_loss=0.09769, pruned_loss=0.0182, audio_tagging_loss=0.009822, over 3035569.61 frames. ], batch size: 55, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:32:51,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1210026.6666666667, ans=0.125
2023-11-20 20:32:59,234 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.95 vs. limit=15.0
2023-11-20 20:33:15,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1210160.0, ans=0.125
2023-11-20 20:33:17,341 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.60 vs. limit=15.0
2023-11-20 20:33:18,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1210160.0, ans=0.125
2023-11-20 20:33:21,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1210160.0, ans=15.0
2023-11-20 20:33:26,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1210226.6666666667, ans=0.125
2023-11-20 20:33:29,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1210226.6666666667, ans=0.0
2023-11-20 20:33:31,136 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:33:42,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181550
2023-11-20 20:33:46,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1210293.3333333333, ans=0.1
2023-11-20 20:33:49,345 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1200, loss[loss=0.0715, simple_loss=0.0878, pruned_loss=0.017, audio_tagging_loss=0.0106, over 14988.00 frames. ], tot_loss[loss=0.07663, simple_loss=0.09772, pruned_loss=0.01806, audio_tagging_loss=0.009711, over 3046983.69 frames. ], batch size: 56, lr: 4.36e-03, grad_scale: 32.0
2023-11-20 20:33:52,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1210360.0, ans=0.125
2023-11-20 20:33:56,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1210360.0, ans=0.125
2023-11-20 20:33:57,410 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.318e+01 9.034e+01 9.990e+01 1.243e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-20 20:34:26,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1210493.3333333333, ans=0.125
2023-11-20 20:34:27,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1210560.0, ans=0.05
2023-11-20 20:34:45,279 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.71 vs. limit=15.0
2023-11-20 20:34:45,943 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181600
2023-11-20 20:34:49,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1210626.6666666667, ans=0.125
2023-11-20 20:34:54,254 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1250, loss[loss=0.08135, simple_loss=0.1054, pruned_loss=0.01857, audio_tagging_loss=0.0101, over 15078.00 frames. ], tot_loss[loss=0.07649, simple_loss=0.09741, pruned_loss=0.01803, audio_tagging_loss=0.009755, over 3046834.47 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 32.0
2023-11-20 20:34:54,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1210693.3333333333, ans=0.0
2023-11-20 20:34:55,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1210693.3333333333, ans=0.125
2023-11-20 20:35:05,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1210760.0, ans=0.125
2023-11-20 20:35:28,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1210826.6666666667, ans=0.125
2023-11-20 20:35:37,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1210893.3333333333, ans=0.1
2023-11-20 20:35:45,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1210960.0, ans=0.125
2023-11-20 20:35:50,142 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181650
2023-11-20 20:35:50,991 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.94 vs. limit=8.0
2023-11-20 20:35:53,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1210960.0, ans=0.125
2023-11-20 20:35:56,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1211026.6666666667, ans=0.125
2023-11-20 20:35:57,753 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1300, loss[loss=0.08009, simple_loss=0.1021, pruned_loss=0.01832, audio_tagging_loss=0.01072, over 15760.00 frames. ], tot_loss[loss=0.07644, simple_loss=0.09747, pruned_loss=0.01808, audio_tagging_loss=0.009628, over 3046533.71 frames. ], batch size: 58, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:36:06,288 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.079e+01 9.042e+01 9.644e+01 1.533e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-20 20:36:09,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1211093.3333333333, ans=0.035
2023-11-20 20:36:24,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1211160.0, ans=0.125
2023-11-20 20:36:25,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.03 vs. limit=10.0
2023-11-20 20:36:34,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1211226.6666666667, ans=0.2
2023-11-20 20:36:53,194 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181700
2023-11-20 20:36:58,491 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.83 vs. limit=15.0
2023-11-20 20:37:00,280 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1350, loss[loss=0.06729, simple_loss=0.07882, pruned_loss=0.0167, audio_tagging_loss=0.01118, over 15607.00 frames. ], tot_loss[loss=0.07696, simple_loss=0.09788, pruned_loss=0.0183, audio_tagging_loss=0.009715, over 3044920.56 frames. ], batch size: 59, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:37:05,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1211360.0, ans=0.0
2023-11-20 20:37:44,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1211560.0, ans=0.125
2023-11-20 20:37:46,310 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:37:56,982 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181750
2023-11-20 20:38:04,313 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1400, loss[loss=0.06825, simple_loss=0.08876, pruned_loss=0.01517, audio_tagging_loss=0.0087, over 15950.00 frames. ], tot_loss[loss=0.07739, simple_loss=0.09848, pruned_loss=0.01844, audio_tagging_loss=0.009707, over 3044401.09 frames. ], batch size: 57, lr: 4.36e-03, grad_scale: 16.0
2023-11-20 20:38:13,604 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.529e+01 8.254e+01 8.824e+01 9.596e+01 1.357e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-20 20:38:39,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1211826.6666666667, ans=0.125
2023-11-20 20:39:01,528 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181800
2023-11-20 20:39:08,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1212026.6666666667, ans=0.025
2023-11-20 20:39:09,401 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1450, loss[loss=0.09135, simple_loss=0.1235, pruned_loss=0.0224, audio_tagging_loss=0.007215, over 15129.00 frames. ], tot_loss[loss=0.07743, simple_loss=0.0985, pruned_loss=0.01838, audio_tagging_loss=0.009805, over 3039698.35 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 16.0
2023-11-20 20:39:13,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1212026.6666666667, ans=0.035
2023-11-20 20:39:31,445 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.15 vs. limit=15.0
2023-11-20 20:40:03,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1212293.3333333333, ans=0.125
2023-11-20 20:40:05,979 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181850
2023-11-20 20:40:13,044 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1500, loss[loss=0.07641, simple_loss=0.08941, pruned_loss=0.01976, audio_tagging_loss=0.01194, over 15150.00 frames. ], tot_loss[loss=0.0772, simple_loss=0.09802, pruned_loss=0.01835, audio_tagging_loss=0.009839, over 3044600.87 frames. ], batch size: 59, lr: 4.35e-03, grad_scale: 16.0
2023-11-20 20:40:22,246 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.058e+01 8.717e+01 9.373e+01 1.477e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-20 20:40:50,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1212560.0, ans=0.5
2023-11-20 20:41:08,320 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181900
2023-11-20 20:41:16,570 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1550, loss[loss=0.07532, simple_loss=0.1014, pruned_loss=0.01505, audio_tagging_loss=0.009582, over 15410.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.09794, pruned_loss=0.01825, audio_tagging_loss=0.009955, over 3050232.82 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 16.0
2023-11-20 20:41:24,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.44 vs. limit=10.0
2023-11-20 20:42:13,190 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 181950
2023-11-20 20:42:20,980 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1600, loss[loss=0.0699, simple_loss=0.07691, pruned_loss=0.01959, audio_tagging_loss=0.01185, over 15191.00 frames. ], tot_loss[loss=0.07741, simple_loss=0.09784, pruned_loss=0.01841, audio_tagging_loss=0.01008, over 3053815.80 frames. ], batch size: 58, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:42:30,055 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.255e+01 8.909e+01 9.457e+01 1.581e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-20 20:42:44,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1213093.3333333333, ans=0.09899494936611666
2023-11-20 20:42:49,991 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.14 vs. limit=15.0
2023-11-20 20:42:53,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1213160.0, ans=0.2
2023-11-20 20:43:17,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182000
2023-11-20 20:43:24,947 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1650, loss[loss=0.08808, simple_loss=0.1075, pruned_loss=0.02332, audio_tagging_loss=0.01101, over 14627.00 frames. ], tot_loss[loss=0.07769, simple_loss=0.09798, pruned_loss=0.01852, audio_tagging_loss=0.01017, over 3050132.55 frames. ], batch size: 53, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:43:51,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1213493.3333333333, ans=0.1
2023-11-20 20:43:54,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.95 vs. limit=15.0
2023-11-20 20:43:57,857 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.26 vs. limit=10.0
2023-11-20 20:44:14,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1213626.6666666667, ans=0.04949747468305833
2023-11-20 20:44:17,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1213626.6666666667, ans=0.0
2023-11-20 20:44:18,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1213626.6666666667, ans=0.0
2023-11-20 20:44:20,701 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182050
2023-11-20 20:44:22,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1213626.6666666667, ans=0.09899494936611666
2023-11-20 20:44:28,444 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1700, loss[loss=0.08758, simple_loss=0.1152, pruned_loss=0.01903, audio_tagging_loss=0.01095, over 15935.00 frames. ], tot_loss[loss=0.07761, simple_loss=0.09804, pruned_loss=0.01837, audio_tagging_loss=0.01022, over 3050570.97 frames. ], batch size: 57, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:44:30,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1213693.3333333333, ans=0.2
2023-11-20 20:44:32,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1213693.3333333333, ans=0.0
2023-11-20 20:44:37,001 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.816e+01 8.092e+01 8.985e+01 9.523e+01 1.287e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-20 20:44:43,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1213760.0, ans=0.0
2023-11-20 20:45:23,890 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182100
2023-11-20 20:45:31,267 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1750, loss[loss=0.09611, simple_loss=0.1315, pruned_loss=0.02356, audio_tagging_loss=0.006796, over 15094.00 frames. ], tot_loss[loss=0.07777, simple_loss=0.09842, pruned_loss=0.01844, audio_tagging_loss=0.01012, over 3049747.57 frames. ], batch size: 55, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:45:44,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1214093.3333333333, ans=0.125
2023-11-20 20:45:55,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1214160.0, ans=0.125
2023-11-20 20:46:03,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1214160.0, ans=0.035
2023-11-20 20:46:18,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1214226.6666666667, ans=0.0
2023-11-20 20:46:23,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1214293.3333333333, ans=0.2
2023-11-20 20:46:27,721 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182150
2023-11-20 20:46:28,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1214293.3333333333, ans=0.1
2023-11-20 20:46:35,500 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1800, loss[loss=0.09998, simple_loss=0.1264, pruned_loss=0.02923, audio_tagging_loss=0.00754, over 15866.00 frames. ], tot_loss[loss=0.07793, simple_loss=0.09881, pruned_loss=0.01849, audio_tagging_loss=0.01003, over 3054137.26 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:46:43,842 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.678e+01 8.260e+01 8.761e+01 9.578e+01 1.185e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-20 20:46:52,998 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.24 vs. limit=15.0
2023-11-20 20:47:00,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.80 vs. limit=12.0
2023-11-20 20:47:07,824 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.10 vs. limit=15.0
2023-11-20 20:47:11,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1214560.0, ans=0.0
2023-11-20 20:47:22,376 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.30 vs. limit=12.0
2023-11-20 20:47:31,391 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182200
2023-11-20 20:47:40,236 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1850, loss[loss=0.04629, simple_loss=0.05537, pruned_loss=0.009528, audio_tagging_loss=0.009077, over 14635.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.0986, pruned_loss=0.01838, audio_tagging_loss=0.009943, over 3053799.72 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:47:44,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1214693.3333333333, ans=0.125
2023-11-20 20:48:37,001 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182250
2023-11-20 20:48:44,200 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1900, loss[loss=0.08412, simple_loss=0.1076, pruned_loss=0.02072, audio_tagging_loss=0.009613, over 14851.00 frames. ], tot_loss[loss=0.07862, simple_loss=0.1002, pruned_loss=0.01868, audio_tagging_loss=0.009831, over 3053891.21 frames. ], batch size: 55, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:48:47,324 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.67 vs. limit=15.0
2023-11-20 20:48:53,243 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 7.917e+01 8.580e+01 9.433e+01 1.165e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-20 20:49:06,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1215093.3333333333, ans=0.0
2023-11-20 20:49:11,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1215160.0, ans=0.0
2023-11-20 20:49:13,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1215160.0, ans=0.125
2023-11-20 20:49:34,904 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.71 vs. limit=15.0
2023-11-20 20:49:40,670 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182300
2023-11-20 20:49:42,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1215293.3333333333, ans=0.125
2023-11-20 20:49:48,452 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 1950, loss[loss=0.06196, simple_loss=0.07332, pruned_loss=0.01473, audio_tagging_loss=0.01057, over 13741.00 frames. ], tot_loss[loss=0.07827, simple_loss=0.09988, pruned_loss=0.01851, audio_tagging_loss=0.009816, over 3051193.66 frames. ], batch size: 55, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:49:48,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1215360.0, ans=0.1
2023-11-20 20:49:52,171 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.59 vs. limit=15.0
2023-11-20 20:50:11,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1215426.6666666667, ans=0.025
2023-11-20 20:50:25,559 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.93 vs. limit=6.0
2023-11-20 20:50:26,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1215560.0, ans=0.2
2023-11-20 20:50:45,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182350
2023-11-20 20:50:53,437 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2000, loss[loss=0.06642, simple_loss=0.08921, pruned_loss=0.01203, audio_tagging_loss=0.009781, over 14707.00 frames. ], tot_loss[loss=0.07791, simple_loss=0.09933, pruned_loss=0.01844, audio_tagging_loss=0.009808, over 3046661.81 frames. ], batch size: 55, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:50:55,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1215693.3333333333, ans=0.125
2023-11-20 20:50:57,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.14 vs. limit=15.0
2023-11-20 20:50:58,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1215693.3333333333, ans=0.1
2023-11-20 20:51:01,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1215693.3333333333, ans=0.125
2023-11-20 20:51:02,685 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.938e+01 8.733e+01 9.578e+01 1.409e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-20 20:51:04,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1215693.3333333333, ans=0.0
2023-11-20 20:51:05,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1215760.0, ans=0.0
2023-11-20 20:51:07,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1215760.0, ans=0.125
2023-11-20 20:51:49,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1215960.0, ans=0.125
2023-11-20 20:51:49,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1215960.0, ans=0.0
2023-11-20 20:51:50,582 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182400
2023-11-20 20:51:50,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1215960.0, ans=0.1
2023-11-20 20:51:50,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1215960.0, ans=0.0
2023-11-20 20:51:58,164 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2050, loss[loss=0.05396, simple_loss=0.06374, pruned_loss=0.0108, audio_tagging_loss=0.01129, over 14962.00 frames. ], tot_loss[loss=0.0776, simple_loss=0.09899, pruned_loss=0.0184, audio_tagging_loss=0.009702, over 3048990.10 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:52:28,089 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.07 vs. limit=10.0
2023-11-20 20:52:41,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1216226.6666666667, ans=0.0
2023-11-20 20:52:44,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1216226.6666666667, ans=0.0
2023-11-20 20:52:53,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1216293.3333333333, ans=0.0
2023-11-20 20:52:54,325 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182450
2023-11-20 20:52:57,128 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.552e-01
2023-11-20 20:53:02,341 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2100, loss[loss=0.05962, simple_loss=0.07701, pruned_loss=0.01225, audio_tagging_loss=0.008866, over 16887.00 frames. ], tot_loss[loss=0.07707, simple_loss=0.0982, pruned_loss=0.01829, audio_tagging_loss=0.009681, over 3055089.94 frames. ], batch size: 64, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:53:11,033 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.190e+01 8.920e+01 9.750e+01 1.833e+02, threshold=1.784e+02, percent-clipped=1.0
2023-11-20 20:53:37,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1216493.3333333333, ans=0.0
2023-11-20 20:53:58,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182500
2023-11-20 20:53:59,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1216626.6666666667, ans=0.0
2023-11-20 20:54:03,311 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.14 vs. limit=22.5
2023-11-20 20:54:06,773 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2150, loss[loss=0.06139, simple_loss=0.07629, pruned_loss=0.01319, audio_tagging_loss=0.01005, over 14780.00 frames. ], tot_loss[loss=0.07685, simple_loss=0.09821, pruned_loss=0.01805, audio_tagging_loss=0.009686, over 3053774.03 frames. ], batch size: 56, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:54:18,289 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.93 vs. limit=22.5
2023-11-20 20:54:34,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.96 vs. limit=10.0
2023-11-20 20:54:40,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1216826.6666666667, ans=0.1
2023-11-20 20:54:43,824 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:54:44,636 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.37 vs. limit=15.0
2023-11-20 20:54:57,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1216960.0, ans=0.125
2023-11-20 20:55:04,010 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182550
2023-11-20 20:55:11,336 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2200, loss[loss=0.06883, simple_loss=0.08615, pruned_loss=0.01468, audio_tagging_loss=0.01107, over 15853.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.0976, pruned_loss=0.01787, audio_tagging_loss=0.009747, over 3057235.76 frames. ], batch size: 61, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:55:19,778 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.514e+01 9.084e+01 9.823e+01 1.256e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-20 20:55:21,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1217026.6666666667, ans=0.0
2023-11-20 20:55:23,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.99 vs. limit=15.0
2023-11-20 20:55:29,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1217093.3333333333, ans=0.125
2023-11-20 20:55:30,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1217093.3333333333, ans=0.125
2023-11-20 20:55:58,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.95 vs. limit=22.5
2023-11-20 20:56:07,978 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182600
2023-11-20 20:56:13,438 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:56:16,440 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2250, loss[loss=0.08491, simple_loss=0.1051, pruned_loss=0.02194, audio_tagging_loss=0.01044, over 15309.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09822, pruned_loss=0.01795, audio_tagging_loss=0.009783, over 3056608.09 frames. ], batch size: 59, lr: 4.35e-03, grad_scale: 32.0
2023-11-20 20:56:25,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1217360.0, ans=0.125
2023-11-20 20:56:31,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1217426.6666666667, ans=0.0
2023-11-20 20:56:50,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1217493.3333333333, ans=0.125
2023-11-20 20:56:57,539 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.02 vs. limit=10.0
2023-11-20 20:57:11,862 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182650
2023-11-20 20:57:19,142 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2300, loss[loss=0.07946, simple_loss=0.1046, pruned_loss=0.01766, audio_tagging_loss=0.009503, over 15769.00 frames. ], tot_loss[loss=0.07747, simple_loss=0.09883, pruned_loss=0.01827, audio_tagging_loss=0.009779, over 3055106.76 frames. ], batch size: 58, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 20:57:23,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1217693.3333333333, ans=0.125
2023-11-20 20:57:29,294 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.519e+01 8.084e+01 8.941e+01 9.693e+01 1.110e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-20 20:57:30,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=1217693.3333333333, ans=0.05
2023-11-20 20:57:32,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1217760.0, ans=0.1
2023-11-20 20:57:32,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1217760.0, ans=0.125
2023-11-20 20:57:34,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1217760.0, ans=0.125
2023-11-20 20:57:44,882 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=15.0
2023-11-20 20:57:46,402 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.70 vs. limit=10.0
2023-11-20 20:57:48,719 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.18 vs. limit=22.5
2023-11-20 20:57:51,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.68 vs. limit=15.0
2023-11-20 20:57:57,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1217893.3333333333, ans=0.1
2023-11-20 20:58:15,668 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 20:58:16,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182700
2023-11-20 20:58:17,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1217960.0, ans=0.1
2023-11-20 20:58:17,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1217960.0, ans=0.125
2023-11-20 20:58:20,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1217960.0, ans=0.125
2023-11-20 20:58:23,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1218026.6666666667, ans=0.0
2023-11-20 20:58:24,704 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2350, loss[loss=0.05662, simple_loss=0.06562, pruned_loss=0.01191, audio_tagging_loss=0.01189, over 15729.00 frames. ], tot_loss[loss=0.07768, simple_loss=0.09915, pruned_loss=0.01829, audio_tagging_loss=0.009817, over 3047835.86 frames. ], batch size: 61, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 20:58:26,527 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.55 vs. limit=15.0
2023-11-20 20:58:32,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1218026.6666666667, ans=10.0
2023-11-20 20:58:58,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1218160.0, ans=0.0
2023-11-20 20:59:17,666 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 20:59:21,109 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182750
2023-11-20 20:59:28,335 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2400, loss[loss=0.05983, simple_loss=0.07342, pruned_loss=0.01421, audio_tagging_loss=0.008912, over 14318.00 frames. ], tot_loss[loss=0.0776, simple_loss=0.09848, pruned_loss=0.01833, audio_tagging_loss=0.01003, over 3051938.26 frames. ], batch size: 56, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 20:59:31,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1218360.0, ans=0.125
2023-11-20 20:59:34,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1218360.0, ans=0.125
2023-11-20 20:59:38,638 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.624e+01 8.066e+01 8.791e+01 9.748e+01 1.239e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-20 20:59:40,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1218426.6666666667, ans=0.2
2023-11-20 20:59:46,896 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.64 vs. limit=6.0
2023-11-20 20:59:50,378 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.49 vs. limit=6.0
2023-11-20 20:59:59,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=1218493.3333333333, ans=0.2
2023-11-20 21:00:16,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1218560.0, ans=0.0
2023-11-20 21:00:17,796 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.01 vs. limit=15.0
2023-11-20 21:00:25,007 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182800
2023-11-20 21:00:32,677 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2450, loss[loss=0.07347, simple_loss=0.09734, pruned_loss=0.01207, audio_tagging_loss=0.01273, over 16288.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.0987, pruned_loss=0.01839, audio_tagging_loss=0.01006, over 3049359.02 frames. ], batch size: 61, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:00:36,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1218693.3333333333, ans=0.0
2023-11-20 21:00:45,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1218760.0, ans=0.0
2023-11-20 21:00:50,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1218760.0, ans=0.05
2023-11-20 21:00:55,849 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.97 vs. limit=22.5
2023-11-20 21:01:10,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1218893.3333333333, ans=0.125
2023-11-20 21:01:30,050 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182850
2023-11-20 21:01:37,753 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2500, loss[loss=0.06293, simple_loss=0.07318, pruned_loss=0.01441, audio_tagging_loss=0.01193, over 15499.00 frames. ], tot_loss[loss=0.07797, simple_loss=0.09889, pruned_loss=0.0185, audio_tagging_loss=0.01003, over 3050934.50 frames. ], batch size: 58, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:01:48,247 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.825e+01 7.891e+01 8.728e+01 9.490e+01 1.260e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-20 21:02:02,878 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:02:04,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1219160.0, ans=0.125
2023-11-20 21:02:35,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182900
2023-11-20 21:02:42,997 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2550, loss[loss=0.07736, simple_loss=0.09876, pruned_loss=0.01617, audio_tagging_loss=0.01181, over 14734.00 frames. ], tot_loss[loss=0.07825, simple_loss=0.09933, pruned_loss=0.01861, audio_tagging_loss=0.009973, over 3052071.84 frames. ], batch size: 57, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:02:45,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1219360.0, ans=0.1
2023-11-20 21:02:50,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1219360.0, ans=0.0
2023-11-20 21:03:15,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1219493.3333333333, ans=0.125
2023-11-20 21:03:24,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1219560.0, ans=0.0
2023-11-20 21:03:40,350 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 182950
2023-11-20 21:03:40,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1219626.6666666667, ans=0.125
2023-11-20 21:03:47,483 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2600, loss[loss=0.0787, simple_loss=0.1074, pruned_loss=0.0161, audio_tagging_loss=0.008892, over 14692.00 frames. ], tot_loss[loss=0.07747, simple_loss=0.09842, pruned_loss=0.01833, audio_tagging_loss=0.009931, over 3046336.36 frames. ], batch size: 56, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:03:57,809 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.668e+01 8.265e+01 9.445e+01 1.045e+02 1.287e+02, threshold=1.889e+02, percent-clipped=0.0
2023-11-20 21:04:07,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1219760.0, ans=0.05
2023-11-20 21:04:08,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1219760.0, ans=0.0
2023-11-20 21:04:12,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1219826.6666666667, ans=0.0
2023-11-20 21:04:23,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1219826.6666666667, ans=0.125
2023-11-20 21:04:31,872 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:04:36,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1219893.3333333333, ans=0.0
2023-11-20 21:04:40,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1219960.0, ans=0.125
2023-11-20 21:04:43,955 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183000
2023-11-20 21:04:52,362 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2650, loss[loss=0.05926, simple_loss=0.06607, pruned_loss=0.01083, audio_tagging_loss=0.01538, over 15722.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09791, pruned_loss=0.01827, audio_tagging_loss=0.009911, over 3047225.37 frames. ], batch size: 60, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:04:58,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1220026.6666666667, ans=0.2
2023-11-20 21:05:04,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1220093.3333333333, ans=0.1
2023-11-20 21:05:29,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1220226.6666666667, ans=0.125
2023-11-20 21:05:48,551 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183050
2023-11-20 21:05:56,279 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2700, loss[loss=0.09767, simple_loss=0.1378, pruned_loss=0.02184, audio_tagging_loss=0.006931, over 15584.00 frames. ], tot_loss[loss=0.07733, simple_loss=0.09857, pruned_loss=0.01827, audio_tagging_loss=0.009772, over 3051272.99 frames. ], batch size: 53, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:06:00,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1220360.0, ans=0.2
2023-11-20 21:06:04,644 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.12 vs. limit=12.0
2023-11-20 21:06:06,375 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.684e+01 8.355e+01 9.034e+01 9.882e+01 1.379e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-20 21:06:52,687 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183100
2023-11-20 21:06:56,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1220626.6666666667, ans=0.1
2023-11-20 21:06:59,905 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2750, loss[loss=0.0777, simple_loss=0.1013, pruned_loss=0.01926, audio_tagging_loss=0.007782, over 15564.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.09802, pruned_loss=0.0184, audio_tagging_loss=0.009763, over 3043440.31 frames. ], batch size: 59, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:07:03,161 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.07 vs. limit=22.5
2023-11-20 21:07:08,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1220693.3333333333, ans=0.125
2023-11-20 21:07:18,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1220760.0, ans=0.025
2023-11-20 21:07:25,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1220826.6666666667, ans=0.2
2023-11-20 21:07:43,877 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.03 vs. limit=22.5
2023-11-20 21:07:52,658 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:07:56,335 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183150
2023-11-20 21:08:04,600 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2800, loss[loss=0.06091, simple_loss=0.08384, pruned_loss=0.01006, audio_tagging_loss=0.008929, over 14927.00 frames. ], tot_loss[loss=0.07685, simple_loss=0.09771, pruned_loss=0.01825, audio_tagging_loss=0.009749, over 3051033.50 frames. ], batch size: 57, lr: 4.34e-03, grad_scale: 32.0
2023-11-20 21:08:08,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1221026.6666666667, ans=0.125
2023-11-20 21:08:10,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1221026.6666666667, ans=0.07
2023-11-20 21:08:15,672 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.121e+01 8.798e+01 9.627e+01 1.849e+02, threshold=1.760e+02, percent-clipped=1.0
2023-11-20 21:08:45,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1221226.6666666667, ans=0.125
2023-11-20 21:09:01,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183200
2023-11-20 21:09:03,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1221293.3333333333, ans=0.0
2023-11-20 21:09:04,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1221293.3333333333, ans=10.0
2023-11-20 21:09:06,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1221293.3333333333, ans=0.125
2023-11-20 21:09:08,977 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2850, loss[loss=0.0712, simple_loss=0.086, pruned_loss=0.01504, audio_tagging_loss=0.01317, over 14633.00 frames. ], tot_loss[loss=0.07674, simple_loss=0.0976, pruned_loss=0.01821, audio_tagging_loss=0.009732, over 3042362.90 frames. ], batch size: 56, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:09:14,013 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.52 vs. limit=10.0
2023-11-20 21:09:15,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1221360.0, ans=0.125
2023-11-20 21:09:44,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1221493.3333333333, ans=0.2
2023-11-20 21:09:49,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1221560.0, ans=0.125
2023-11-20 21:09:55,958 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.89 vs. limit=15.0
2023-11-20 21:10:05,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183250
2023-11-20 21:10:13,767 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2900, loss[loss=0.09674, simple_loss=0.1294, pruned_loss=0.02528, audio_tagging_loss=0.006786, over 14863.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09915, pruned_loss=0.01859, audio_tagging_loss=0.009672, over 3044987.16 frames. ], batch size: 52, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:10:15,709 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.34 vs. limit=10.0
2023-11-20 21:10:17,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1221693.3333333333, ans=0.125
2023-11-20 21:10:26,051 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.821e+01 8.647e+01 9.509e+01 1.495e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-20 21:10:34,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=1221760.0, ans=15.0
2023-11-20 21:10:48,893 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.288e-01
2023-11-20 21:11:04,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1221960.0, ans=0.0
2023-11-20 21:11:09,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183300
2023-11-20 21:11:15,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1222026.6666666667, ans=0.0
2023-11-20 21:11:17,207 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 2950, loss[loss=0.06998, simple_loss=0.09995, pruned_loss=0.0112, audio_tagging_loss=0.008807, over 14502.00 frames. ], tot_loss[loss=0.07822, simple_loss=0.09961, pruned_loss=0.01874, audio_tagging_loss=0.009681, over 3044155.46 frames. ], batch size: 55, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:11:24,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1222026.6666666667, ans=0.125
2023-11-20 21:12:13,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183350
2023-11-20 21:12:21,094 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3000, loss[loss=0.102, simple_loss=0.1278, pruned_loss=0.02793, audio_tagging_loss=0.0102, over 15504.00 frames. ], tot_loss[loss=0.07794, simple_loss=0.09892, pruned_loss=0.01861, audio_tagging_loss=0.009868, over 3046188.22 frames. ], batch size: 55, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:12:21,097 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-20 21:12:36,698 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.6987, 3.3341, 3.4941, 2.8938, 3.8365, 3.8626, 3.7672, 3.8001],
       device='cuda:0')
2023-11-20 21:12:59,445 INFO [train_asr.py:1253] (0/4) Epoch 16, validation: loss=0.06057, simple_loss=0.053, pruned_loss=0.005481, audio_tagging_loss=0.02859, over 4681554.00 frames. 
2023-11-20 21:12:59,446 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-20 21:13:12,786 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 8.173e+01 8.695e+01 9.591e+01 1.296e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-20 21:13:32,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1222493.3333333333, ans=0.0
2023-11-20 21:13:42,293 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.14 vs. limit=15.0
2023-11-20 21:13:51,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1222626.6666666667, ans=0.1
2023-11-20 21:13:55,684 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183400
2023-11-20 21:14:04,224 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3050, loss[loss=0.08142, simple_loss=0.1036, pruned_loss=0.02019, audio_tagging_loss=0.009438, over 15231.00 frames. ], tot_loss[loss=0.07804, simple_loss=0.09936, pruned_loss=0.01855, audio_tagging_loss=0.009814, over 3045746.86 frames. ], batch size: 57, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:14:26,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1222760.0, ans=0.09899494936611666
2023-11-20 21:14:27,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1222826.6666666667, ans=0.1
2023-11-20 21:14:27,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1222826.6666666667, ans=0.125
2023-11-20 21:14:39,844 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:14:47,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1222893.3333333333, ans=0.1
2023-11-20 21:14:53,713 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.68 vs. limit=15.0
2023-11-20 21:15:00,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183450
2023-11-20 21:15:02,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1222960.0, ans=0.125
2023-11-20 21:15:03,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1222960.0, ans=0.125
2023-11-20 21:15:07,782 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3100, loss[loss=0.08893, simple_loss=0.1159, pruned_loss=0.02319, audio_tagging_loss=0.007776, over 16308.00 frames. ], tot_loss[loss=0.07828, simple_loss=0.09961, pruned_loss=0.01859, audio_tagging_loss=0.009886, over 3054540.70 frames. ], batch size: 60, lr: 4.34e-03, grad_scale: 16.0
2023-11-20 21:15:15,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1223026.6666666667, ans=0.2
2023-11-20 21:15:20,359 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.627e+01 8.321e+01 8.878e+01 9.736e+01 1.416e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-20 21:15:20,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1223093.3333333333, ans=0.125
2023-11-20 21:15:22,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1223093.3333333333, ans=0.1
2023-11-20 21:15:36,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1223160.0, ans=0.125
2023-11-20 21:16:03,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183500
2023-11-20 21:16:10,257 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.35 vs. limit=5.0
2023-11-20 21:16:11,736 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3150, loss[loss=0.06355, simple_loss=0.07654, pruned_loss=0.01329, audio_tagging_loss=0.01199, over 15792.00 frames. ], tot_loss[loss=0.07736, simple_loss=0.09856, pruned_loss=0.01809, audio_tagging_loss=0.00999, over 3051471.44 frames. ], batch size: 62, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:16:14,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1223360.0, ans=0.125
2023-11-20 21:16:16,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1223360.0, ans=10.0
2023-11-20 21:16:19,057 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.97 vs. limit=15.0
2023-11-20 21:16:51,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1223560.0, ans=0.125
2023-11-20 21:17:07,898 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183550
2023-11-20 21:17:16,815 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3200, loss[loss=0.07836, simple_loss=0.1042, pruned_loss=0.01665, audio_tagging_loss=0.00963, over 14681.00 frames. ], tot_loss[loss=0.07756, simple_loss=0.09882, pruned_loss=0.01815, audio_tagging_loss=0.009999, over 3052720.68 frames. ], batch size: 55, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:17:27,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1223760.0, ans=0.0
2023-11-20 21:17:28,834 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.148e+01 8.173e+01 8.835e+01 9.713e+01 1.308e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-20 21:17:55,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1223893.3333333333, ans=0.125
2023-11-20 21:18:10,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1223960.0, ans=0.2
2023-11-20 21:18:12,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183600
2023-11-20 21:18:20,318 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3250, loss[loss=0.06935, simple_loss=0.0863, pruned_loss=0.01675, audio_tagging_loss=0.009454, over 15434.00 frames. ], tot_loss[loss=0.07758, simple_loss=0.09864, pruned_loss=0.01812, audio_tagging_loss=0.01014, over 3045031.13 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:18:24,917 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.86 vs. limit=15.0
2023-11-20 21:18:42,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1224093.3333333333, ans=0.95
2023-11-20 21:18:46,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1224160.0, ans=0.1
2023-11-20 21:18:50,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1224160.0, ans=0.2
2023-11-20 21:18:54,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1224160.0, ans=0.0
2023-11-20 21:18:57,759 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.96 vs. limit=15.0
2023-11-20 21:19:16,935 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183650
2023-11-20 21:19:24,581 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3300, loss[loss=0.09302, simple_loss=0.1288, pruned_loss=0.01948, audio_tagging_loss=0.009124, over 15173.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.09808, pruned_loss=0.01798, audio_tagging_loss=0.01015, over 3045488.39 frames. ], batch size: 56, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:19:38,016 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.036e+01 8.648e+01 9.640e+01 1.721e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-20 21:20:01,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1224560.0, ans=0.0
2023-11-20 21:20:05,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1224560.0, ans=0.1
2023-11-20 21:20:20,146 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183700
2023-11-20 21:20:28,187 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3350, loss[loss=0.06158, simple_loss=0.07888, pruned_loss=0.01303, audio_tagging_loss=0.009109, over 15428.00 frames. ], tot_loss[loss=0.0776, simple_loss=0.09891, pruned_loss=0.01813, audio_tagging_loss=0.01002, over 3043764.14 frames. ], batch size: 58, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:20:28,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1224693.3333333333, ans=0.1
2023-11-20 21:20:37,374 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=22.5
2023-11-20 21:20:52,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1224760.0, ans=0.125
2023-11-20 21:20:53,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1224826.6666666667, ans=0.125
2023-11-20 21:21:02,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1224826.6666666667, ans=0.0
2023-11-20 21:21:05,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=12.0
2023-11-20 21:21:24,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1224960.0, ans=0.2
2023-11-20 21:21:26,541 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183750
2023-11-20 21:21:33,813 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3400, loss[loss=0.0763, simple_loss=0.09894, pruned_loss=0.01693, audio_tagging_loss=0.009901, over 15774.00 frames. ], tot_loss[loss=0.07796, simple_loss=0.09963, pruned_loss=0.01832, audio_tagging_loss=0.009823, over 3045444.20 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:21:47,182 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.496e+01 8.162e+01 8.769e+01 9.534e+01 1.233e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-20 21:21:51,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1225093.3333333333, ans=0.1
2023-11-20 21:22:07,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1225160.0, ans=0.125
2023-11-20 21:22:13,158 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.099e-01
2023-11-20 21:22:15,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1225226.6666666667, ans=0.0
2023-11-20 21:22:29,952 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183800
2023-11-20 21:22:32,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1225293.3333333333, ans=0.125
2023-11-20 21:22:38,212 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3450, loss[loss=0.07239, simple_loss=0.09174, pruned_loss=0.01547, audio_tagging_loss=0.01105, over 16249.00 frames. ], tot_loss[loss=0.07819, simple_loss=0.09992, pruned_loss=0.01849, audio_tagging_loss=0.009735, over 3043522.89 frames. ], batch size: 62, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:22:52,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1225426.6666666667, ans=0.125
2023-11-20 21:22:54,484 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:23:15,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1225560.0, ans=0.0
2023-11-20 21:23:18,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1225560.0, ans=0.125
2023-11-20 21:23:24,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1225560.0, ans=0.125
2023-11-20 21:23:33,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1225626.6666666667, ans=0.2
2023-11-20 21:23:34,381 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183850
2023-11-20 21:23:41,695 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3500, loss[loss=0.0756, simple_loss=0.09424, pruned_loss=0.0185, audio_tagging_loss=0.009981, over 15125.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.0992, pruned_loss=0.01852, audio_tagging_loss=0.009677, over 3039736.72 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:23:50,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1225693.3333333333, ans=0.0
2023-11-20 21:23:57,065 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.353e+01 9.150e+01 1.014e+02 1.535e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-20 21:24:09,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1225826.6666666667, ans=0.0
2023-11-20 21:24:12,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1225826.6666666667, ans=0.0
2023-11-20 21:24:13,365 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:24:20,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1225893.3333333333, ans=0.1
2023-11-20 21:24:39,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183900
2023-11-20 21:24:47,802 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3550, loss[loss=0.07776, simple_loss=0.09305, pruned_loss=0.02232, audio_tagging_loss=0.008911, over 15014.00 frames. ], tot_loss[loss=0.07765, simple_loss=0.09876, pruned_loss=0.01857, audio_tagging_loss=0.0097, over 3037257.20 frames. ], batch size: 55, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:24:49,945 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.44 vs. limit=22.5
2023-11-20 21:24:55,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1226026.6666666667, ans=0.025
2023-11-20 21:24:56,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1226026.6666666667, ans=0.125
2023-11-20 21:25:17,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.11 vs. limit=12.0
2023-11-20 21:25:44,371 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 183950
2023-11-20 21:25:51,789 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3600, loss[loss=0.08209, simple_loss=0.1132, pruned_loss=0.01534, audio_tagging_loss=0.01013, over 16055.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.0981, pruned_loss=0.01835, audio_tagging_loss=0.009755, over 3034477.40 frames. ], batch size: 59, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:26:05,880 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.147e+01 8.022e+01 8.880e+01 9.578e+01 1.162e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-20 21:26:06,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.75 vs. limit=15.0
2023-11-20 21:26:19,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1226493.3333333333, ans=0.0
2023-11-20 21:26:31,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1226560.0, ans=0.015
2023-11-20 21:26:48,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184000
2023-11-20 21:26:50,382 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-184000.pt
2023-11-20 21:27:00,039 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3650, loss[loss=0.06447, simple_loss=0.08153, pruned_loss=0.01429, audio_tagging_loss=0.009418, over 14380.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.09807, pruned_loss=0.01831, audio_tagging_loss=0.00968, over 3035880.39 frames. ], batch size: 54, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:27:28,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1226826.6666666667, ans=0.05
2023-11-20 21:27:30,970 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.82 vs. limit=15.0
2023-11-20 21:27:40,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1226893.3333333333, ans=0.0
2023-11-20 21:27:45,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1226893.3333333333, ans=0.125
2023-11-20 21:27:57,989 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184050
2023-11-20 21:27:59,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1226960.0, ans=0.125
2023-11-20 21:28:00,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1226960.0, ans=0.125
2023-11-20 21:28:05,165 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3700, loss[loss=0.08198, simple_loss=0.08979, pruned_loss=0.02687, audio_tagging_loss=0.01022, over 14668.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.09879, pruned_loss=0.01829, audio_tagging_loss=0.009616, over 3049538.32 frames. ], batch size: 57, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:28:11,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1227026.6666666667, ans=0.09899494936611666
2023-11-20 21:28:13,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1227026.6666666667, ans=0.125
2023-11-20 21:28:18,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1227093.3333333333, ans=0.125
2023-11-20 21:28:18,985 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.63 vs. limit=12.0
2023-11-20 21:28:19,421 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.284e+01 8.009e+01 8.576e+01 9.316e+01 1.305e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-20 21:28:26,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1227093.3333333333, ans=0.125
2023-11-20 21:28:30,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.55 vs. limit=15.0
2023-11-20 21:28:43,582 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.98 vs. limit=15.0
2023-11-20 21:28:45,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1227226.6666666667, ans=0.125
2023-11-20 21:28:48,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1227226.6666666667, ans=0.125
2023-11-20 21:29:02,777 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184100
2023-11-20 21:29:10,250 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3750, loss[loss=0.05144, simple_loss=0.06582, pruned_loss=0.006896, audio_tagging_loss=0.01163, over 15986.00 frames. ], tot_loss[loss=0.07729, simple_loss=0.09843, pruned_loss=0.01833, audio_tagging_loss=0.009745, over 3040641.51 frames. ], batch size: 62, lr: 4.33e-03, grad_scale: 32.0
2023-11-20 21:29:11,988 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=5.69 vs. limit=15.0
2023-11-20 21:29:43,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1227493.3333333333, ans=0.125
2023-11-20 21:29:52,887 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:30:06,718 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184150
2023-11-20 21:30:10,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1227626.6666666667, ans=0.125
2023-11-20 21:30:13,957 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3800, loss[loss=0.05769, simple_loss=0.06954, pruned_loss=0.01008, audio_tagging_loss=0.01284, over 15343.00 frames. ], tot_loss[loss=0.0779, simple_loss=0.09924, pruned_loss=0.01846, audio_tagging_loss=0.009812, over 3042372.67 frames. ], batch size: 58, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:30:25,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1227693.3333333333, ans=0.125
2023-11-20 21:30:29,926 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.193e+01 8.281e+01 9.046e+01 9.670e+01 1.407e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-20 21:31:00,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1227893.3333333333, ans=0.125
2023-11-20 21:31:11,029 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184200
2023-11-20 21:31:11,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1227960.0, ans=0.0
2023-11-20 21:31:17,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1228026.6666666667, ans=0.125
2023-11-20 21:31:18,622 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3850, loss[loss=0.07561, simple_loss=0.09083, pruned_loss=0.02058, audio_tagging_loss=0.009613, over 14315.00 frames. ], tot_loss[loss=0.0772, simple_loss=0.09806, pruned_loss=0.0184, audio_tagging_loss=0.009778, over 3043748.45 frames. ], batch size: 56, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:31:45,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1228160.0, ans=0.07
2023-11-20 21:31:48,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1228160.0, ans=0.04949747468305833
2023-11-20 21:31:56,285 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.67 vs. limit=10.0
2023-11-20 21:31:58,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1228226.6666666667, ans=0.0
2023-11-20 21:32:01,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=1228226.6666666667, ans=6.0
2023-11-20 21:32:15,350 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184250
2023-11-20 21:32:15,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1228293.3333333333, ans=0.0
2023-11-20 21:32:17,056 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.01 vs. limit=10.0
2023-11-20 21:32:23,167 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3900, loss[loss=0.09049, simple_loss=0.1211, pruned_loss=0.02139, audio_tagging_loss=0.008544, over 16312.00 frames. ], tot_loss[loss=0.07729, simple_loss=0.09834, pruned_loss=0.0182, audio_tagging_loss=0.009924, over 3049252.81 frames. ], batch size: 61, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:32:31,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1228360.0, ans=0.1
2023-11-20 21:32:38,154 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.176e+01 8.890e+01 9.815e+01 1.146e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-20 21:32:42,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1228426.6666666667, ans=0.0
2023-11-20 21:33:12,360 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:33:17,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1228626.6666666667, ans=0.0
2023-11-20 21:33:20,113 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184300
2023-11-20 21:33:24,611 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.70 vs. limit=12.0
2023-11-20 21:33:27,626 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 3950, loss[loss=0.08683, simple_loss=0.1098, pruned_loss=0.02142, audio_tagging_loss=0.01049, over 14473.00 frames. ], tot_loss[loss=0.07806, simple_loss=0.09937, pruned_loss=0.01837, audio_tagging_loss=0.01001, over 3051220.33 frames. ], batch size: 56, lr: 4.33e-03, grad_scale: 16.0
2023-11-20 21:33:31,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1228693.3333333333, ans=0.125
2023-11-20 21:33:34,509 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.73 vs. limit=15.0
2023-11-20 21:33:35,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1228693.3333333333, ans=0.0
2023-11-20 21:33:39,970 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.21 vs. limit=22.5
2023-11-20 21:33:43,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1228760.0, ans=0.1
2023-11-20 21:33:56,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1228826.6666666667, ans=0.2
2023-11-20 21:34:01,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1228826.6666666667, ans=0.125
2023-11-20 21:34:02,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1228826.6666666667, ans=0.125
2023-11-20 21:34:14,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1228893.3333333333, ans=0.125
2023-11-20 21:34:23,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1228960.0, ans=0.1
2023-11-20 21:34:24,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184350
2023-11-20 21:34:32,595 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4000, loss[loss=0.08167, simple_loss=0.09322, pruned_loss=0.02365, audio_tagging_loss=0.01141, over 14796.00 frames. ], tot_loss[loss=0.0786, simple_loss=0.09981, pruned_loss=0.01863, audio_tagging_loss=0.01007, over 3043843.27 frames. ], batch size: 55, lr: 4.32e-03, grad_scale: 32.0
2023-11-20 21:34:40,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1229026.6666666667, ans=0.5
2023-11-20 21:34:43,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1229093.3333333333, ans=0.0
2023-11-20 21:34:47,101 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.174e+01 8.766e+01 9.644e+01 1.258e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-20 21:35:13,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1229226.6666666667, ans=0.125
2023-11-20 21:35:13,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.58 vs. limit=15.0
2023-11-20 21:35:24,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1229293.3333333333, ans=0.125
2023-11-20 21:35:28,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184400
2023-11-20 21:35:36,135 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4050, loss[loss=0.07905, simple_loss=0.1021, pruned_loss=0.01777, audio_tagging_loss=0.01021, over 15557.00 frames. ], tot_loss[loss=0.07866, simple_loss=0.09992, pruned_loss=0.01862, audio_tagging_loss=0.01008, over 3043937.24 frames. ], batch size: 57, lr: 4.32e-03, grad_scale: 32.0
2023-11-20 21:35:37,471 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:35:47,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1229360.0, ans=0.2
2023-11-20 21:35:54,691 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.91 vs. limit=15.0
2023-11-20 21:35:55,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1229426.6666666667, ans=0.0
2023-11-20 21:35:56,072 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.24 vs. limit=15.0
2023-11-20 21:36:12,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1229493.3333333333, ans=0.125
2023-11-20 21:36:26,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1229626.6666666667, ans=0.125
2023-11-20 21:36:32,134 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184450
2023-11-20 21:36:35,882 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:36:40,466 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4100, loss[loss=0.102, simple_loss=0.1405, pruned_loss=0.0242, audio_tagging_loss=0.00755, over 16021.00 frames. ], tot_loss[loss=0.07856, simple_loss=0.09995, pruned_loss=0.0185, audio_tagging_loss=0.01009, over 3048226.21 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:36:42,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1229693.3333333333, ans=10.0
2023-11-20 21:36:57,518 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.177e+01 8.908e+01 9.687e+01 1.316e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-20 21:37:18,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1229893.3333333333, ans=0.1
2023-11-20 21:37:20,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1229893.3333333333, ans=0.0
2023-11-20 21:37:36,990 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184500
2023-11-20 21:37:41,710 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.97 vs. limit=6.0
2023-11-20 21:37:44,733 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4150, loss[loss=0.06447, simple_loss=0.07868, pruned_loss=0.01643, audio_tagging_loss=0.008697, over 14315.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09918, pruned_loss=0.01828, audio_tagging_loss=0.009957, over 3047068.27 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:37:50,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1230026.6666666667, ans=0.05
2023-11-20 21:37:58,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1230093.3333333333, ans=0.1
2023-11-20 21:38:07,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1230093.3333333333, ans=0.0
2023-11-20 21:38:08,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1230093.3333333333, ans=0.2
2023-11-20 21:38:08,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1230093.3333333333, ans=0.2
2023-11-20 21:38:20,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1230160.0, ans=0.1
2023-11-20 21:38:28,797 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:38:41,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1230293.3333333333, ans=0.1
2023-11-20 21:38:42,163 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184550
2023-11-20 21:38:42,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1230293.3333333333, ans=0.125
2023-11-20 21:38:44,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1230293.3333333333, ans=0.125
2023-11-20 21:38:49,322 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4200, loss[loss=0.05996, simple_loss=0.0771, pruned_loss=0.01149, audio_tagging_loss=0.009916, over 16175.00 frames. ], tot_loss[loss=0.07705, simple_loss=0.09832, pruned_loss=0.01807, audio_tagging_loss=0.009821, over 3045755.60 frames. ], batch size: 61, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:39:07,579 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.428e+01 8.014e+01 8.553e+01 9.363e+01 1.203e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-20 21:39:18,114 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.46 vs. limit=12.0
2023-11-20 21:39:31,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1230560.0, ans=0.125
2023-11-20 21:39:45,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184600
2023-11-20 21:39:45,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1230626.6666666667, ans=0.125
2023-11-20 21:39:48,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1230626.6666666667, ans=0.2
2023-11-20 21:39:52,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1230693.3333333333, ans=0.1
2023-11-20 21:39:53,598 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4250, loss[loss=0.09568, simple_loss=0.1293, pruned_loss=0.02433, audio_tagging_loss=0.006693, over 15411.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09884, pruned_loss=0.01817, audio_tagging_loss=0.009779, over 3044626.68 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 8.0
2023-11-20 21:40:00,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1230693.3333333333, ans=0.125
2023-11-20 21:40:10,005 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.68 vs. limit=22.5
2023-11-20 21:40:28,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1230826.6666666667, ans=0.125
2023-11-20 21:40:50,048 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184650
2023-11-20 21:40:57,905 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4300, loss[loss=0.08626, simple_loss=0.1012, pruned_loss=0.02707, audio_tagging_loss=0.008602, over 14732.00 frames. ], tot_loss[loss=0.07747, simple_loss=0.09893, pruned_loss=0.01832, audio_tagging_loss=0.009687, over 3049401.64 frames. ], batch size: 56, lr: 4.32e-03, grad_scale: 8.0
2023-11-20 21:41:00,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1231026.6666666667, ans=0.95
2023-11-20 21:41:04,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1231026.6666666667, ans=0.125
2023-11-20 21:41:15,307 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.115e+01 8.273e+01 8.996e+01 9.641e+01 1.140e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-20 21:41:33,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1231160.0, ans=0.125
2023-11-20 21:41:54,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184700
2023-11-20 21:42:01,390 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4350, loss[loss=0.07915, simple_loss=0.1065, pruned_loss=0.01708, audio_tagging_loss=0.008837, over 14528.00 frames. ], tot_loss[loss=0.07731, simple_loss=0.09859, pruned_loss=0.01822, audio_tagging_loss=0.009799, over 3047027.85 frames. ], batch size: 54, lr: 4.32e-03, grad_scale: 8.0
2023-11-20 21:42:07,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1231360.0, ans=0.125
2023-11-20 21:42:10,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1231360.0, ans=0.1
2023-11-20 21:42:26,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.11 vs. limit=15.0
2023-11-20 21:42:35,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1231493.3333333333, ans=0.0
2023-11-20 21:42:36,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1231493.3333333333, ans=0.125
2023-11-20 21:42:52,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1231626.6666666667, ans=0.2
2023-11-20 21:42:57,351 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184750
2023-11-20 21:42:59,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1231626.6666666667, ans=0.1
2023-11-20 21:43:05,105 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4400, loss[loss=0.06105, simple_loss=0.07519, pruned_loss=0.01253, audio_tagging_loss=0.01093, over 15063.00 frames. ], tot_loss[loss=0.07726, simple_loss=0.09856, pruned_loss=0.01822, audio_tagging_loss=0.009767, over 3039371.25 frames. ], batch size: 57, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:43:22,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1231760.0, ans=0.0
2023-11-20 21:43:23,435 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.140e+01 8.871e+01 9.490e+01 1.181e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-20 21:43:31,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1231826.6666666667, ans=0.125
2023-11-20 21:43:37,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1231826.6666666667, ans=0.0
2023-11-20 21:43:56,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1231960.0, ans=0.125
2023-11-20 21:43:57,109 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.60 vs. limit=22.5
2023-11-20 21:44:01,697 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184800
2023-11-20 21:44:07,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1231960.0, ans=0.1
2023-11-20 21:44:09,931 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4450, loss[loss=0.07209, simple_loss=0.08966, pruned_loss=0.01594, audio_tagging_loss=0.01132, over 14854.00 frames. ], tot_loss[loss=0.07805, simple_loss=0.09977, pruned_loss=0.01857, audio_tagging_loss=0.009597, over 3038242.26 frames. ], batch size: 55, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:44:28,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1232093.3333333333, ans=0.1
2023-11-20 21:44:57,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1232226.6666666667, ans=0.2
2023-11-20 21:45:04,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1232293.3333333333, ans=0.2
2023-11-20 21:45:07,713 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184850
2023-11-20 21:45:07,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1232293.3333333333, ans=0.125
2023-11-20 21:45:15,001 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4500, loss[loss=0.09253, simple_loss=0.1123, pruned_loss=0.02642, audio_tagging_loss=0.009957, over 13931.00 frames. ], tot_loss[loss=0.07821, simple_loss=0.1, pruned_loss=0.01855, audio_tagging_loss=0.009646, over 3036498.55 frames. ], batch size: 53, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:45:15,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1232360.0, ans=0.1
2023-11-20 21:45:32,800 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.142e+01 8.902e+01 9.690e+01 1.395e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-20 21:45:52,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1232560.0, ans=0.125
2023-11-20 21:45:55,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1232560.0, ans=0.1
2023-11-20 21:46:08,098 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:46:11,426 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184900
2023-11-20 21:46:11,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1232626.6666666667, ans=0.0
2023-11-20 21:46:18,661 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4550, loss[loss=0.08096, simple_loss=0.09971, pruned_loss=0.02022, audio_tagging_loss=0.01089, over 14896.00 frames. ], tot_loss[loss=0.07727, simple_loss=0.09869, pruned_loss=0.01828, audio_tagging_loss=0.009646, over 3036159.64 frames. ], batch size: 55, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:46:50,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1232826.6666666667, ans=0.5
2023-11-20 21:47:05,287 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 21:47:12,611 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=14.38 vs. limit=15.0
2023-11-20 21:47:15,661 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 184950
2023-11-20 21:47:22,850 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4600, loss[loss=0.09192, simple_loss=0.121, pruned_loss=0.02445, audio_tagging_loss=0.006986, over 15404.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.09807, pruned_loss=0.01833, audio_tagging_loss=0.00979, over 3034098.26 frames. ], batch size: 59, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:47:36,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1233093.3333333333, ans=0.125
2023-11-20 21:47:41,016 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.813e+01 7.895e+01 8.619e+01 9.471e+01 1.250e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-20 21:47:46,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.38 vs. limit=15.0
2023-11-20 21:47:52,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1233160.0, ans=0.125
2023-11-20 21:47:56,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1233160.0, ans=0.0
2023-11-20 21:48:15,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1233293.3333333333, ans=22.5
2023-11-20 21:48:19,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185000
2023-11-20 21:48:27,591 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4650, loss[loss=0.09535, simple_loss=0.1294, pruned_loss=0.02232, audio_tagging_loss=0.008318, over 15629.00 frames. ], tot_loss[loss=0.07667, simple_loss=0.09721, pruned_loss=0.01817, audio_tagging_loss=0.009895, over 3029735.79 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:48:33,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1233360.0, ans=0.0
2023-11-20 21:48:39,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1233426.6666666667, ans=0.125
2023-11-20 21:49:23,541 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185050
2023-11-20 21:49:30,680 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4700, loss[loss=0.09807, simple_loss=0.1163, pruned_loss=0.03007, audio_tagging_loss=0.00986, over 14624.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09852, pruned_loss=0.01844, audio_tagging_loss=0.009934, over 3038181.65 frames. ], batch size: 52, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:49:48,744 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.970e+01 8.027e+01 8.804e+01 9.916e+01 1.555e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-20 21:50:11,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1233893.3333333333, ans=0.0
2023-11-20 21:50:15,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1233893.3333333333, ans=0.125
2023-11-20 21:50:15,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1233893.3333333333, ans=0.0
2023-11-20 21:50:28,632 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185100
2023-11-20 21:50:35,791 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4750, loss[loss=0.0902, simple_loss=0.1155, pruned_loss=0.0248, audio_tagging_loss=0.007655, over 15824.00 frames. ], tot_loss[loss=0.07775, simple_loss=0.09864, pruned_loss=0.01838, audio_tagging_loss=0.01005, over 3035651.13 frames. ], batch size: 58, lr: 4.32e-03, grad_scale: 16.0
2023-11-20 21:50:53,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1234093.3333333333, ans=0.0
2023-11-20 21:51:02,104 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:51:14,873 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:51:19,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1234226.6666666667, ans=0.1
2023-11-20 21:51:32,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185150
2023-11-20 21:51:32,820 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.76 vs. limit=22.5
2023-11-20 21:51:39,955 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4800, loss[loss=0.08059, simple_loss=0.1032, pruned_loss=0.01745, audio_tagging_loss=0.01155, over 13781.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.09764, pruned_loss=0.01807, audio_tagging_loss=0.01026, over 3037517.00 frames. ], batch size: 53, lr: 4.32e-03, grad_scale: 32.0
2023-11-20 21:51:40,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1234360.0, ans=0.0
2023-11-20 21:51:54,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1234426.6666666667, ans=0.125
2023-11-20 21:51:55,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1234426.6666666667, ans=0.0
2023-11-20 21:51:58,551 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.531e+01 9.078e+01 1.027e+02 1.282e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-20 21:52:07,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1234493.3333333333, ans=0.0
2023-11-20 21:52:11,810 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:52:22,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.77 vs. limit=22.5
2023-11-20 21:52:26,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1234560.0, ans=0.1
2023-11-20 21:52:28,569 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:52:35,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185200
2023-11-20 21:52:44,115 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4850, loss[loss=0.06918, simple_loss=0.08607, pruned_loss=0.01352, audio_tagging_loss=0.01263, over 16445.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09764, pruned_loss=0.01822, audio_tagging_loss=0.01033, over 3038204.18 frames. ], batch size: 62, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 21:52:56,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1234760.0, ans=0.125
2023-11-20 21:52:59,989 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.70 vs. limit=12.0
2023-11-20 21:53:03,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.84 vs. limit=12.0
2023-11-20 21:53:20,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1234826.6666666667, ans=0.0
2023-11-20 21:53:40,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185250
2023-11-20 21:53:47,783 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4900, loss[loss=0.07405, simple_loss=0.1012, pruned_loss=0.01532, audio_tagging_loss=0.008147, over 15779.00 frames. ], tot_loss[loss=0.07738, simple_loss=0.09789, pruned_loss=0.0182, audio_tagging_loss=0.01023, over 3041654.84 frames. ], batch size: 60, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 21:53:55,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1235026.6666666667, ans=0.0
2023-11-20 21:53:57,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1235026.6666666667, ans=0.125
2023-11-20 21:54:06,866 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.089e+01 8.842e+01 9.821e+01 1.319e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 21:54:20,166 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:54:43,771 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185300
2023-11-20 21:54:48,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1235293.3333333333, ans=0.125
2023-11-20 21:54:49,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1235293.3333333333, ans=0.2
2023-11-20 21:54:50,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1235360.0, ans=0.125
2023-11-20 21:54:52,117 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 4950, loss[loss=0.0763, simple_loss=0.1002, pruned_loss=0.01923, audio_tagging_loss=0.006974, over 15054.00 frames. ], tot_loss[loss=0.07729, simple_loss=0.09795, pruned_loss=0.01823, audio_tagging_loss=0.01008, over 3043189.94 frames. ], batch size: 56, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 21:54:55,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1235360.0, ans=0.0
2023-11-20 21:55:02,527 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.22 vs. limit=15.0
2023-11-20 21:55:13,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1235426.6666666667, ans=0.0
2023-11-20 21:55:16,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1235493.3333333333, ans=0.125
2023-11-20 21:55:21,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1235493.3333333333, ans=0.0
2023-11-20 21:55:48,239 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185350
2023-11-20 21:55:55,706 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5000, loss[loss=0.09119, simple_loss=0.1276, pruned_loss=0.01948, audio_tagging_loss=0.007939, over 15481.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09894, pruned_loss=0.01837, audio_tagging_loss=0.009863, over 3046147.47 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 21:56:08,766 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.75 vs. limit=6.0
2023-11-20 21:56:16,477 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.618e+01 7.784e+01 8.478e+01 9.107e+01 1.087e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-20 21:56:20,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1235826.6666666667, ans=0.2
2023-11-20 21:56:46,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1235960.0, ans=0.0
2023-11-20 21:56:47,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1235960.0, ans=0.1
2023-11-20 21:56:47,375 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 21:56:52,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185400
2023-11-20 21:56:57,962 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.37 vs. limit=6.0
2023-11-20 21:57:01,015 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5050, loss[loss=0.06568, simple_loss=0.09054, pruned_loss=0.01374, audio_tagging_loss=0.006673, over 15165.00 frames. ], tot_loss[loss=0.07712, simple_loss=0.09825, pruned_loss=0.01826, audio_tagging_loss=0.00973, over 3054172.01 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 8.0
2023-11-20 21:57:03,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1236026.6666666667, ans=0.2
2023-11-20 21:57:12,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1236093.3333333333, ans=0.0
2023-11-20 21:57:15,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1236093.3333333333, ans=0.0
2023-11-20 21:57:37,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1236226.6666666667, ans=0.0
2023-11-20 21:57:42,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1236226.6666666667, ans=0.125
2023-11-20 21:57:51,881 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.85 vs. limit=15.0
2023-11-20 21:57:57,566 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185450
2023-11-20 21:58:05,603 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5100, loss[loss=0.0942, simple_loss=0.121, pruned_loss=0.02322, audio_tagging_loss=0.0105, over 14964.00 frames. ], tot_loss[loss=0.07808, simple_loss=0.09953, pruned_loss=0.01858, audio_tagging_loss=0.009732, over 3035390.73 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 8.0
2023-11-20 21:58:09,668 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.02 vs. limit=15.0
2023-11-20 21:58:25,680 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.564e+01 8.354e+01 9.032e+01 1.014e+02 3.240e+02, threshold=1.806e+02, percent-clipped=1.0
2023-11-20 21:58:43,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1236560.0, ans=0.125
2023-11-20 21:58:44,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.17 vs. limit=15.0
2023-11-20 21:59:02,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185500
2023-11-20 21:59:05,220 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.91 vs. limit=6.0
2023-11-20 21:59:09,305 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5150, loss[loss=0.07767, simple_loss=0.1046, pruned_loss=0.01798, audio_tagging_loss=0.007391, over 15694.00 frames. ], tot_loss[loss=0.07777, simple_loss=0.09875, pruned_loss=0.01861, audio_tagging_loss=0.009793, over 3034544.72 frames. ], batch size: 59, lr: 4.31e-03, grad_scale: 8.0
2023-11-20 21:59:25,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1236760.0, ans=0.1
2023-11-20 21:59:40,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1236826.6666666667, ans=0.125
2023-11-20 22:00:05,812 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185550
2023-11-20 22:00:14,344 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5200, loss[loss=0.06944, simple_loss=0.08474, pruned_loss=0.01451, audio_tagging_loss=0.01256, over 15652.00 frames. ], tot_loss[loss=0.07786, simple_loss=0.09929, pruned_loss=0.0185, audio_tagging_loss=0.009714, over 3033376.66 frames. ], batch size: 59, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:00:18,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1237026.6666666667, ans=0.125
2023-11-20 22:00:19,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1237026.6666666667, ans=0.0
2023-11-20 22:00:25,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1237093.3333333333, ans=0.125
2023-11-20 22:00:28,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1237093.3333333333, ans=0.1
2023-11-20 22:00:31,555 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.89 vs. limit=12.0
2023-11-20 22:00:34,955 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.910e+01 8.252e+01 8.979e+01 9.707e+01 1.443e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-20 22:00:52,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1237226.6666666667, ans=0.1
2023-11-20 22:01:01,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1237226.6666666667, ans=0.1
2023-11-20 22:01:10,308 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185600
2023-11-20 22:01:12,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1237293.3333333333, ans=0.2
2023-11-20 22:01:18,597 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5250, loss[loss=0.08545, simple_loss=0.1066, pruned_loss=0.02251, audio_tagging_loss=0.00963, over 15238.00 frames. ], tot_loss[loss=0.07712, simple_loss=0.09813, pruned_loss=0.01828, audio_tagging_loss=0.009769, over 3032179.20 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:01:21,735 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.86 vs. limit=15.0
2023-11-20 22:01:37,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1237426.6666666667, ans=0.125
2023-11-20 22:01:54,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1237493.3333333333, ans=0.0
2023-11-20 22:02:01,604 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.35 vs. limit=15.0
2023-11-20 22:02:06,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1237560.0, ans=0.5
2023-11-20 22:02:15,574 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185650
2023-11-20 22:02:22,648 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5300, loss[loss=0.07918, simple_loss=0.09165, pruned_loss=0.0233, audio_tagging_loss=0.01006, over 13809.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09821, pruned_loss=0.01831, audio_tagging_loss=0.009715, over 3034173.79 frames. ], batch size: 54, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:02:42,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.01 vs. limit=6.0
2023-11-20 22:02:42,488 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.578e+01 8.164e+01 8.663e+01 9.355e+01 1.569e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-20 22:02:47,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1237826.6666666667, ans=0.125
2023-11-20 22:03:06,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1237893.3333333333, ans=0.1
2023-11-20 22:03:18,382 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185700
2023-11-20 22:03:22,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1237960.0, ans=0.125
2023-11-20 22:03:22,829 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.06 vs. limit=15.0
2023-11-20 22:03:26,268 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5350, loss[loss=0.0805, simple_loss=0.1057, pruned_loss=0.02085, audio_tagging_loss=0.006776, over 14567.00 frames. ], tot_loss[loss=0.07749, simple_loss=0.09884, pruned_loss=0.01837, audio_tagging_loss=0.009705, over 3039062.83 frames. ], batch size: 54, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:03:33,899 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.03 vs. limit=15.0
2023-11-20 22:04:04,204 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:04:11,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1238226.6666666667, ans=0.0
2023-11-20 22:04:19,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1238293.3333333333, ans=10.0
2023-11-20 22:04:23,041 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185750
2023-11-20 22:04:23,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1238293.3333333333, ans=0.125
2023-11-20 22:04:25,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1238293.3333333333, ans=0.0
2023-11-20 22:04:26,509 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.81 vs. limit=6.0
2023-11-20 22:04:31,040 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5400, loss[loss=0.05662, simple_loss=0.06089, pruned_loss=0.01369, audio_tagging_loss=0.01248, over 14036.00 frames. ], tot_loss[loss=0.07799, simple_loss=0.09958, pruned_loss=0.01846, audio_tagging_loss=0.009737, over 3047985.21 frames. ], batch size: 55, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:04:41,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1238360.0, ans=0.0
2023-11-20 22:04:48,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1238426.6666666667, ans=0.125
2023-11-20 22:04:51,611 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.664e+01 7.937e+01 8.475e+01 9.191e+01 1.261e+02, threshold=1.695e+02, percent-clipped=0.0
2023-11-20 22:05:07,472 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.11 vs. limit=15.0
2023-11-20 22:05:09,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=1238560.0, ans=0.05
2023-11-20 22:05:28,237 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185800
2023-11-20 22:05:32,805 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.85 vs. limit=15.0
2023-11-20 22:05:35,796 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5450, loss[loss=0.06792, simple_loss=0.07782, pruned_loss=0.02016, audio_tagging_loss=0.008847, over 14443.00 frames. ], tot_loss[loss=0.0781, simple_loss=0.09939, pruned_loss=0.01868, audio_tagging_loss=0.009724, over 3042662.64 frames. ], batch size: 53, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:05:44,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1238693.3333333333, ans=0.09899494936611666
2023-11-20 22:05:54,316 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.63 vs. limit=22.5
2023-11-20 22:06:03,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1238826.6666666667, ans=0.0
2023-11-20 22:06:05,755 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.00 vs. limit=22.5
2023-11-20 22:06:08,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1238826.6666666667, ans=0.125
2023-11-20 22:06:12,152 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.72 vs. limit=22.5
2023-11-20 22:06:14,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1238893.3333333333, ans=0.125
2023-11-20 22:06:31,501 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185850
2023-11-20 22:06:35,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1238960.0, ans=0.125
2023-11-20 22:06:38,739 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5500, loss[loss=0.07679, simple_loss=0.08672, pruned_loss=0.02152, audio_tagging_loss=0.01191, over 15499.00 frames. ], tot_loss[loss=0.07772, simple_loss=0.09902, pruned_loss=0.01847, audio_tagging_loss=0.009742, over 3045879.87 frames. ], batch size: 59, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:06:58,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1239093.3333333333, ans=0.125
2023-11-20 22:07:00,047 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.304e+01 8.468e+01 9.055e+01 9.983e+01 1.429e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-20 22:07:04,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1239160.0, ans=0.125
2023-11-20 22:07:17,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1239226.6666666667, ans=0.125
2023-11-20 22:07:35,561 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185900
2023-11-20 22:07:42,621 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5550, loss[loss=0.08265, simple_loss=0.1068, pruned_loss=0.0194, audio_tagging_loss=0.009826, over 14726.00 frames. ], tot_loss[loss=0.07763, simple_loss=0.09851, pruned_loss=0.01848, audio_tagging_loss=0.009894, over 3047946.24 frames. ], batch size: 56, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:07:50,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1239360.0, ans=0.125
2023-11-20 22:07:52,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1239360.0, ans=0.125
2023-11-20 22:08:31,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1239560.0, ans=0.125
2023-11-20 22:08:34,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1239626.6666666667, ans=0.1
2023-11-20 22:08:36,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1239626.6666666667, ans=0.125
2023-11-20 22:08:37,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1239626.6666666667, ans=0.125
2023-11-20 22:08:40,172 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 185950
2023-11-20 22:08:45,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1239626.6666666667, ans=0.125
2023-11-20 22:08:48,142 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5600, loss[loss=0.07529, simple_loss=0.0948, pruned_loss=0.01789, audio_tagging_loss=0.01, over 15262.00 frames. ], tot_loss[loss=0.07751, simple_loss=0.09826, pruned_loss=0.01838, audio_tagging_loss=0.009996, over 3050403.82 frames. ], batch size: 57, lr: 4.31e-03, grad_scale: 32.0
2023-11-20 22:09:08,125 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 7.915e+01 8.506e+01 9.381e+01 1.103e+02, threshold=1.701e+02, percent-clipped=0.0
2023-11-20 22:09:19,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1239826.6666666667, ans=0.0
2023-11-20 22:09:23,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.30 vs. limit=12.0
2023-11-20 22:09:31,245 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 22:09:43,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186000
2023-11-20 22:09:43,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1239960.0, ans=0.125
2023-11-20 22:09:50,699 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5650, loss[loss=0.06493, simple_loss=0.07214, pruned_loss=0.01449, audio_tagging_loss=0.01437, over 15879.00 frames. ], tot_loss[loss=0.07786, simple_loss=0.09881, pruned_loss=0.01848, audio_tagging_loss=0.009975, over 3057213.07 frames. ], batch size: 62, lr: 4.31e-03, grad_scale: 32.0
2023-11-20 22:10:10,377 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.61 vs. limit=22.5
2023-11-20 22:10:26,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1240160.0, ans=0.0
2023-11-20 22:10:31,229 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.91 vs. limit=15.0
2023-11-20 22:10:47,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186050
2023-11-20 22:10:54,513 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5700, loss[loss=0.1026, simple_loss=0.1263, pruned_loss=0.02776, audio_tagging_loss=0.01171, over 14918.00 frames. ], tot_loss[loss=0.07797, simple_loss=0.09896, pruned_loss=0.01854, audio_tagging_loss=0.009945, over 3050427.83 frames. ], batch size: 54, lr: 4.31e-03, grad_scale: 16.0
2023-11-20 22:10:58,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1240360.0, ans=0.125
2023-11-20 22:11:16,792 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.611e+01 8.091e+01 8.750e+01 9.672e+01 1.332e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-20 22:11:18,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1240426.6666666667, ans=0.125
2023-11-20 22:11:32,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1240560.0, ans=0.0
2023-11-20 22:11:40,938 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:11:48,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1240626.6666666667, ans=0.125
2023-11-20 22:11:51,041 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186100
2023-11-20 22:11:51,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1240626.6666666667, ans=0.04949747468305833
2023-11-20 22:11:58,997 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5750, loss[loss=0.06692, simple_loss=0.08089, pruned_loss=0.01352, audio_tagging_loss=0.01295, over 14213.00 frames. ], tot_loss[loss=0.07722, simple_loss=0.09774, pruned_loss=0.01843, audio_tagging_loss=0.00992, over 3051096.98 frames. ], batch size: 55, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:12:12,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1240760.0, ans=0.0
2023-11-20 22:12:16,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1240760.0, ans=0.125
2023-11-20 22:12:25,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1240826.6666666667, ans=0.125
2023-11-20 22:12:32,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1240826.6666666667, ans=0.1
2023-11-20 22:12:32,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1240826.6666666667, ans=0.0
2023-11-20 22:12:48,660 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.29 vs. limit=22.5
2023-11-20 22:12:55,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186150
2023-11-20 22:13:00,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1240960.0, ans=0.125
2023-11-20 22:13:02,405 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5800, loss[loss=0.06422, simple_loss=0.08247, pruned_loss=0.01115, audio_tagging_loss=0.01184, over 13854.00 frames. ], tot_loss[loss=0.07657, simple_loss=0.09683, pruned_loss=0.01823, audio_tagging_loss=0.00992, over 3050973.96 frames. ], batch size: 52, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:13:09,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1241026.6666666667, ans=0.1
2023-11-20 22:13:21,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.84 vs. limit=22.5
2023-11-20 22:13:23,633 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.344e+01 8.039e+01 8.914e+01 9.519e+01 1.369e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-20 22:13:41,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1241226.6666666667, ans=0.0
2023-11-20 22:13:58,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186200
2023-11-20 22:14:01,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1241293.3333333333, ans=0.125
2023-11-20 22:14:04,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1241293.3333333333, ans=0.0
2023-11-20 22:14:04,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1241293.3333333333, ans=0.1
2023-11-20 22:14:05,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1241360.0, ans=0.125
2023-11-20 22:14:06,353 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5850, loss[loss=0.08461, simple_loss=0.1028, pruned_loss=0.02162, audio_tagging_loss=0.01162, over 14352.00 frames. ], tot_loss[loss=0.07639, simple_loss=0.09688, pruned_loss=0.01812, audio_tagging_loss=0.009829, over 3048228.97 frames. ], batch size: 54, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:14:07,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1241360.0, ans=0.09899494936611666
2023-11-20 22:14:11,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1241360.0, ans=0.0
2023-11-20 22:14:12,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1241360.0, ans=0.09899494936611666
2023-11-20 22:14:23,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1241426.6666666667, ans=0.125
2023-11-20 22:14:23,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1241426.6666666667, ans=0.125
2023-11-20 22:14:55,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1241560.0, ans=0.125
2023-11-20 22:14:55,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1241560.0, ans=0.1
2023-11-20 22:14:55,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1241560.0, ans=0.125
2023-11-20 22:14:59,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1241626.6666666667, ans=0.09899494936611666
2023-11-20 22:15:02,777 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186250
2023-11-20 22:15:03,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1241626.6666666667, ans=0.95
2023-11-20 22:15:10,120 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.15 vs. limit=10.0
2023-11-20 22:15:11,175 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5900, loss[loss=0.06253, simple_loss=0.08475, pruned_loss=0.0139, audio_tagging_loss=0.006254, over 15154.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09687, pruned_loss=0.01794, audio_tagging_loss=0.009794, over 3047359.30 frames. ], batch size: 59, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:15:12,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1241693.3333333333, ans=0.125
2023-11-20 22:15:32,173 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.423e+01 8.032e+01 8.653e+01 9.726e+01 1.327e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-20 22:15:39,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1241826.6666666667, ans=0.0
2023-11-20 22:15:52,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1241893.3333333333, ans=0.125
2023-11-20 22:16:06,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186300
2023-11-20 22:16:14,540 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 5950, loss[loss=0.09947, simple_loss=0.1104, pruned_loss=0.03214, audio_tagging_loss=0.01213, over 14477.00 frames. ], tot_loss[loss=0.07683, simple_loss=0.09784, pruned_loss=0.0182, audio_tagging_loss=0.009709, over 3056748.89 frames. ], batch size: 55, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:16:36,640 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.08 vs. limit=22.5
2023-11-20 22:16:41,807 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.84 vs. limit=15.0
2023-11-20 22:16:47,491 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:16:53,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1242226.6666666667, ans=0.1
2023-11-20 22:17:11,496 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186350
2023-11-20 22:17:19,229 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6000, loss[loss=0.07428, simple_loss=0.1043, pruned_loss=0.0138, audio_tagging_loss=0.008303, over 15827.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.09828, pruned_loss=0.01817, audio_tagging_loss=0.009708, over 3059578.17 frames. ], batch size: 60, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:17:19,233 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-20 22:17:55,338 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([0.8948, 3.1715, 2.8368, 2.8141, 3.4441, 3.4785, 3.1469, 3.5710],
       device='cuda:0')
2023-11-20 22:18:03,618 INFO [train_asr.py:1253] (0/4) Epoch 16, validation: loss=0.06177, simple_loss=0.05296, pruned_loss=0.005445, audio_tagging_loss=0.02985, over 4681554.00 frames. 
2023-11-20 22:18:03,619 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-20 22:18:14,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1242360.0, ans=0.0
2023-11-20 22:18:16,048 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.88 vs. limit=15.0
2023-11-20 22:18:21,667 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:18:25,407 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.385e+01 8.060e+01 8.625e+01 9.633e+01 1.979e+02, threshold=1.725e+02, percent-clipped=1.0
2023-11-20 22:18:44,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1242560.0, ans=0.0
2023-11-20 22:18:45,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1242560.0, ans=0.125
2023-11-20 22:18:47,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1242560.0, ans=0.125
2023-11-20 22:18:47,833 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 22:18:58,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1242626.6666666667, ans=0.0
2023-11-20 22:18:59,536 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186400
2023-11-20 22:19:07,789 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6050, loss[loss=0.08977, simple_loss=0.1204, pruned_loss=0.0224, audio_tagging_loss=0.007168, over 14571.00 frames. ], tot_loss[loss=0.077, simple_loss=0.09837, pruned_loss=0.01818, audio_tagging_loss=0.009634, over 3051016.18 frames. ], batch size: 52, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:19:42,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.33 vs. limit=15.0
2023-11-20 22:19:46,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1242893.3333333333, ans=0.2
2023-11-20 22:20:04,999 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186450
2023-11-20 22:20:12,058 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6100, loss[loss=0.07241, simple_loss=0.09149, pruned_loss=0.01567, audio_tagging_loss=0.01099, over 14349.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09839, pruned_loss=0.01827, audio_tagging_loss=0.009674, over 3051067.03 frames. ], batch size: 57, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:20:17,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1243026.6666666667, ans=0.125
2023-11-20 22:20:25,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1243093.3333333333, ans=0.125
2023-11-20 22:20:34,065 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.632e+01 8.005e+01 8.728e+01 9.355e+01 1.254e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-20 22:20:40,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1243160.0, ans=0.2
2023-11-20 22:20:51,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1243226.6666666667, ans=0.0
2023-11-20 22:21:08,396 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186500
2023-11-20 22:21:16,653 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6150, loss[loss=0.08987, simple_loss=0.1236, pruned_loss=0.01898, audio_tagging_loss=0.009096, over 15965.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09927, pruned_loss=0.01858, audio_tagging_loss=0.009618, over 3051225.63 frames. ], batch size: 58, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:21:30,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1243426.6666666667, ans=0.0
2023-11-20 22:21:45,738 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.72 vs. limit=15.0
2023-11-20 22:21:59,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1243560.0, ans=0.0
2023-11-20 22:22:01,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1243560.0, ans=0.125
2023-11-20 22:22:05,797 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:22:12,923 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186550
2023-11-20 22:22:20,057 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6200, loss[loss=0.09403, simple_loss=0.1211, pruned_loss=0.02443, audio_tagging_loss=0.009045, over 15273.00 frames. ], tot_loss[loss=0.07785, simple_loss=0.09924, pruned_loss=0.0185, audio_tagging_loss=0.009729, over 3047739.36 frames. ], batch size: 55, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:22:22,551 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.17 vs. limit=15.0
2023-11-20 22:22:42,088 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.113e+01 8.838e+01 9.519e+01 1.384e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 22:22:44,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1243826.6666666667, ans=0.125
2023-11-20 22:23:03,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1243893.3333333333, ans=0.0
2023-11-20 22:23:17,840 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186600
2023-11-20 22:23:25,497 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6250, loss[loss=0.08996, simple_loss=0.1154, pruned_loss=0.02239, audio_tagging_loss=0.00985, over 13998.00 frames. ], tot_loss[loss=0.07757, simple_loss=0.09831, pruned_loss=0.01851, audio_tagging_loss=0.009904, over 3055782.06 frames. ], batch size: 53, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:23:59,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1244160.0, ans=0.0
2023-11-20 22:24:02,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1244226.6666666667, ans=0.125
2023-11-20 22:24:21,350 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186650
2023-11-20 22:24:27,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1244360.0, ans=0.125
2023-11-20 22:24:29,285 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6300, loss[loss=0.0929, simple_loss=0.1253, pruned_loss=0.02092, audio_tagging_loss=0.009351, over 15342.00 frames. ], tot_loss[loss=0.07755, simple_loss=0.09805, pruned_loss=0.01855, audio_tagging_loss=0.00998, over 3049307.51 frames. ], batch size: 58, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:24:30,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1244360.0, ans=0.125
2023-11-20 22:24:35,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1244360.0, ans=0.125
2023-11-20 22:24:42,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1244426.6666666667, ans=0.125
2023-11-20 22:24:43,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1244426.6666666667, ans=0.125
2023-11-20 22:24:47,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1244426.6666666667, ans=0.0
2023-11-20 22:24:51,669 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.323e+01 8.066e+01 8.583e+01 9.288e+01 1.245e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-20 22:25:11,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1244560.0, ans=0.125
2023-11-20 22:25:16,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1244560.0, ans=0.125
2023-11-20 22:25:24,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.09 vs. limit=10.0
2023-11-20 22:25:25,516 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186700
2023-11-20 22:25:32,757 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6350, loss[loss=0.07973, simple_loss=0.1027, pruned_loss=0.01915, audio_tagging_loss=0.009226, over 15678.00 frames. ], tot_loss[loss=0.0774, simple_loss=0.09825, pruned_loss=0.01828, audio_tagging_loss=0.009997, over 3049117.49 frames. ], batch size: 60, lr: 4.30e-03, grad_scale: 16.0
2023-11-20 22:25:40,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.72 vs. limit=10.0
2023-11-20 22:26:02,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1244826.6666666667, ans=0.1
2023-11-20 22:26:29,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186750
2023-11-20 22:26:37,569 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6400, loss[loss=0.07656, simple_loss=0.08431, pruned_loss=0.01957, audio_tagging_loss=0.01483, over 14487.00 frames. ], tot_loss[loss=0.07745, simple_loss=0.09797, pruned_loss=0.01828, audio_tagging_loss=0.01019, over 3041945.18 frames. ], batch size: 56, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:26:42,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1245026.6666666667, ans=0.125
2023-11-20 22:26:55,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1245093.3333333333, ans=0.0
2023-11-20 22:27:00,824 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.359e+01 8.103e+01 8.559e+01 9.205e+01 1.089e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-20 22:27:01,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1245093.3333333333, ans=0.0
2023-11-20 22:27:01,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1245093.3333333333, ans=0.125
2023-11-20 22:27:12,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1245160.0, ans=0.125
2023-11-20 22:27:21,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1245226.6666666667, ans=0.125
2023-11-20 22:27:34,057 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186800
2023-11-20 22:27:34,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1245293.3333333333, ans=0.125
2023-11-20 22:27:41,540 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6450, loss[loss=0.07724, simple_loss=0.09465, pruned_loss=0.01941, audio_tagging_loss=0.0105, over 15121.00 frames. ], tot_loss[loss=0.07665, simple_loss=0.0968, pruned_loss=0.01801, audio_tagging_loss=0.01024, over 3044276.63 frames. ], batch size: 57, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:27:47,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1245360.0, ans=0.2
2023-11-20 22:28:15,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1245493.3333333333, ans=0.125
2023-11-20 22:28:23,749 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.28 vs. limit=15.0
2023-11-20 22:28:33,959 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:28:38,629 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.30 vs. limit=15.0
2023-11-20 22:28:39,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186850
2023-11-20 22:28:44,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1245626.6666666667, ans=0.1
2023-11-20 22:28:46,236 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6500, loss[loss=0.07879, simple_loss=0.09728, pruned_loss=0.01875, audio_tagging_loss=0.0114, over 14608.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.0963, pruned_loss=0.01798, audio_tagging_loss=0.01024, over 3053275.07 frames. ], batch size: 55, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:28:49,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1245693.3333333333, ans=0.125
2023-11-20 22:28:59,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1245760.0, ans=0.5
2023-11-20 22:29:03,958 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.84 vs. limit=15.0
2023-11-20 22:29:09,943 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.106e+01 8.759e+01 9.505e+01 1.385e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-20 22:29:12,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1245826.6666666667, ans=0.0
2023-11-20 22:29:42,948 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186900
2023-11-20 22:29:43,514 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.96 vs. limit=15.0
2023-11-20 22:29:44,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1245960.0, ans=0.05
2023-11-20 22:29:50,882 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6550, loss[loss=0.07507, simple_loss=0.0997, pruned_loss=0.01405, audio_tagging_loss=0.01117, over 15522.00 frames. ], tot_loss[loss=0.0768, simple_loss=0.09757, pruned_loss=0.01801, audio_tagging_loss=0.01001, over 3062171.89 frames. ], batch size: 58, lr: 4.30e-03, grad_scale: 32.0
2023-11-20 22:29:59,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1246026.6666666667, ans=0.09899494936611666
2023-11-20 22:30:01,112 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.95 vs. limit=15.0
2023-11-20 22:30:03,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1246093.3333333333, ans=0.05
2023-11-20 22:30:25,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1246160.0, ans=0.125
2023-11-20 22:30:38,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1246226.6666666667, ans=0.125
2023-11-20 22:30:48,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 186950
2023-11-20 22:30:55,613 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6600, loss[loss=0.09526, simple_loss=0.1248, pruned_loss=0.02204, audio_tagging_loss=0.01084, over 15496.00 frames. ], tot_loss[loss=0.07678, simple_loss=0.09763, pruned_loss=0.01801, audio_tagging_loss=0.009955, over 3059110.73 frames. ], batch size: 54, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:31:15,858 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.28 vs. limit=15.0
2023-11-20 22:31:17,298 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.98 vs. limit=22.5
2023-11-20 22:31:18,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1246426.6666666667, ans=0.125
2023-11-20 22:31:18,841 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.367e+01 8.854e+01 9.653e+01 1.222e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-20 22:31:52,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187000
2023-11-20 22:32:00,546 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6650, loss[loss=0.08826, simple_loss=0.1123, pruned_loss=0.02068, audio_tagging_loss=0.01143, over 14119.00 frames. ], tot_loss[loss=0.07725, simple_loss=0.09833, pruned_loss=0.01818, audio_tagging_loss=0.009909, over 3049527.81 frames. ], batch size: 54, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:32:07,244 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.40 vs. limit=10.0
2023-11-20 22:32:33,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1246826.6666666667, ans=0.07
2023-11-20 22:32:42,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.76 vs. limit=15.0
2023-11-20 22:32:44,998 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.34 vs. limit=12.0
2023-11-20 22:32:45,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1246893.3333333333, ans=0.125
2023-11-20 22:32:54,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1246960.0, ans=0.1
2023-11-20 22:32:56,674 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187050
2023-11-20 22:32:59,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1246960.0, ans=0.125
2023-11-20 22:33:04,404 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6700, loss[loss=0.08377, simple_loss=0.1135, pruned_loss=0.01673, audio_tagging_loss=0.01031, over 15911.00 frames. ], tot_loss[loss=0.07731, simple_loss=0.09856, pruned_loss=0.01814, audio_tagging_loss=0.009891, over 3048556.80 frames. ], batch size: 58, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:33:27,693 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.055e+01 8.769e+01 9.493e+01 1.327e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-20 22:33:32,386 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.901e-01
2023-11-20 22:33:36,259 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.77 vs. limit=15.0
2023-11-20 22:34:00,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187100
2023-11-20 22:34:03,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1247293.3333333333, ans=0.125
2023-11-20 22:34:05,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1247293.3333333333, ans=0.0
2023-11-20 22:34:08,152 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6750, loss[loss=0.1078, simple_loss=0.1426, pruned_loss=0.02844, audio_tagging_loss=0.008105, over 15807.00 frames. ], tot_loss[loss=0.07686, simple_loss=0.09798, pruned_loss=0.01805, audio_tagging_loss=0.009819, over 3043467.24 frames. ], batch size: 58, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:34:14,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1247360.0, ans=10.0
2023-11-20 22:34:29,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1247426.6666666667, ans=0.125
2023-11-20 22:34:30,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1247426.6666666667, ans=0.125
2023-11-20 22:34:51,684 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.15 vs. limit=22.5
2023-11-20 22:34:58,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1247626.6666666667, ans=0.2
2023-11-20 22:35:05,415 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187150
2023-11-20 22:35:13,215 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6800, loss[loss=0.07606, simple_loss=0.1021, pruned_loss=0.0164, audio_tagging_loss=0.008593, over 15857.00 frames. ], tot_loss[loss=0.07645, simple_loss=0.09731, pruned_loss=0.01793, audio_tagging_loss=0.009858, over 3045850.43 frames. ], batch size: 57, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:35:24,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1247760.0, ans=0.125
2023-11-20 22:35:35,483 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.835e+01 8.011e+01 8.725e+01 9.590e+01 1.326e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-20 22:35:58,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1247893.3333333333, ans=0.035
2023-11-20 22:36:08,977 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187200
2023-11-20 22:36:16,469 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6850, loss[loss=0.0764, simple_loss=0.09265, pruned_loss=0.01974, audio_tagging_loss=0.01033, over 15381.00 frames. ], tot_loss[loss=0.07623, simple_loss=0.0971, pruned_loss=0.01791, audio_tagging_loss=0.009772, over 3044484.51 frames. ], batch size: 57, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:36:36,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1248093.3333333333, ans=0.0
2023-11-20 22:36:47,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1248160.0, ans=0.035
2023-11-20 22:36:55,836 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.53 vs. limit=22.5
2023-11-20 22:37:06,701 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.79 vs. limit=22.5
2023-11-20 22:37:10,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1248293.3333333333, ans=0.0
2023-11-20 22:37:12,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187250
2023-11-20 22:37:20,281 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6900, loss[loss=0.08018, simple_loss=0.1129, pruned_loss=0.01688, audio_tagging_loss=0.006859, over 15574.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.0976, pruned_loss=0.01806, audio_tagging_loss=0.009667, over 3047184.70 frames. ], batch size: 57, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:37:24,836 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.05 vs. limit=15.0
2023-11-20 22:37:34,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1248426.6666666667, ans=0.125
2023-11-20 22:37:44,213 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.341e+01 8.921e+01 9.872e+01 1.364e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-20 22:38:06,845 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 22:38:14,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1248626.6666666667, ans=0.0
2023-11-20 22:38:15,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1248626.6666666667, ans=0.125
2023-11-20 22:38:16,649 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187300
2023-11-20 22:38:25,011 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 6950, loss[loss=0.08904, simple_loss=0.1218, pruned_loss=0.02224, audio_tagging_loss=0.005912, over 15866.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09877, pruned_loss=0.01823, audio_tagging_loss=0.009605, over 3052319.98 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:38:27,949 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.30 vs. limit=22.5
2023-11-20 22:38:39,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1248760.0, ans=0.125
2023-11-20 22:38:52,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1248826.6666666667, ans=0.125
2023-11-20 22:38:53,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1248826.6666666667, ans=0.07
2023-11-20 22:39:04,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1248893.3333333333, ans=0.125
2023-11-20 22:39:19,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1248960.0, ans=0.125
2023-11-20 22:39:20,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.11 vs. limit=22.5
2023-11-20 22:39:21,193 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187350
2023-11-20 22:39:21,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1248960.0, ans=0.125
2023-11-20 22:39:28,604 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7000, loss[loss=0.07685, simple_loss=0.0965, pruned_loss=0.01966, audio_tagging_loss=0.008935, over 16074.00 frames. ], tot_loss[loss=0.07659, simple_loss=0.09755, pruned_loss=0.01807, audio_tagging_loss=0.009751, over 3042013.79 frames. ], batch size: 60, lr: 4.29e-03, grad_scale: 16.0
2023-11-20 22:39:36,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1249026.6666666667, ans=0.125
2023-11-20 22:39:41,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1249093.3333333333, ans=0.2
2023-11-20 22:39:49,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1249093.3333333333, ans=0.2
2023-11-20 22:39:50,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1249093.3333333333, ans=0.125
2023-11-20 22:39:51,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1249093.3333333333, ans=0.125
2023-11-20 22:39:52,395 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.760e+01 8.148e+01 8.784e+01 9.512e+01 1.267e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-20 22:39:58,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1249160.0, ans=0.125
2023-11-20 22:40:05,458 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:40:13,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1249226.6666666667, ans=0.125
2023-11-20 22:40:17,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1249226.6666666667, ans=0.125
2023-11-20 22:40:24,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1249293.3333333333, ans=0.125
2023-11-20 22:40:25,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187400
2023-11-20 22:40:33,255 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7050, loss[loss=0.08312, simple_loss=0.1076, pruned_loss=0.01928, audio_tagging_loss=0.01005, over 15086.00 frames. ], tot_loss[loss=0.07662, simple_loss=0.09787, pruned_loss=0.01795, audio_tagging_loss=0.009738, over 3049367.68 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 16.0
2023-11-20 22:40:50,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1249426.6666666667, ans=0.125
2023-11-20 22:40:57,493 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.86 vs. limit=15.0
2023-11-20 22:40:59,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1249493.3333333333, ans=0.0
2023-11-20 22:41:00,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1249493.3333333333, ans=0.0
2023-11-20 22:41:16,355 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.96 vs. limit=6.0
2023-11-20 22:41:27,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1249626.6666666667, ans=0.0
2023-11-20 22:41:29,397 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187450
2023-11-20 22:41:37,927 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7100, loss[loss=0.08162, simple_loss=0.09417, pruned_loss=0.02544, audio_tagging_loss=0.009097, over 14218.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09837, pruned_loss=0.01809, audio_tagging_loss=0.009853, over 3050188.13 frames. ], batch size: 54, lr: 4.29e-03, grad_scale: 16.0
2023-11-20 22:41:46,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1249693.3333333333, ans=0.1
2023-11-20 22:41:51,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1249760.0, ans=0.0
2023-11-20 22:42:02,251 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.191e+01 8.839e+01 9.583e+01 1.103e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 22:42:04,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.36 vs. limit=15.0
2023-11-20 22:42:16,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1249893.3333333333, ans=0.125
2023-11-20 22:42:18,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1249893.3333333333, ans=0.0
2023-11-20 22:42:32,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.67 vs. limit=15.0
2023-11-20 22:42:34,647 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187500
2023-11-20 22:42:34,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1249960.0, ans=0.125
2023-11-20 22:42:41,836 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7150, loss[loss=0.06626, simple_loss=0.08441, pruned_loss=0.01625, audio_tagging_loss=0.007814, over 14766.00 frames. ], tot_loss[loss=0.07739, simple_loss=0.09857, pruned_loss=0.0182, audio_tagging_loss=0.009907, over 3050556.94 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 16.0
2023-11-20 22:42:57,687 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.95 vs. limit=15.0
2023-11-20 22:42:58,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=1250093.3333333333, ans=0.05
2023-11-20 22:43:29,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1250226.6666666667, ans=0.0
2023-11-20 22:43:38,727 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187550
2023-11-20 22:43:45,999 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7200, loss[loss=0.1017, simple_loss=0.1219, pruned_loss=0.02927, audio_tagging_loss=0.01148, over 14804.00 frames. ], tot_loss[loss=0.07728, simple_loss=0.09824, pruned_loss=0.01814, audio_tagging_loss=0.01002, over 3052128.69 frames. ], batch size: 53, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:43:47,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1250360.0, ans=0.0
2023-11-20 22:44:10,056 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.608e+01 8.172e+01 8.869e+01 9.437e+01 2.740e+02, threshold=1.774e+02, percent-clipped=1.0
2023-11-20 22:44:10,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1250493.3333333333, ans=0.5
2023-11-20 22:44:15,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1250493.3333333333, ans=0.2
2023-11-20 22:44:23,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1250560.0, ans=0.2
2023-11-20 22:44:29,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.17 vs. limit=15.0
2023-11-20 22:44:31,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1250560.0, ans=0.2
2023-11-20 22:44:36,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1250626.6666666667, ans=0.125
2023-11-20 22:44:42,013 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187600
2023-11-20 22:44:42,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1250626.6666666667, ans=0.125
2023-11-20 22:44:46,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1250626.6666666667, ans=0.0
2023-11-20 22:44:46,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1250626.6666666667, ans=0.125
2023-11-20 22:44:47,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1250626.6666666667, ans=0.125
2023-11-20 22:44:50,287 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7250, loss[loss=0.09087, simple_loss=0.119, pruned_loss=0.02276, audio_tagging_loss=0.008611, over 14706.00 frames. ], tot_loss[loss=0.07756, simple_loss=0.09846, pruned_loss=0.01825, audio_tagging_loss=0.01008, over 3046325.68 frames. ], batch size: 52, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:45:03,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1250760.0, ans=0.1
2023-11-20 22:45:47,114 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187650
2023-11-20 22:45:55,061 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7300, loss[loss=0.08571, simple_loss=0.1135, pruned_loss=0.02164, audio_tagging_loss=0.007329, over 15149.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.09805, pruned_loss=0.01811, audio_tagging_loss=0.01001, over 3049784.87 frames. ], batch size: 56, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:45:58,268 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.32 vs. limit=15.0
2023-11-20 22:46:05,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.01 vs. limit=12.0
2023-11-20 22:46:18,768 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.538e+01 8.202e+01 8.886e+01 9.738e+01 1.326e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-20 22:46:26,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.55 vs. limit=15.0
2023-11-20 22:46:28,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1251160.0, ans=0.125
2023-11-20 22:46:30,065 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.12 vs. limit=15.0
2023-11-20 22:46:38,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1251226.6666666667, ans=0.1
2023-11-20 22:46:47,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1251293.3333333333, ans=0.125
2023-11-20 22:46:50,808 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0
2023-11-20 22:46:51,328 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187700
2023-11-20 22:46:59,104 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7350, loss[loss=0.08877, simple_loss=0.1227, pruned_loss=0.01863, audio_tagging_loss=0.008813, over 14328.00 frames. ], tot_loss[loss=0.07757, simple_loss=0.09881, pruned_loss=0.01833, audio_tagging_loss=0.009835, over 3048104.37 frames. ], batch size: 54, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:47:09,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1251360.0, ans=0.125
2023-11-20 22:47:44,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1251560.0, ans=0.0
2023-11-20 22:47:55,172 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187750
2023-11-20 22:48:02,468 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7400, loss[loss=0.06726, simple_loss=0.08672, pruned_loss=0.0167, audio_tagging_loss=0.007195, over 15518.00 frames. ], tot_loss[loss=0.07795, simple_loss=0.09966, pruned_loss=0.01845, audio_tagging_loss=0.00967, over 3050677.75 frames. ], batch size: 60, lr: 4.29e-03, grad_scale: 32.0
2023-11-20 22:48:27,608 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.213e+01 8.738e+01 9.709e+01 1.431e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-20 22:48:45,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1251893.3333333333, ans=0.2
2023-11-20 22:48:50,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1251893.3333333333, ans=0.0
2023-11-20 22:48:51,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1251893.3333333333, ans=0.1
2023-11-20 22:48:52,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.54 vs. limit=8.0
2023-11-20 22:48:52,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1251960.0, ans=0.125
2023-11-20 22:48:59,317 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187800
2023-11-20 22:49:06,813 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7450, loss[loss=0.1078, simple_loss=0.1411, pruned_loss=0.02661, audio_tagging_loss=0.01067, over 16084.00 frames. ], tot_loss[loss=0.07767, simple_loss=0.09923, pruned_loss=0.01837, audio_tagging_loss=0.009693, over 3043509.70 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:49:07,560 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.64 vs. limit=12.0
2023-11-20 22:49:30,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1252093.3333333333, ans=0.0
2023-11-20 22:49:55,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1252226.6666666667, ans=0.0
2023-11-20 22:50:01,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1252293.3333333333, ans=0.0
2023-11-20 22:50:02,424 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187850
2023-11-20 22:50:10,846 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7500, loss[loss=0.0622, simple_loss=0.0705, pruned_loss=0.01408, audio_tagging_loss=0.01287, over 15363.00 frames. ], tot_loss[loss=0.07741, simple_loss=0.09869, pruned_loss=0.01834, audio_tagging_loss=0.009731, over 3042032.59 frames. ], batch size: 59, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:50:14,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1252360.0, ans=0.125
2023-11-20 22:50:25,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1252426.6666666667, ans=0.1
2023-11-20 22:50:29,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1252426.6666666667, ans=0.0
2023-11-20 22:50:33,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1252426.6666666667, ans=0.0
2023-11-20 22:50:34,840 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.198e+01 8.879e+01 9.496e+01 1.257e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-20 22:50:40,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1252493.3333333333, ans=0.125
2023-11-20 22:50:53,547 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:50:56,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1252560.0, ans=0.5
2023-11-20 22:51:06,706 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187900
2023-11-20 22:51:09,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1252626.6666666667, ans=0.125
2023-11-20 22:51:10,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1252626.6666666667, ans=0.125
2023-11-20 22:51:13,979 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7550, loss[loss=0.07754, simple_loss=0.1005, pruned_loss=0.01693, audio_tagging_loss=0.01034, over 15732.00 frames. ], tot_loss[loss=0.07734, simple_loss=0.09882, pruned_loss=0.01827, audio_tagging_loss=0.009662, over 3045034.80 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:51:14,356 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 22:52:10,930 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 187950
2023-11-20 22:52:11,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1252960.0, ans=0.125
2023-11-20 22:52:17,973 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7600, loss[loss=0.07196, simple_loss=0.09257, pruned_loss=0.01628, audio_tagging_loss=0.009388, over 15905.00 frames. ], tot_loss[loss=0.07696, simple_loss=0.09827, pruned_loss=0.01819, audio_tagging_loss=0.009631, over 3048389.03 frames. ], batch size: 60, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:52:30,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1253093.3333333333, ans=0.125
2023-11-20 22:52:42,489 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.631e+01 8.162e+01 8.819e+01 9.727e+01 1.348e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-20 22:52:49,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1253160.0, ans=0.0
2023-11-20 22:52:49,643 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.46 vs. limit=15.0
2023-11-20 22:52:52,032 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=22.41 vs. limit=22.5
2023-11-20 22:52:59,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1253226.6666666667, ans=0.0
2023-11-20 22:53:14,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188000
2023-11-20 22:53:16,399 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-188000.pt
2023-11-20 22:53:25,805 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7650, loss[loss=0.06094, simple_loss=0.07478, pruned_loss=0.01331, audio_tagging_loss=0.01023, over 14925.00 frames. ], tot_loss[loss=0.0766, simple_loss=0.09768, pruned_loss=0.01811, audio_tagging_loss=0.009646, over 3049574.92 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:53:38,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1253426.6666666667, ans=0.125
2023-11-20 22:54:08,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1253560.0, ans=0.0
2023-11-20 22:54:16,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1253626.6666666667, ans=0.1
2023-11-20 22:54:22,918 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188050
2023-11-20 22:54:26,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.81 vs. limit=12.0
2023-11-20 22:54:30,072 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7700, loss[loss=0.0769, simple_loss=0.08478, pruned_loss=0.02296, audio_tagging_loss=0.01155, over 15988.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09684, pruned_loss=0.01792, audio_tagging_loss=0.009693, over 3047571.29 frames. ], batch size: 61, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:54:30,776 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.26 vs. limit=6.0
2023-11-20 22:54:34,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1253693.3333333333, ans=0.125
2023-11-20 22:54:49,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1253760.0, ans=0.125
2023-11-20 22:54:54,496 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.666e+01 7.984e+01 8.508e+01 9.260e+01 1.144e+02, threshold=1.702e+02, percent-clipped=0.0
2023-11-20 22:55:14,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1253893.3333333333, ans=0.2
2023-11-20 22:55:26,750 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188100
2023-11-20 22:55:33,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1254026.6666666667, ans=0.0
2023-11-20 22:55:34,507 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7750, loss[loss=0.08251, simple_loss=0.1122, pruned_loss=0.01892, audio_tagging_loss=0.007481, over 16301.00 frames. ], tot_loss[loss=0.07672, simple_loss=0.09795, pruned_loss=0.01812, audio_tagging_loss=0.009628, over 3043657.91 frames. ], batch size: 59, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:55:59,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1254160.0, ans=0.125
2023-11-20 22:56:05,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1254160.0, ans=0.0
2023-11-20 22:56:29,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1254293.3333333333, ans=0.125
2023-11-20 22:56:30,405 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188150
2023-11-20 22:56:36,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1254360.0, ans=0.125
2023-11-20 22:56:37,573 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7800, loss[loss=0.07256, simple_loss=0.09003, pruned_loss=0.01616, audio_tagging_loss=0.01138, over 15493.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.09847, pruned_loss=0.01818, audio_tagging_loss=0.009679, over 3040565.61 frames. ], batch size: 59, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:56:56,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1254426.6666666667, ans=0.025
2023-11-20 22:57:01,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1254426.6666666667, ans=0.125
2023-11-20 22:57:01,924 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.318e+01 8.003e+01 8.886e+01 9.530e+01 1.591e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-20 22:57:06,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.11 vs. limit=10.0
2023-11-20 22:57:21,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1254560.0, ans=0.1
2023-11-20 22:57:33,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1254626.6666666667, ans=0.2
2023-11-20 22:57:34,354 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188200
2023-11-20 22:57:42,500 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7850, loss[loss=0.06763, simple_loss=0.08362, pruned_loss=0.01287, audio_tagging_loss=0.01295, over 14451.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.09917, pruned_loss=0.01843, audio_tagging_loss=0.009724, over 3047061.69 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:57:51,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1254693.3333333333, ans=0.05
2023-11-20 22:57:57,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1254760.0, ans=0.0
2023-11-20 22:57:57,926 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.60 vs. limit=15.0
2023-11-20 22:58:04,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1254760.0, ans=0.125
2023-11-20 22:58:04,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1254760.0, ans=0.125
2023-11-20 22:58:06,530 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.26 vs. limit=22.5
2023-11-20 22:58:28,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1254893.3333333333, ans=0.2
2023-11-20 22:58:39,031 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188250
2023-11-20 22:58:47,263 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7900, loss[loss=0.07179, simple_loss=0.09884, pruned_loss=0.01413, audio_tagging_loss=0.008236, over 15593.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09929, pruned_loss=0.01837, audio_tagging_loss=0.009775, over 3047674.43 frames. ], batch size: 57, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 22:58:51,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1255026.6666666667, ans=10.0
2023-11-20 22:58:58,987 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.37 vs. limit=22.5
2023-11-20 22:59:09,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1255093.3333333333, ans=0.125
2023-11-20 22:59:10,913 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.630e+01 8.310e+01 8.933e+01 9.750e+01 1.293e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-20 22:59:43,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188300
2023-11-20 22:59:43,995 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.67 vs. limit=15.0
2023-11-20 22:59:50,525 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 7950, loss[loss=0.067, simple_loss=0.08571, pruned_loss=0.01437, audio_tagging_loss=0.009776, over 14335.00 frames. ], tot_loss[loss=0.07753, simple_loss=0.09861, pruned_loss=0.01831, audio_tagging_loss=0.00992, over 3048230.08 frames. ], batch size: 54, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:00:04,582 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:00:11,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1255426.6666666667, ans=0.125
2023-11-20 23:00:36,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1255560.0, ans=0.0
2023-11-20 23:00:45,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1255626.6666666667, ans=0.0
2023-11-20 23:00:46,742 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188350
2023-11-20 23:00:54,427 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8000, loss[loss=0.08182, simple_loss=0.1012, pruned_loss=0.02104, audio_tagging_loss=0.01016, over 16535.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09815, pruned_loss=0.01811, audio_tagging_loss=0.009957, over 3049687.80 frames. ], batch size: 62, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:01:02,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1255693.3333333333, ans=0.125
2023-11-20 23:01:13,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1255760.0, ans=0.2
2023-11-20 23:01:19,528 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.786e+01 8.042e+01 8.522e+01 9.463e+01 1.266e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-20 23:01:24,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1255826.6666666667, ans=0.1
2023-11-20 23:01:34,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.43 vs. limit=6.0
2023-11-20 23:01:43,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1255893.3333333333, ans=0.04949747468305833
2023-11-20 23:01:51,229 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188400
2023-11-20 23:01:59,861 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8050, loss[loss=0.07033, simple_loss=0.08839, pruned_loss=0.01759, audio_tagging_loss=0.008538, over 15632.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.09826, pruned_loss=0.01819, audio_tagging_loss=0.009981, over 3049165.23 frames. ], batch size: 58, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:02:08,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1256026.6666666667, ans=0.125
2023-11-20 23:02:19,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1256093.3333333333, ans=0.125
2023-11-20 23:02:56,172 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188450
2023-11-20 23:03:02,976 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.56 vs. limit=10.0
2023-11-20 23:03:03,312 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8100, loss[loss=0.08966, simple_loss=0.1186, pruned_loss=0.02108, audio_tagging_loss=0.009284, over 15153.00 frames. ], tot_loss[loss=0.07726, simple_loss=0.09843, pruned_loss=0.01818, audio_tagging_loss=0.009867, over 3046890.45 frames. ], batch size: 54, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:03:17,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1256426.6666666667, ans=0.0
2023-11-20 23:03:26,968 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.476e+01 8.237e+01 9.309e+01 1.004e+02 1.229e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-20 23:03:32,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1256493.3333333333, ans=0.0
2023-11-20 23:03:47,488 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.51 vs. limit=22.5
2023-11-20 23:03:57,089 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.70 vs. limit=15.0
2023-11-20 23:03:58,844 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188500
2023-11-20 23:04:04,867 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.31 vs. limit=10.0
2023-11-20 23:04:06,669 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8150, loss[loss=0.07813, simple_loss=0.1038, pruned_loss=0.01518, audio_tagging_loss=0.01106, over 14910.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.09844, pruned_loss=0.01816, audio_tagging_loss=0.009706, over 3047336.43 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:04:09,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1256693.3333333333, ans=0.125
2023-11-20 23:04:28,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1256760.0, ans=0.0
2023-11-20 23:04:53,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1256893.3333333333, ans=0.1
2023-11-20 23:05:00,794 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.45 vs. limit=6.0
2023-11-20 23:05:02,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188550
2023-11-20 23:05:08,401 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:05:09,562 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8200, loss[loss=0.04818, simple_loss=0.05492, pruned_loss=0.009579, audio_tagging_loss=0.01114, over 14699.00 frames. ], tot_loss[loss=0.0772, simple_loss=0.09893, pruned_loss=0.0181, audio_tagging_loss=0.00964, over 3044186.51 frames. ], batch size: 56, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:05:16,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1257026.6666666667, ans=0.1
2023-11-20 23:05:25,736 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.26 vs. limit=15.0
2023-11-20 23:05:32,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1257093.3333333333, ans=0.125
2023-11-20 23:05:34,611 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.637e+01 8.183e+01 8.865e+01 9.614e+01 1.183e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-20 23:05:37,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1257160.0, ans=0.0
2023-11-20 23:05:45,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1257160.0, ans=0.0
2023-11-20 23:05:46,767 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.55 vs. limit=15.0
2023-11-20 23:05:47,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1257226.6666666667, ans=0.0
2023-11-20 23:05:47,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1257226.6666666667, ans=0.1
2023-11-20 23:05:57,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1257226.6666666667, ans=0.125
2023-11-20 23:05:58,617 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:06:05,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1257293.3333333333, ans=0.1
2023-11-20 23:06:07,417 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188600
2023-11-20 23:06:15,091 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8250, loss[loss=0.08718, simple_loss=0.11, pruned_loss=0.02448, audio_tagging_loss=0.007695, over 15848.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.09918, pruned_loss=0.01809, audio_tagging_loss=0.009623, over 3049381.88 frames. ], batch size: 60, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:06:29,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1257426.6666666667, ans=0.0
2023-11-20 23:06:36,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1257426.6666666667, ans=0.0
2023-11-20 23:06:49,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1257493.3333333333, ans=0.0
2023-11-20 23:06:53,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1257560.0, ans=0.125
2023-11-20 23:06:54,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1257560.0, ans=0.1
2023-11-20 23:06:59,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1257560.0, ans=0.125
2023-11-20 23:07:00,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1257560.0, ans=0.125
2023-11-20 23:07:09,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.17 vs. limit=10.0
2023-11-20 23:07:11,229 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188650
2023-11-20 23:07:18,993 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8300, loss[loss=0.07835, simple_loss=0.1023, pruned_loss=0.01694, audio_tagging_loss=0.01027, over 16605.00 frames. ], tot_loss[loss=0.07674, simple_loss=0.09824, pruned_loss=0.01801, audio_tagging_loss=0.009613, over 3052652.12 frames. ], batch size: 60, lr: 4.28e-03, grad_scale: 32.0
2023-11-20 23:07:43,188 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.445e+01 8.113e+01 8.858e+01 9.502e+01 1.553e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-20 23:07:44,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1257826.6666666667, ans=0.1
2023-11-20 23:07:49,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1257826.6666666667, ans=0.5
2023-11-20 23:07:58,071 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.80 vs. limit=15.0
2023-11-20 23:08:01,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1257893.3333333333, ans=0.125
2023-11-20 23:08:08,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1257893.3333333333, ans=0.1
2023-11-20 23:08:15,374 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188700
2023-11-20 23:08:22,656 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8350, loss[loss=0.03912, simple_loss=0.0434, pruned_loss=0.004917, audio_tagging_loss=0.0125, over 14658.00 frames. ], tot_loss[loss=0.07702, simple_loss=0.0987, pruned_loss=0.01803, audio_tagging_loss=0.009645, over 3051014.89 frames. ], batch size: 57, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:08:27,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1258026.6666666667, ans=0.125
2023-11-20 23:08:39,746 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.75 vs. limit=15.0
2023-11-20 23:08:46,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1258093.3333333333, ans=0.0
2023-11-20 23:08:47,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1258160.0, ans=0.0
2023-11-20 23:08:49,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1258160.0, ans=0.1
2023-11-20 23:09:02,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1258226.6666666667, ans=0.2
2023-11-20 23:09:19,709 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188750
2023-11-20 23:09:24,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1258293.3333333333, ans=0.125
2023-11-20 23:09:27,462 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8400, loss[loss=0.06315, simple_loss=0.08528, pruned_loss=0.0114, audio_tagging_loss=0.009113, over 15406.00 frames. ], tot_loss[loss=0.07676, simple_loss=0.09848, pruned_loss=0.01791, audio_tagging_loss=0.009614, over 3051714.20 frames. ], batch size: 57, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:09:44,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1258426.6666666667, ans=0.0
2023-11-20 23:09:45,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1258426.6666666667, ans=0.1
2023-11-20 23:09:51,126 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.077e+01 8.855e+01 9.404e+01 1.397e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-20 23:10:19,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1258626.6666666667, ans=0.07
2023-11-20 23:10:21,182 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:10:23,330 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188800
2023-11-20 23:10:28,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1258626.6666666667, ans=0.125
2023-11-20 23:10:31,300 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8450, loss[loss=0.09863, simple_loss=0.1293, pruned_loss=0.02258, audio_tagging_loss=0.01138, over 15772.00 frames. ], tot_loss[loss=0.07716, simple_loss=0.09904, pruned_loss=0.01806, audio_tagging_loss=0.009584, over 3047366.23 frames. ], batch size: 58, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:10:36,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1258693.3333333333, ans=0.2
2023-11-20 23:10:58,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1258826.6666666667, ans=0.125
2023-11-20 23:11:11,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1258893.3333333333, ans=0.125
2023-11-20 23:11:22,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1258960.0, ans=0.0
2023-11-20 23:11:27,333 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188850
2023-11-20 23:11:29,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1258960.0, ans=0.125
2023-11-20 23:11:34,549 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8500, loss[loss=0.1079, simple_loss=0.1431, pruned_loss=0.03015, audio_tagging_loss=0.006209, over 15232.00 frames. ], tot_loss[loss=0.07797, simple_loss=0.1001, pruned_loss=0.01839, audio_tagging_loss=0.009532, over 3052895.83 frames. ], batch size: 54, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:11:44,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1259026.6666666667, ans=0.125
2023-11-20 23:11:59,756 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.153e+01 8.872e+01 9.707e+01 1.209e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-20 23:12:03,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1259160.0, ans=0.125
2023-11-20 23:12:10,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1259160.0, ans=0.0
2023-11-20 23:12:31,309 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188900
2023-11-20 23:12:39,247 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8550, loss[loss=0.1069, simple_loss=0.1493, pruned_loss=0.02681, audio_tagging_loss=0.005436, over 16297.00 frames. ], tot_loss[loss=0.07804, simple_loss=0.1001, pruned_loss=0.01844, audio_tagging_loss=0.009549, over 3055041.36 frames. ], batch size: 58, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:12:50,147 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.37 vs. limit=10.0
2023-11-20 23:12:58,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1259426.6666666667, ans=0.2
2023-11-20 23:13:08,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1259493.3333333333, ans=0.125
2023-11-20 23:13:08,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1259493.3333333333, ans=0.2
2023-11-20 23:13:13,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1259493.3333333333, ans=0.125
2023-11-20 23:13:35,017 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 188950
2023-11-20 23:13:42,792 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8600, loss[loss=0.06789, simple_loss=0.08417, pruned_loss=0.01419, audio_tagging_loss=0.01161, over 14126.00 frames. ], tot_loss[loss=0.07795, simple_loss=0.1, pruned_loss=0.01841, audio_tagging_loss=0.009547, over 3054566.82 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:14:05,679 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.63 vs. limit=15.0
2023-11-20 23:14:06,715 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.038e+01 8.664e+01 9.287e+01 1.205e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-20 23:14:12,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1259826.6666666667, ans=0.125
2023-11-20 23:14:13,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1259826.6666666667, ans=0.1
2023-11-20 23:14:29,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1259893.3333333333, ans=0.125
2023-11-20 23:14:39,661 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189000
2023-11-20 23:14:42,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1259960.0, ans=0.125
2023-11-20 23:14:44,359 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.55 vs. limit=15.0
2023-11-20 23:14:47,254 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8650, loss[loss=0.05338, simple_loss=0.05489, pruned_loss=0.0107, audio_tagging_loss=0.01523, over 13659.00 frames. ], tot_loss[loss=0.07791, simple_loss=0.09955, pruned_loss=0.0184, audio_tagging_loss=0.00974, over 3053390.06 frames. ], batch size: 56, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:15:15,919 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.54 vs. limit=15.0
2023-11-20 23:15:20,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.49 vs. limit=10.0
2023-11-20 23:15:25,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1260226.6666666667, ans=0.125
2023-11-20 23:15:31,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1260226.6666666667, ans=0.125
2023-11-20 23:15:43,520 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189050
2023-11-20 23:15:52,080 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8700, loss[loss=0.1021, simple_loss=0.1404, pruned_loss=0.02374, audio_tagging_loss=0.008146, over 15985.00 frames. ], tot_loss[loss=0.07842, simple_loss=0.1002, pruned_loss=0.01858, audio_tagging_loss=0.009741, over 3053362.09 frames. ], batch size: 57, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:15:53,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1260360.0, ans=0.1
2023-11-20 23:16:05,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1260426.6666666667, ans=0.015
2023-11-20 23:16:05,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1260426.6666666667, ans=0.0
2023-11-20 23:16:17,583 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.837e+01 8.100e+01 8.533e+01 9.341e+01 1.224e+02, threshold=1.707e+02, percent-clipped=0.0
2023-11-20 23:16:18,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1260493.3333333333, ans=0.0
2023-11-20 23:16:18,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=1260493.3333333333, ans=6.0
2023-11-20 23:16:18,246 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.24 vs. limit=15.0
2023-11-20 23:16:18,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1260493.3333333333, ans=0.05
2023-11-20 23:16:26,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1260493.3333333333, ans=0.5
2023-11-20 23:16:48,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189100
2023-11-20 23:16:53,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1260626.6666666667, ans=0.1
2023-11-20 23:16:55,209 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8750, loss[loss=0.08903, simple_loss=0.1246, pruned_loss=0.01922, audio_tagging_loss=0.007488, over 14613.00 frames. ], tot_loss[loss=0.07771, simple_loss=0.09883, pruned_loss=0.01835, audio_tagging_loss=0.009946, over 3049495.04 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:17:18,361 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:17:19,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1260826.6666666667, ans=0.5
2023-11-20 23:17:26,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1260826.6666666667, ans=0.1
2023-11-20 23:17:43,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1260893.3333333333, ans=0.0
2023-11-20 23:17:51,324 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189150
2023-11-20 23:17:56,680 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.35 vs. limit=15.0
2023-11-20 23:17:58,597 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8800, loss[loss=0.09605, simple_loss=0.1236, pruned_loss=0.02482, audio_tagging_loss=0.009402, over 14643.00 frames. ], tot_loss[loss=0.07835, simple_loss=0.09959, pruned_loss=0.01855, audio_tagging_loss=0.01001, over 3049459.23 frames. ], batch size: 56, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:18:17,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1261093.3333333333, ans=0.025
2023-11-20 23:18:24,225 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.201e+01 8.889e+01 9.657e+01 1.880e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-20 23:18:26,391 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.66 vs. limit=15.0
2023-11-20 23:18:30,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1261160.0, ans=0.0
2023-11-20 23:18:54,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189200
2023-11-20 23:19:00,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1261293.3333333333, ans=0.1
2023-11-20 23:19:00,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1261293.3333333333, ans=0.1
2023-11-20 23:19:03,169 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8850, loss[loss=0.08508, simple_loss=0.1103, pruned_loss=0.02215, audio_tagging_loss=0.007767, over 15769.00 frames. ], tot_loss[loss=0.07788, simple_loss=0.09893, pruned_loss=0.01833, audio_tagging_loss=0.01008, over 3044913.72 frames. ], batch size: 62, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:19:13,009 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:19:40,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1261560.0, ans=0.2
2023-11-20 23:19:41,106 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.50 vs. limit=15.0
2023-11-20 23:19:47,002 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:19:53,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1261626.6666666667, ans=0.125
2023-11-20 23:19:58,742 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189250
2023-11-20 23:20:03,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1261626.6666666667, ans=0.2
2023-11-20 23:20:05,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1261693.3333333333, ans=0.125
2023-11-20 23:20:05,957 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8900, loss[loss=0.09449, simple_loss=0.1233, pruned_loss=0.02412, audio_tagging_loss=0.008744, over 15210.00 frames. ], tot_loss[loss=0.07817, simple_loss=0.09966, pruned_loss=0.01853, audio_tagging_loss=0.009813, over 3054811.41 frames. ], batch size: 56, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:20:09,217 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.74 vs. limit=15.0
2023-11-20 23:20:26,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1261760.0, ans=0.125
2023-11-20 23:20:27,184 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.95 vs. limit=10.0
2023-11-20 23:20:31,206 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.227e+01 8.677e+01 9.378e+01 1.174e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-20 23:20:31,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1261826.6666666667, ans=0.125
2023-11-20 23:20:47,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.62 vs. limit=6.0
2023-11-20 23:21:01,361 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189300
2023-11-20 23:21:01,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1261960.0, ans=0.0
2023-11-20 23:21:09,626 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 8950, loss[loss=0.06385, simple_loss=0.07977, pruned_loss=0.01298, audio_tagging_loss=0.01099, over 14746.00 frames. ], tot_loss[loss=0.07733, simple_loss=0.09837, pruned_loss=0.01831, audio_tagging_loss=0.009838, over 3056349.60 frames. ], batch size: 55, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:21:10,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1262026.6666666667, ans=0.1
2023-11-20 23:21:11,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1262026.6666666667, ans=0.0
2023-11-20 23:21:12,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.whiten.whitening_limit, batch_count=1262026.6666666667, ans=12.0
2023-11-20 23:21:20,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1262093.3333333333, ans=0.1
2023-11-20 23:21:23,607 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.78 vs. limit=15.0
2023-11-20 23:21:47,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=1262226.6666666667, ans=15.0
2023-11-20 23:21:55,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.whiten.whitening_limit, batch_count=1262226.6666666667, ans=12.0
2023-11-20 23:22:05,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189350
2023-11-20 23:22:07,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1262293.3333333333, ans=0.125
2023-11-20 23:22:13,077 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9000, loss[loss=0.07974, simple_loss=0.09919, pruned_loss=0.02144, audio_tagging_loss=0.008706, over 17141.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.09914, pruned_loss=0.01848, audio_tagging_loss=0.009692, over 3058699.37 frames. ], batch size: 63, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:22:13,080 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-20 23:22:53,337 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.6129, 3.4946, 3.7269, 3.4390], device='cuda:0')
2023-11-20 23:22:55,285 INFO [train_asr.py:1253] (0/4) Epoch 16, validation: loss=0.06115, simple_loss=0.05296, pruned_loss=0.005511, audio_tagging_loss=0.02916, over 4681554.00 frames. 
2023-11-20 23:22:55,286 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-20 23:23:09,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1262426.6666666667, ans=0.2
2023-11-20 23:23:15,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn2.whiten.whitening_limit, batch_count=1262426.6666666667, ans=22.5
2023-11-20 23:23:22,726 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.262e+01 9.175e+01 9.933e+01 1.311e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-20 23:23:22,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1262493.3333333333, ans=0.2
2023-11-20 23:23:29,697 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:23:51,951 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189400
2023-11-20 23:23:53,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1262626.6666666667, ans=0.1
2023-11-20 23:23:59,693 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9050, loss[loss=0.09642, simple_loss=0.1271, pruned_loss=0.02634, audio_tagging_loss=0.006533, over 15309.00 frames. ], tot_loss[loss=0.07819, simple_loss=0.1, pruned_loss=0.01856, audio_tagging_loss=0.009621, over 3061700.85 frames. ], batch size: 59, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:23:59,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1262693.3333333333, ans=0.1
2023-11-20 23:24:03,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1262693.3333333333, ans=0.125
2023-11-20 23:24:12,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1262760.0, ans=0.125
2023-11-20 23:24:12,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1262760.0, ans=0.125
2023-11-20 23:24:20,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1262760.0, ans=0.07
2023-11-20 23:24:29,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1262826.6666666667, ans=0.1
2023-11-20 23:24:41,697 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=10.08 vs. limit=15.0
2023-11-20 23:24:45,220 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.54 vs. limit=15.0
2023-11-20 23:24:46,429 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.77 vs. limit=15.0
2023-11-20 23:24:56,126 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189450
2023-11-20 23:25:04,472 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9100, loss[loss=0.0718, simple_loss=0.0967, pruned_loss=0.0158, audio_tagging_loss=0.007648, over 16609.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.0989, pruned_loss=0.01828, audio_tagging_loss=0.009565, over 3053822.67 frames. ], batch size: 60, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:25:07,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1263026.6666666667, ans=0.125
2023-11-20 23:25:16,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1263093.3333333333, ans=0.125
2023-11-20 23:25:30,561 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.518e+01 8.183e+01 8.786e+01 9.651e+01 1.253e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-20 23:26:00,252 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189500
2023-11-20 23:26:07,497 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9150, loss[loss=0.07056, simple_loss=0.09373, pruned_loss=0.01466, audio_tagging_loss=0.009037, over 16013.00 frames. ], tot_loss[loss=0.07745, simple_loss=0.09932, pruned_loss=0.01819, audio_tagging_loss=0.009604, over 3056407.20 frames. ], batch size: 62, lr: 4.27e-03, grad_scale: 16.0
2023-11-20 23:26:15,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1263360.0, ans=0.1
2023-11-20 23:26:29,410 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.46 vs. limit=22.5
2023-11-20 23:26:52,535 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.64 vs. limit=15.0
2023-11-20 23:26:59,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1263626.6666666667, ans=0.125
2023-11-20 23:27:03,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189550
2023-11-20 23:27:10,368 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9200, loss[loss=0.07558, simple_loss=0.09944, pruned_loss=0.01706, audio_tagging_loss=0.008801, over 14763.00 frames. ], tot_loss[loss=0.07818, simple_loss=0.1003, pruned_loss=0.01843, audio_tagging_loss=0.009626, over 3060700.57 frames. ], batch size: 57, lr: 4.27e-03, grad_scale: 32.0
2023-11-20 23:27:12,286 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.60 vs. limit=12.0
2023-11-20 23:27:37,269 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.197e+01 8.151e+01 8.702e+01 9.490e+01 1.171e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-20 23:28:06,302 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189600
2023-11-20 23:28:14,510 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9250, loss[loss=0.07259, simple_loss=0.08876, pruned_loss=0.01642, audio_tagging_loss=0.01178, over 15478.00 frames. ], tot_loss[loss=0.07694, simple_loss=0.09849, pruned_loss=0.018, audio_tagging_loss=0.009699, over 3058312.41 frames. ], batch size: 61, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:28:34,173 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:28:34,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1264093.3333333333, ans=0.05
2023-11-20 23:28:34,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1264093.3333333333, ans=0.125
2023-11-20 23:28:51,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1264226.6666666667, ans=0.125
2023-11-20 23:28:52,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1264226.6666666667, ans=0.0
2023-11-20 23:29:10,867 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189650
2023-11-20 23:29:18,288 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9300, loss[loss=0.1096, simple_loss=0.1391, pruned_loss=0.0324, audio_tagging_loss=0.007662, over 15042.00 frames. ], tot_loss[loss=0.07678, simple_loss=0.09841, pruned_loss=0.01785, audio_tagging_loss=0.009717, over 3057514.71 frames. ], batch size: 56, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:29:25,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1264360.0, ans=0.125
2023-11-20 23:29:39,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1264426.6666666667, ans=0.1
2023-11-20 23:29:42,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1264493.3333333333, ans=0.125
2023-11-20 23:29:44,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.07 vs. limit=10.0
2023-11-20 23:29:44,508 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.124e+01 8.662e+01 9.573e+01 1.521e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-20 23:29:50,307 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:30:14,564 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189700
2023-11-20 23:30:19,443 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.76 vs. limit=12.0
2023-11-20 23:30:22,385 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9350, loss[loss=0.06687, simple_loss=0.08847, pruned_loss=0.01345, audio_tagging_loss=0.009182, over 15776.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09739, pruned_loss=0.01759, audio_tagging_loss=0.009868, over 3057612.96 frames. ], batch size: 60, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:30:43,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1264760.0, ans=0.125
2023-11-20 23:30:46,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1264760.0, ans=0.125
2023-11-20 23:31:17,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1264960.0, ans=0.125
2023-11-20 23:31:18,383 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189750
2023-11-20 23:31:26,051 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9400, loss[loss=0.08936, simple_loss=0.1239, pruned_loss=0.01662, audio_tagging_loss=0.01079, over 15373.00 frames. ], tot_loss[loss=0.07671, simple_loss=0.09805, pruned_loss=0.01779, audio_tagging_loss=0.009891, over 3058405.44 frames. ], batch size: 56, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:31:31,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1265026.6666666667, ans=0.1
2023-11-20 23:31:38,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1265093.3333333333, ans=0.125
2023-11-20 23:31:49,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1265093.3333333333, ans=0.0
2023-11-20 23:31:49,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1265093.3333333333, ans=0.0
2023-11-20 23:31:54,706 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.398e+01 8.103e+01 8.759e+01 9.361e+01 1.221e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-20 23:31:55,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1265160.0, ans=0.0
2023-11-20 23:32:07,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1265226.6666666667, ans=0.125
2023-11-20 23:32:22,990 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189800
2023-11-20 23:32:27,727 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:32:31,404 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9450, loss[loss=0.07835, simple_loss=0.09764, pruned_loss=0.0187, audio_tagging_loss=0.01083, over 15244.00 frames. ], tot_loss[loss=0.07727, simple_loss=0.0986, pruned_loss=0.01805, audio_tagging_loss=0.009927, over 3051297.51 frames. ], batch size: 58, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:33:05,420 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.92 vs. limit=15.0
2023-11-20 23:33:10,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1265560.0, ans=0.0
2023-11-20 23:33:22,104 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.82 vs. limit=15.0
2023-11-20 23:33:27,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189850
2023-11-20 23:33:32,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1265626.6666666667, ans=0.125
2023-11-20 23:33:34,941 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9500, loss[loss=0.09184, simple_loss=0.1128, pruned_loss=0.02374, audio_tagging_loss=0.01169, over 16557.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09831, pruned_loss=0.01799, audio_tagging_loss=0.009995, over 3055722.30 frames. ], batch size: 60, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:33:36,955 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.60 vs. limit=12.0
2023-11-20 23:33:43,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1265693.3333333333, ans=0.04949747468305833
2023-11-20 23:34:02,683 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.779e+01 8.467e+01 9.097e+01 1.010e+02 1.655e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-20 23:34:23,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1265893.3333333333, ans=0.125
2023-11-20 23:34:30,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189900
2023-11-20 23:34:38,072 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9550, loss[loss=0.08101, simple_loss=0.1027, pruned_loss=0.02213, audio_tagging_loss=0.007513, over 14410.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09876, pruned_loss=0.01815, audio_tagging_loss=0.01005, over 3057154.88 frames. ], batch size: 53, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:35:31,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1266293.3333333333, ans=0.2
2023-11-20 23:35:31,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1266293.3333333333, ans=0.125
2023-11-20 23:35:35,282 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 189950
2023-11-20 23:35:42,347 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9600, loss[loss=0.06507, simple_loss=0.07034, pruned_loss=0.01807, audio_tagging_loss=0.01184, over 15885.00 frames. ], tot_loss[loss=0.07654, simple_loss=0.09745, pruned_loss=0.01772, audio_tagging_loss=0.0101, over 3056893.89 frames. ], batch size: 62, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:35:42,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1266360.0, ans=0.0
2023-11-20 23:35:43,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.52 vs. limit=15.0
2023-11-20 23:35:48,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1266360.0, ans=0.1
2023-11-20 23:35:51,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1266360.0, ans=0.1
2023-11-20 23:35:54,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1266426.6666666667, ans=0.1
2023-11-20 23:36:10,700 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.084e+01 8.487e+01 9.415e+01 1.113e+02, threshold=1.697e+02, percent-clipped=0.0
2023-11-20 23:36:15,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1266493.3333333333, ans=0.0
2023-11-20 23:36:16,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1266493.3333333333, ans=0.0
2023-11-20 23:36:25,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=1266560.0, ans=10.0
2023-11-20 23:36:39,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190000
2023-11-20 23:36:47,537 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9650, loss[loss=0.07603, simple_loss=0.1031, pruned_loss=0.01469, audio_tagging_loss=0.009795, over 15778.00 frames. ], tot_loss[loss=0.07726, simple_loss=0.09812, pruned_loss=0.0181, audio_tagging_loss=0.0101, over 3050321.33 frames. ], batch size: 57, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:36:54,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1266693.3333333333, ans=0.0
2023-11-20 23:37:06,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1266760.0, ans=0.1
2023-11-20 23:37:10,060 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.91 vs. limit=15.0
2023-11-20 23:37:15,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1266826.6666666667, ans=0.0
2023-11-20 23:37:43,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1266960.0, ans=0.0
2023-11-20 23:37:44,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190050
2023-11-20 23:37:51,310 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9700, loss[loss=0.09355, simple_loss=0.1126, pruned_loss=0.02581, audio_tagging_loss=0.01142, over 14586.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09813, pruned_loss=0.01811, audio_tagging_loss=0.009937, over 3043524.79 frames. ], batch size: 55, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:38:14,080 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.37 vs. limit=15.0
2023-11-20 23:38:16,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1267160.0, ans=0.0
2023-11-20 23:38:21,190 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.452e+01 8.015e+01 8.951e+01 9.775e+01 1.301e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-20 23:38:39,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1267226.6666666667, ans=0.0
2023-11-20 23:38:48,757 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190100
2023-11-20 23:38:56,273 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9750, loss[loss=0.07963, simple_loss=0.1091, pruned_loss=0.01759, audio_tagging_loss=0.0075, over 15751.00 frames. ], tot_loss[loss=0.07778, simple_loss=0.09941, pruned_loss=0.01834, audio_tagging_loss=0.009741, over 3043782.82 frames. ], batch size: 58, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:38:56,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1267360.0, ans=0.5
2023-11-20 23:38:59,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.12 vs. limit=15.0
2023-11-20 23:39:10,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1267426.6666666667, ans=0.0
2023-11-20 23:39:16,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1267426.6666666667, ans=0.025
2023-11-20 23:39:31,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1267493.3333333333, ans=0.125
2023-11-20 23:39:32,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1267493.3333333333, ans=0.0
2023-11-20 23:39:33,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1267560.0, ans=0.125
2023-11-20 23:39:38,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1267560.0, ans=0.125
2023-11-20 23:39:52,143 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.41 vs. limit=15.0
2023-11-20 23:39:52,895 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190150
2023-11-20 23:40:00,682 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9800, loss[loss=0.05424, simple_loss=0.06554, pruned_loss=0.01314, audio_tagging_loss=0.008332, over 14164.00 frames. ], tot_loss[loss=0.07812, simple_loss=0.09997, pruned_loss=0.0185, audio_tagging_loss=0.00964, over 3035694.25 frames. ], batch size: 54, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:40:06,989 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.62 vs. limit=10.0
2023-11-20 23:40:17,597 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.21 vs. limit=15.0
2023-11-20 23:40:30,320 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.597e+01 7.945e+01 8.740e+01 9.752e+01 1.304e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-20 23:40:34,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1267826.6666666667, ans=0.125
2023-11-20 23:40:56,344 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:40:57,698 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190200
2023-11-20 23:41:05,209 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9850, loss[loss=0.08004, simple_loss=0.1017, pruned_loss=0.01805, audio_tagging_loss=0.01114, over 15305.00 frames. ], tot_loss[loss=0.07834, simple_loss=0.1004, pruned_loss=0.01855, audio_tagging_loss=0.009614, over 3032414.44 frames. ], batch size: 58, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:41:21,582 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.71 vs. limit=6.0
2023-11-20 23:41:38,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1268160.0, ans=0.2
2023-11-20 23:41:41,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1268160.0, ans=0.125
2023-11-20 23:41:42,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1268226.6666666667, ans=0.125
2023-11-20 23:41:58,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1268293.3333333333, ans=0.2
2023-11-20 23:42:00,333 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190250
2023-11-20 23:42:07,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1268360.0, ans=0.2
2023-11-20 23:42:08,619 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9900, loss[loss=0.08079, simple_loss=0.1056, pruned_loss=0.02034, audio_tagging_loss=0.007666, over 13756.00 frames. ], tot_loss[loss=0.07816, simple_loss=0.1002, pruned_loss=0.01843, audio_tagging_loss=0.00964, over 3031886.72 frames. ], batch size: 52, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:42:10,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1268360.0, ans=0.125
2023-11-20 23:42:12,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1268360.0, ans=0.0
2023-11-20 23:42:37,753 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.516e+01 8.053e+01 8.764e+01 9.412e+01 1.307e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-20 23:42:43,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1268493.3333333333, ans=0.125
2023-11-20 23:43:00,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1268626.6666666667, ans=0.2
2023-11-20 23:43:04,908 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190300
2023-11-20 23:43:12,107 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 9950, loss[loss=0.09563, simple_loss=0.1337, pruned_loss=0.02219, audio_tagging_loss=0.00658, over 15181.00 frames. ], tot_loss[loss=0.07799, simple_loss=0.1002, pruned_loss=0.01834, audio_tagging_loss=0.009559, over 3044859.27 frames. ], batch size: 56, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:43:27,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1268760.0, ans=10.0
2023-11-20 23:43:38,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1268826.6666666667, ans=0.125
2023-11-20 23:43:46,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1268826.6666666667, ans=0.07
2023-11-20 23:43:48,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1268826.6666666667, ans=0.025
2023-11-20 23:43:56,810 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.11 vs. limit=6.0
2023-11-20 23:44:06,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1268960.0, ans=0.0
2023-11-20 23:44:08,795 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190350
2023-11-20 23:44:16,750 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10000, loss[loss=0.05551, simple_loss=0.07117, pruned_loss=0.01042, audio_tagging_loss=0.009506, over 15221.00 frames. ], tot_loss[loss=0.07761, simple_loss=0.09949, pruned_loss=0.01825, audio_tagging_loss=0.009619, over 3040498.03 frames. ], batch size: 56, lr: 4.26e-03, grad_scale: 32.0
2023-11-20 23:44:23,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.88 vs. limit=15.0
2023-11-20 23:44:24,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=1269026.6666666667, ans=0.025
2023-11-20 23:44:24,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1269026.6666666667, ans=0.125
2023-11-20 23:44:37,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1269093.3333333333, ans=10.0
2023-11-20 23:44:45,926 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.173e+01 8.806e+01 9.776e+01 1.486e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-20 23:44:58,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1269226.6666666667, ans=0.0
2023-11-20 23:45:09,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1269293.3333333333, ans=0.0
2023-11-20 23:45:13,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190400
2023-11-20 23:45:13,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1269293.3333333333, ans=0.125
2023-11-20 23:45:21,705 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10050, loss[loss=0.07433, simple_loss=0.1053, pruned_loss=0.01389, audio_tagging_loss=0.007782, over 15286.00 frames. ], tot_loss[loss=0.07746, simple_loss=0.09948, pruned_loss=0.01811, audio_tagging_loss=0.009602, over 3047757.06 frames. ], batch size: 56, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:45:23,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1269360.0, ans=0.0
2023-11-20 23:45:31,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1269360.0, ans=0.07
2023-11-20 23:45:36,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1269426.6666666667, ans=0.0
2023-11-20 23:45:42,696 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=5.10 vs. limit=15.0
2023-11-20 23:45:59,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1269560.0, ans=0.125
2023-11-20 23:46:17,965 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190450
2023-11-20 23:46:23,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1269626.6666666667, ans=0.0
2023-11-20 23:46:25,284 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10100, loss[loss=0.05915, simple_loss=0.07067, pruned_loss=0.01174, audio_tagging_loss=0.01208, over 14702.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09894, pruned_loss=0.01795, audio_tagging_loss=0.009693, over 3047856.63 frames. ], batch size: 55, lr: 4.26e-03, grad_scale: 16.0
2023-11-20 23:46:45,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1269760.0, ans=0.04949747468305833
2023-11-20 23:46:56,617 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 8.119e+01 8.749e+01 9.415e+01 1.269e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-20 23:47:00,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1269826.6666666667, ans=0.0
2023-11-20 23:47:11,033 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.02 vs. limit=22.5
2023-11-20 23:47:14,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1269893.3333333333, ans=0.125
2023-11-20 23:47:15,173 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:47:15,948 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.49 vs. limit=12.0
2023-11-20 23:47:16,984 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.15 vs. limit=15.0
2023-11-20 23:47:22,274 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190500
2023-11-20 23:47:22,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1269960.0, ans=0.035
2023-11-20 23:47:29,424 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10150, loss[loss=0.05315, simple_loss=0.05961, pruned_loss=0.01267, audio_tagging_loss=0.01067, over 14865.00 frames. ], tot_loss[loss=0.07662, simple_loss=0.09807, pruned_loss=0.01779, audio_tagging_loss=0.009795, over 3047654.59 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:47:57,651 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:48:02,209 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.13 vs. limit=15.0
2023-11-20 23:48:23,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1270293.3333333333, ans=0.125
2023-11-20 23:48:25,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1270293.3333333333, ans=0.0
2023-11-20 23:48:25,845 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190550
2023-11-20 23:48:33,567 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10200, loss[loss=0.1096, simple_loss=0.1312, pruned_loss=0.03469, audio_tagging_loss=0.009329, over 14329.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.09815, pruned_loss=0.01789, audio_tagging_loss=0.0098, over 3049740.20 frames. ], batch size: 53, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:48:40,223 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.93 vs. limit=15.0
2023-11-20 23:48:54,069 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-20 23:48:56,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1270426.6666666667, ans=0.025
2023-11-20 23:49:02,939 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.808e+01 8.167e+01 8.853e+01 9.625e+01 1.935e+02, threshold=1.771e+02, percent-clipped=1.0
2023-11-20 23:49:08,112 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:49:08,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1270493.3333333333, ans=0.2
2023-11-20 23:49:22,009 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.31 vs. limit=15.0
2023-11-20 23:49:28,629 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190600
2023-11-20 23:49:35,987 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10250, loss[loss=0.08219, simple_loss=0.1112, pruned_loss=0.01617, audio_tagging_loss=0.01041, over 14939.00 frames. ], tot_loss[loss=0.07724, simple_loss=0.09869, pruned_loss=0.01809, audio_tagging_loss=0.009804, over 3041805.24 frames. ], batch size: 54, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:49:58,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1270760.0, ans=0.125
2023-11-20 23:50:17,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1270893.3333333333, ans=0.125
2023-11-20 23:50:23,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1270893.3333333333, ans=0.0
2023-11-20 23:50:32,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190650
2023-11-20 23:50:37,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1270960.0, ans=0.125
2023-11-20 23:50:39,974 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10300, loss[loss=0.07723, simple_loss=0.1055, pruned_loss=0.01544, audio_tagging_loss=0.009056, over 16246.00 frames. ], tot_loss[loss=0.07717, simple_loss=0.09854, pruned_loss=0.01803, audio_tagging_loss=0.009872, over 3047474.11 frames. ], batch size: 60, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:50:43,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1271026.6666666667, ans=0.04949747468305833
2023-11-20 23:50:45,267 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:51:02,112 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.22 vs. limit=22.5
2023-11-20 23:51:11,224 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.201e+01 8.870e+01 9.848e+01 1.361e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-20 23:51:12,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=23.18 vs. limit=22.5
2023-11-20 23:51:15,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1271160.0, ans=0.1
2023-11-20 23:51:18,410 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.86 vs. limit=15.0
2023-11-20 23:51:36,428 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190700
2023-11-20 23:51:36,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1271293.3333333333, ans=0.0
2023-11-20 23:51:37,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1271293.3333333333, ans=0.125
2023-11-20 23:51:44,205 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10350, loss[loss=0.07126, simple_loss=0.09633, pruned_loss=0.0147, audio_tagging_loss=0.008396, over 15087.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.0987, pruned_loss=0.01812, audio_tagging_loss=0.009896, over 3048428.08 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 8.0
2023-11-20 23:51:44,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1271360.0, ans=0.0
2023-11-20 23:51:52,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0
2023-11-20 23:52:07,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1271426.6666666667, ans=0.125
2023-11-20 23:52:09,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1271493.3333333333, ans=0.0
2023-11-20 23:52:18,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1271493.3333333333, ans=0.125
2023-11-20 23:52:23,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1271560.0, ans=0.2
2023-11-20 23:52:31,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1271560.0, ans=0.125
2023-11-20 23:52:35,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1271626.6666666667, ans=0.125
2023-11-20 23:52:40,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190750
2023-11-20 23:52:47,659 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10400, loss[loss=0.07827, simple_loss=0.09964, pruned_loss=0.02009, audio_tagging_loss=0.008349, over 14542.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09827, pruned_loss=0.01804, audio_tagging_loss=0.00996, over 3039248.98 frames. ], batch size: 55, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:52:56,498 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:52:56,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1271693.3333333333, ans=0.125
2023-11-20 23:52:59,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1271760.0, ans=0.125
2023-11-20 23:53:06,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.00 vs. limit=6.0
2023-11-20 23:53:10,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1271760.0, ans=0.125
2023-11-20 23:53:17,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1271826.6666666667, ans=0.125
2023-11-20 23:53:18,608 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.195e+01 8.838e+01 9.520e+01 1.388e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-20 23:53:43,146 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190800
2023-11-20 23:53:51,007 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10450, loss[loss=0.1033, simple_loss=0.1444, pruned_loss=0.02366, audio_tagging_loss=0.007437, over 15689.00 frames. ], tot_loss[loss=0.07696, simple_loss=0.09819, pruned_loss=0.01798, audio_tagging_loss=0.009889, over 3036758.54 frames. ], batch size: 55, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:53:58,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1272026.6666666667, ans=0.125
2023-11-20 23:54:02,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1272093.3333333333, ans=0.125
2023-11-20 23:54:05,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1272093.3333333333, ans=0.1
2023-11-20 23:54:35,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1272226.6666666667, ans=0.1
2023-11-20 23:54:46,589 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190850
2023-11-20 23:54:49,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1272293.3333333333, ans=0.125
2023-11-20 23:54:52,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1272360.0, ans=0.1
2023-11-20 23:54:53,834 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10500, loss[loss=0.06025, simple_loss=0.07215, pruned_loss=0.01498, audio_tagging_loss=0.009186, over 14979.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09668, pruned_loss=0.0176, audio_tagging_loss=0.009869, over 3037622.77 frames. ], batch size: 59, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:54:56,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1272360.0, ans=0.0
2023-11-20 23:55:13,739 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:55:26,076 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.127e+01 8.997e+01 9.660e+01 1.233e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-20 23:55:27,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1272493.3333333333, ans=0.125
2023-11-20 23:55:50,105 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190900
2023-11-20 23:55:57,802 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10550, loss[loss=0.07538, simple_loss=0.0911, pruned_loss=0.02037, audio_tagging_loss=0.009459, over 15896.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09726, pruned_loss=0.01778, audio_tagging_loss=0.009736, over 3036028.45 frames. ], batch size: 64, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:56:25,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1272826.6666666667, ans=0.125
2023-11-20 23:56:29,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1272826.6666666667, ans=0.125
2023-11-20 23:56:31,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1272826.6666666667, ans=0.0
2023-11-20 23:56:38,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1272893.3333333333, ans=0.0
2023-11-20 23:56:48,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1272960.0, ans=0.0
2023-11-20 23:56:53,473 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 190950
2023-11-20 23:56:59,106 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:57:01,271 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10600, loss[loss=0.07551, simple_loss=0.08953, pruned_loss=0.02112, audio_tagging_loss=0.009634, over 14867.00 frames. ], tot_loss[loss=0.07585, simple_loss=0.09696, pruned_loss=0.01763, audio_tagging_loss=0.009736, over 3038345.80 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:57:01,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1273026.6666666667, ans=0.125
2023-11-20 23:57:04,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1273026.6666666667, ans=0.125
2023-11-20 23:57:22,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1273093.3333333333, ans=0.125
2023-11-20 23:57:28,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1273160.0, ans=0.125
2023-11-20 23:57:32,708 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.948e+01 7.871e+01 8.482e+01 9.357e+01 1.125e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-20 23:57:42,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1273226.6666666667, ans=0.1
2023-11-20 23:57:56,264 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191000
2023-11-20 23:58:03,691 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10650, loss[loss=0.065, simple_loss=0.07942, pruned_loss=0.0142, audio_tagging_loss=0.01109, over 15532.00 frames. ], tot_loss[loss=0.07582, simple_loss=0.09687, pruned_loss=0.01771, audio_tagging_loss=0.009682, over 3041546.39 frames. ], batch size: 59, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:58:05,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1273360.0, ans=0.125
2023-11-20 23:58:18,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1273426.6666666667, ans=0.125
2023-11-20 23:58:24,677 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-20 23:59:00,055 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191050
2023-11-20 23:59:07,278 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10700, loss[loss=0.07383, simple_loss=0.0839, pruned_loss=0.0178, audio_tagging_loss=0.01408, over 14977.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09729, pruned_loss=0.01778, audio_tagging_loss=0.009673, over 3042554.27 frames. ], batch size: 58, lr: 4.25e-03, grad_scale: 16.0
2023-11-20 23:59:12,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1273693.3333333333, ans=0.125
2023-11-20 23:59:21,982 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.05 vs. limit=15.0
2023-11-20 23:59:24,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1273760.0, ans=0.2
2023-11-20 23:59:38,726 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 7.953e+01 8.635e+01 9.470e+01 1.499e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-20 23:59:53,342 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.35 vs. limit=12.0
2023-11-21 00:00:03,158 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191100
2023-11-21 00:00:10,872 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10750, loss[loss=0.05327, simple_loss=0.06619, pruned_loss=0.01213, audio_tagging_loss=0.008041, over 15896.00 frames. ], tot_loss[loss=0.07605, simple_loss=0.09709, pruned_loss=0.01782, audio_tagging_loss=0.009691, over 3040160.39 frames. ], batch size: 61, lr: 4.25e-03, grad_scale: 16.0
2023-11-21 00:00:11,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1274026.6666666667, ans=0.1
2023-11-21 00:00:12,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1274026.6666666667, ans=0.125
2023-11-21 00:00:26,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1274093.3333333333, ans=0.1
2023-11-21 00:00:27,171 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.43 vs. limit=15.0
2023-11-21 00:01:06,203 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191150
2023-11-21 00:01:13,358 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10800, loss[loss=0.08534, simple_loss=0.1091, pruned_loss=0.02192, audio_tagging_loss=0.008857, over 15411.00 frames. ], tot_loss[loss=0.076, simple_loss=0.09738, pruned_loss=0.01772, audio_tagging_loss=0.009595, over 3044761.20 frames. ], batch size: 58, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:01:18,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1274360.0, ans=0.1
2023-11-21 00:01:21,137 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.49 vs. limit=12.0
2023-11-21 00:01:27,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1274426.6666666667, ans=0.125
2023-11-21 00:01:27,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1274426.6666666667, ans=0.0
2023-11-21 00:01:38,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1274493.3333333333, ans=0.0
2023-11-21 00:01:44,976 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.243e+01 8.007e+01 8.718e+01 9.213e+01 1.138e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 00:02:08,915 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191200
2023-11-21 00:02:14,145 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.94 vs. limit=15.0
2023-11-21 00:02:16,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1274693.3333333333, ans=0.125
2023-11-21 00:02:17,044 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10850, loss[loss=0.0701, simple_loss=0.08693, pruned_loss=0.01692, audio_tagging_loss=0.009706, over 14776.00 frames. ], tot_loss[loss=0.07539, simple_loss=0.09656, pruned_loss=0.01742, audio_tagging_loss=0.009692, over 3042329.59 frames. ], batch size: 54, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:02:17,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1274693.3333333333, ans=0.125
2023-11-21 00:02:18,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.40 vs. limit=15.0
2023-11-21 00:02:38,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1274760.0, ans=0.0
2023-11-21 00:02:38,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1274760.0, ans=0.1
2023-11-21 00:02:50,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1274826.6666666667, ans=0.125
2023-11-21 00:03:13,476 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191250
2023-11-21 00:03:15,973 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:03:17,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1274960.0, ans=0.125
2023-11-21 00:03:21,478 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10900, loss[loss=0.1013, simple_loss=0.1385, pruned_loss=0.02404, audio_tagging_loss=0.007965, over 16003.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09748, pruned_loss=0.01764, audio_tagging_loss=0.009663, over 3034140.49 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:03:37,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1275093.3333333333, ans=0.125
2023-11-21 00:03:43,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1275093.3333333333, ans=0.0
2023-11-21 00:03:49,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.98 vs. limit=22.5
2023-11-21 00:03:52,726 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.158e+01 8.756e+01 9.674e+01 1.348e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 00:03:53,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1275160.0, ans=0.125
2023-11-21 00:04:16,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191300
2023-11-21 00:04:23,773 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 10950, loss[loss=0.0712, simple_loss=0.09594, pruned_loss=0.01359, audio_tagging_loss=0.009643, over 15974.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09766, pruned_loss=0.01765, audio_tagging_loss=0.00972, over 3039683.65 frames. ], batch size: 57, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:04:48,466 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.94 vs. limit=15.0
2023-11-21 00:05:00,559 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0
2023-11-21 00:05:19,754 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191350
2023-11-21 00:05:25,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1275626.6666666667, ans=0.1
2023-11-21 00:05:27,998 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11000, loss[loss=0.05649, simple_loss=0.06861, pruned_loss=0.01207, audio_tagging_loss=0.01011, over 14880.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09735, pruned_loss=0.01771, audio_tagging_loss=0.009789, over 3045517.49 frames. ], batch size: 56, lr: 4.25e-03, grad_scale: 32.0
2023-11-21 00:05:30,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1275693.3333333333, ans=0.07
2023-11-21 00:05:36,710 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:05:59,926 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.401e+01 8.037e+01 8.729e+01 9.395e+01 1.274e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 00:06:01,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1275826.6666666667, ans=0.0
2023-11-21 00:06:13,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1275893.3333333333, ans=0.025
2023-11-21 00:06:24,757 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191400
2023-11-21 00:06:29,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1275960.0, ans=0.07
2023-11-21 00:06:32,593 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11050, loss[loss=0.0591, simple_loss=0.07713, pruned_loss=0.009577, audio_tagging_loss=0.01096, over 13858.00 frames. ], tot_loss[loss=0.07592, simple_loss=0.09692, pruned_loss=0.01754, audio_tagging_loss=0.009916, over 3044518.68 frames. ], batch size: 52, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:06:52,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1276093.3333333333, ans=0.125
2023-11-21 00:07:10,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1276226.6666666667, ans=0.125
2023-11-21 00:07:14,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1276226.6666666667, ans=0.0
2023-11-21 00:07:23,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1276293.3333333333, ans=0.0
2023-11-21 00:07:29,119 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191450
2023-11-21 00:07:32,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1276293.3333333333, ans=0.125
2023-11-21 00:07:36,877 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11100, loss[loss=0.06877, simple_loss=0.09297, pruned_loss=0.01649, audio_tagging_loss=0.005795, over 15675.00 frames. ], tot_loss[loss=0.07588, simple_loss=0.09678, pruned_loss=0.01747, audio_tagging_loss=0.01002, over 3050905.78 frames. ], batch size: 58, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:07:37,103 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:07:39,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.07 vs. limit=22.5
2023-11-21 00:07:42,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1276360.0, ans=0.0
2023-11-21 00:08:05,513 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.26 vs. limit=15.0
2023-11-21 00:08:09,715 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.351e+01 9.133e+01 9.830e+01 1.252e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-21 00:08:32,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191500
2023-11-21 00:08:35,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1276626.6666666667, ans=0.125
2023-11-21 00:08:40,573 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11150, loss[loss=0.06347, simple_loss=0.07675, pruned_loss=0.01747, audio_tagging_loss=0.007627, over 13793.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09774, pruned_loss=0.01782, audio_tagging_loss=0.01012, over 3053508.75 frames. ], batch size: 54, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:08:51,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1276760.0, ans=0.1
2023-11-21 00:09:00,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1276760.0, ans=0.0
2023-11-21 00:09:04,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1276826.6666666667, ans=0.2
2023-11-21 00:09:08,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1276826.6666666667, ans=0.0
2023-11-21 00:09:12,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1276826.6666666667, ans=0.0
2023-11-21 00:09:15,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1276826.6666666667, ans=0.0
2023-11-21 00:09:32,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1276960.0, ans=0.0
2023-11-21 00:09:37,047 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191550
2023-11-21 00:09:39,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1276960.0, ans=0.125
2023-11-21 00:09:44,276 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11200, loss[loss=0.08438, simple_loss=0.1089, pruned_loss=0.02068, audio_tagging_loss=0.009249, over 15392.00 frames. ], tot_loss[loss=0.07643, simple_loss=0.09705, pruned_loss=0.01775, audio_tagging_loss=0.01016, over 3044973.02 frames. ], batch size: 57, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:10:18,325 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.318e+01 8.964e+01 9.909e+01 1.504e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 00:10:22,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1277226.6666666667, ans=0.125
2023-11-21 00:10:23,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1277226.6666666667, ans=0.1
2023-11-21 00:10:38,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1277293.3333333333, ans=0.125
2023-11-21 00:10:41,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191600
2023-11-21 00:10:42,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1277293.3333333333, ans=0.2
2023-11-21 00:10:47,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1277360.0, ans=0.0
2023-11-21 00:10:48,730 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11250, loss[loss=0.07218, simple_loss=0.08948, pruned_loss=0.01705, audio_tagging_loss=0.01039, over 15154.00 frames. ], tot_loss[loss=0.07685, simple_loss=0.09744, pruned_loss=0.01799, audio_tagging_loss=0.01014, over 3044570.24 frames. ], batch size: 58, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:10:58,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1277360.0, ans=0.1
2023-11-21 00:10:59,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1277360.0, ans=0.1
2023-11-21 00:11:20,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1277493.3333333333, ans=0.0
2023-11-21 00:11:46,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191650
2023-11-21 00:11:47,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1277626.6666666667, ans=0.2
2023-11-21 00:11:53,473 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11300, loss[loss=0.06109, simple_loss=0.07242, pruned_loss=0.01329, audio_tagging_loss=0.01159, over 14921.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09682, pruned_loss=0.01785, audio_tagging_loss=0.01001, over 3045730.26 frames. ], batch size: 57, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:11:55,617 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:11:59,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1277693.3333333333, ans=0.1
2023-11-21 00:12:26,955 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.170e+01 7.843e+01 8.601e+01 9.135e+01 1.178e+02, threshold=1.720e+02, percent-clipped=0.0
2023-11-21 00:12:36,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1277893.3333333333, ans=0.0
2023-11-21 00:12:44,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1277960.0, ans=0.125
2023-11-21 00:12:44,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1277960.0, ans=0.1
2023-11-21 00:12:44,850 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.21 vs. limit=15.0
2023-11-21 00:12:50,751 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191700
2023-11-21 00:12:55,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1277960.0, ans=0.125
2023-11-21 00:12:57,882 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11350, loss[loss=0.1029, simple_loss=0.1359, pruned_loss=0.0287, audio_tagging_loss=0.006259, over 15174.00 frames. ], tot_loss[loss=0.07626, simple_loss=0.09701, pruned_loss=0.01791, audio_tagging_loss=0.009845, over 3046631.98 frames. ], batch size: 56, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:13:08,925 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.35 vs. limit=12.0
2023-11-21 00:13:44,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1278226.6666666667, ans=0.1
2023-11-21 00:13:52,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1278293.3333333333, ans=0.1
2023-11-21 00:13:53,034 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191750
2023-11-21 00:14:00,789 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11400, loss[loss=0.07278, simple_loss=0.09124, pruned_loss=0.017, audio_tagging_loss=0.01016, over 15606.00 frames. ], tot_loss[loss=0.07695, simple_loss=0.09828, pruned_loss=0.01815, audio_tagging_loss=0.009656, over 3047040.28 frames. ], batch size: 59, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:14:03,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1278360.0, ans=0.125
2023-11-21 00:14:04,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1278360.0, ans=0.125
2023-11-21 00:14:24,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1278426.6666666667, ans=0.2
2023-11-21 00:14:34,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1278493.3333333333, ans=0.125
2023-11-21 00:14:35,210 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.101e+01 8.048e+01 8.638e+01 9.436e+01 1.129e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 00:14:39,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1278560.0, ans=0.2
2023-11-21 00:14:43,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1278560.0, ans=0.0
2023-11-21 00:14:43,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1278560.0, ans=0.0
2023-11-21 00:14:43,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1278560.0, ans=0.5
2023-11-21 00:14:57,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191800
2023-11-21 00:15:01,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1278626.6666666667, ans=0.0
2023-11-21 00:15:05,470 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11450, loss[loss=0.06266, simple_loss=0.08092, pruned_loss=0.01173, audio_tagging_loss=0.01047, over 16477.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.09864, pruned_loss=0.0182, audio_tagging_loss=0.009559, over 3051309.61 frames. ], batch size: 62, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:15:09,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1278693.3333333333, ans=0.125
2023-11-21 00:15:14,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1278693.3333333333, ans=0.2
2023-11-21 00:15:21,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1278760.0, ans=0.125
2023-11-21 00:15:28,508 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:15:28,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1278760.0, ans=0.125
2023-11-21 00:15:35,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1278826.6666666667, ans=0.07
2023-11-21 00:15:40,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1278826.6666666667, ans=0.0
2023-11-21 00:15:40,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1278826.6666666667, ans=0.1
2023-11-21 00:15:50,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1278893.3333333333, ans=0.025
2023-11-21 00:16:02,282 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191850
2023-11-21 00:16:09,429 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11500, loss[loss=0.08195, simple_loss=0.1076, pruned_loss=0.02115, audio_tagging_loss=0.007012, over 15638.00 frames. ], tot_loss[loss=0.07689, simple_loss=0.09838, pruned_loss=0.01814, audio_tagging_loss=0.00956, over 3046240.79 frames. ], batch size: 58, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:16:33,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2.whitening_limit, batch_count=1279160.0, ans=15.0
2023-11-21 00:16:42,860 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 7.974e+01 8.630e+01 9.256e+01 1.145e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-21 00:16:48,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1279226.6666666667, ans=0.2
2023-11-21 00:16:51,060 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.59 vs. limit=12.0
2023-11-21 00:16:57,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1279226.6666666667, ans=0.0
2023-11-21 00:17:04,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1279293.3333333333, ans=0.125
2023-11-21 00:17:05,384 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191900
2023-11-21 00:17:13,191 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11550, loss[loss=0.05707, simple_loss=0.06122, pruned_loss=0.01204, audio_tagging_loss=0.01442, over 13656.00 frames. ], tot_loss[loss=0.07674, simple_loss=0.09795, pruned_loss=0.0181, audio_tagging_loss=0.009661, over 3050834.69 frames. ], batch size: 52, lr: 4.24e-03, grad_scale: 16.0
2023-11-21 00:17:16,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1279360.0, ans=0.125
2023-11-21 00:17:24,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1279426.6666666667, ans=0.125
2023-11-21 00:17:29,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1279426.6666666667, ans=0.1
2023-11-21 00:17:35,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1279426.6666666667, ans=0.125
2023-11-21 00:17:36,084 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.05 vs. limit=22.5
2023-11-21 00:17:44,577 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.36 vs. limit=12.0
2023-11-21 00:17:51,078 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:18:02,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1279626.6666666667, ans=0.0
2023-11-21 00:18:06,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1279626.6666666667, ans=0.125
2023-11-21 00:18:08,758 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 191950
2023-11-21 00:18:16,017 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11600, loss[loss=0.097, simple_loss=0.1239, pruned_loss=0.02524, audio_tagging_loss=0.009831, over 15865.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09846, pruned_loss=0.01806, audio_tagging_loss=0.00963, over 3047366.04 frames. ], batch size: 60, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:18:41,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1279826.6666666667, ans=0.125
2023-11-21 00:18:51,115 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.722e+01 8.133e+01 8.974e+01 9.770e+01 1.687e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-21 00:18:51,776 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.31 vs. limit=22.5
2023-11-21 00:18:56,535 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.00 vs. limit=15.0
2023-11-21 00:19:13,342 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192000
2023-11-21 00:19:13,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1279960.0, ans=0.1
2023-11-21 00:19:14,822 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-192000.pt
2023-11-21 00:19:21,813 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0
2023-11-21 00:19:24,902 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11650, loss[loss=0.08621, simple_loss=0.1179, pruned_loss=0.02046, audio_tagging_loss=0.0068, over 15627.00 frames. ], tot_loss[loss=0.07685, simple_loss=0.09812, pruned_loss=0.01801, audio_tagging_loss=0.009783, over 3050441.83 frames. ], batch size: 56, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:19:41,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1280093.3333333333, ans=0.025
2023-11-21 00:19:44,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.18 vs. limit=15.0
2023-11-21 00:20:04,665 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.24 vs. limit=15.0
2023-11-21 00:20:06,006 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:20:21,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192050
2023-11-21 00:20:22,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1280293.3333333333, ans=0.2
2023-11-21 00:20:28,813 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11700, loss[loss=0.0949, simple_loss=0.1185, pruned_loss=0.02465, audio_tagging_loss=0.01099, over 15292.00 frames. ], tot_loss[loss=0.07683, simple_loss=0.09813, pruned_loss=0.01805, audio_tagging_loss=0.009719, over 3046621.02 frames. ], batch size: 58, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:20:35,425 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.88 vs. limit=15.0
2023-11-21 00:20:55,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1280493.3333333333, ans=0.125
2023-11-21 00:21:03,124 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.317e+01 8.852e+01 9.641e+01 1.228e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 00:21:08,573 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.87 vs. limit=15.0
2023-11-21 00:21:12,421 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.60 vs. limit=15.0
2023-11-21 00:21:17,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1280560.0, ans=0.125
2023-11-21 00:21:17,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1280560.0, ans=0.2
2023-11-21 00:21:24,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192100
2023-11-21 00:21:32,260 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11750, loss[loss=0.07223, simple_loss=0.09568, pruned_loss=0.01529, audio_tagging_loss=0.009101, over 15242.00 frames. ], tot_loss[loss=0.07715, simple_loss=0.09884, pruned_loss=0.01811, audio_tagging_loss=0.009625, over 3047306.73 frames. ], batch size: 56, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:21:32,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1280693.3333333333, ans=0.1
2023-11-21 00:21:34,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1280693.3333333333, ans=0.125
2023-11-21 00:21:44,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1280760.0, ans=0.09899494936611666
2023-11-21 00:22:01,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1280826.6666666667, ans=0.0
2023-11-21 00:22:19,410 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.81 vs. limit=15.0
2023-11-21 00:22:27,442 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192150
2023-11-21 00:22:33,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1280960.0, ans=0.0
2023-11-21 00:22:35,312 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11800, loss[loss=0.08248, simple_loss=0.1083, pruned_loss=0.01799, audio_tagging_loss=0.01035, over 13583.00 frames. ], tot_loss[loss=0.07644, simple_loss=0.09786, pruned_loss=0.01783, audio_tagging_loss=0.009679, over 3041766.53 frames. ], batch size: 53, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:22:42,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1281026.6666666667, ans=0.125
2023-11-21 00:22:43,556 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 00:23:09,498 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 8.064e+01 8.621e+01 9.481e+01 1.242e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-21 00:23:31,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192200
2023-11-21 00:23:39,821 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11850, loss[loss=0.06647, simple_loss=0.08052, pruned_loss=0.01505, audio_tagging_loss=0.01115, over 14444.00 frames. ], tot_loss[loss=0.07697, simple_loss=0.09864, pruned_loss=0.01787, audio_tagging_loss=0.009782, over 3042797.39 frames. ], batch size: 53, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:23:40,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1281360.0, ans=0.125
2023-11-21 00:23:44,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1281360.0, ans=0.125
2023-11-21 00:23:48,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1281360.0, ans=0.125
2023-11-21 00:24:07,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1281493.3333333333, ans=0.125
2023-11-21 00:24:09,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1281493.3333333333, ans=0.125
2023-11-21 00:24:12,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1281493.3333333333, ans=0.125
2023-11-21 00:24:30,980 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.95 vs. limit=15.0
2023-11-21 00:24:35,102 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192250
2023-11-21 00:24:40,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1281626.6666666667, ans=0.07
2023-11-21 00:24:42,294 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11900, loss[loss=0.08768, simple_loss=0.1168, pruned_loss=0.02115, audio_tagging_loss=0.008125, over 15280.00 frames. ], tot_loss[loss=0.07721, simple_loss=0.09863, pruned_loss=0.01792, audio_tagging_loss=0.009971, over 3037107.08 frames. ], batch size: 58, lr: 4.24e-03, grad_scale: 32.0
2023-11-21 00:24:50,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1281693.3333333333, ans=0.125
2023-11-21 00:25:17,101 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.019e+01 8.295e+01 9.404e+01 1.030e+02 1.674e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-21 00:25:27,819 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.92 vs. limit=15.0
2023-11-21 00:25:28,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1281893.3333333333, ans=0.0
2023-11-21 00:25:37,993 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192300
2023-11-21 00:25:45,986 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 11950, loss[loss=0.06562, simple_loss=0.08348, pruned_loss=0.0125, audio_tagging_loss=0.01137, over 15818.00 frames. ], tot_loss[loss=0.07638, simple_loss=0.09727, pruned_loss=0.01769, audio_tagging_loss=0.01005, over 3039052.64 frames. ], batch size: 58, lr: 4.23e-03, grad_scale: 32.0
2023-11-21 00:25:53,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1282026.6666666667, ans=0.2
2023-11-21 00:26:21,453 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.87 vs. limit=10.0
2023-11-21 00:26:25,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1282226.6666666667, ans=0.1
2023-11-21 00:26:36,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1282293.3333333333, ans=0.125
2023-11-21 00:26:39,419 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192350
2023-11-21 00:26:43,417 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.41 vs. limit=12.0
2023-11-21 00:26:46,485 INFO [train_asr.py:1221] (0/4) Epoch 16, batch 12000, loss[loss=0.07002, simple_loss=0.09009, pruned_loss=0.0136, audio_tagging_loss=0.01137, over 15691.00 frames. ], tot_loss[loss=0.07657, simple_loss=0.09747, pruned_loss=0.01771, audio_tagging_loss=0.01013, over 3049286.51 frames. ], batch size: 56, lr: 4.23e-03, grad_scale: 32.0
2023-11-21 00:26:46,488 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 00:27:30,242 INFO [train_asr.py:1253] (0/4) Epoch 16, validation: loss=0.06114, simple_loss=0.05299, pruned_loss=0.005583, audio_tagging_loss=0.02906, over 4681554.00 frames. 
2023-11-21 00:27:30,243 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 00:27:40,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1282426.6666666667, ans=0.125
2023-11-21 00:27:47,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1282426.6666666667, ans=0.2
2023-11-21 00:27:57,230 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-16.pt
2023-11-21 00:28:33,021 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 0, loss[loss=0.0897, simple_loss=0.1015, pruned_loss=0.01644, audio_tagging_loss=0.0225, over 16644.00 frames. ], tot_loss[loss=0.0897, simple_loss=0.1015, pruned_loss=0.01644, audio_tagging_loss=0.0225, over 16644.00 frames. ], batch size: 60, lr: 4.11e-03, grad_scale: 32.0
2023-11-21 00:28:33,024 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 00:28:55,988 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.0891, 2.3222, 2.9978, 2.5349], device='cuda:0')
2023-11-21 00:29:04,703 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8485, 4.9315, 4.9327, 4.8988], device='cuda:0')
2023-11-21 00:29:12,192 INFO [train_asr.py:1253] (0/4) Epoch 17, validation: loss=0.06074, simple_loss=0.05295, pruned_loss=0.005487, audio_tagging_loss=0.02878, over 4681554.00 frames. 
2023-11-21 00:29:12,193 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 00:29:18,856 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.373e+01 8.043e+01 8.765e+01 9.548e+01 1.252e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 00:29:22,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1282513.3333333333, ans=0.05
2023-11-21 00:29:23,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1282580.0, ans=0.1
2023-11-21 00:29:23,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1282580.0, ans=0.125
2023-11-21 00:29:25,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1282580.0, ans=0.0
2023-11-21 00:29:39,205 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192400
2023-11-21 00:29:41,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1282646.6666666667, ans=0.125
2023-11-21 00:30:16,442 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 50, loss[loss=0.08639, simple_loss=0.1077, pruned_loss=0.01792, audio_tagging_loss=0.01464, over 15701.00 frames. ], tot_loss[loss=0.08611, simple_loss=0.09694, pruned_loss=0.01835, audio_tagging_loss=0.01929, over 686085.07 frames. ], batch size: 58, lr: 4.11e-03, grad_scale: 32.0
2023-11-21 00:30:16,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1282846.6666666667, ans=0.0
2023-11-21 00:30:34,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1282913.3333333333, ans=0.1
2023-11-21 00:30:34,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1282913.3333333333, ans=0.0
2023-11-21 00:30:42,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192450
2023-11-21 00:30:45,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1282980.0, ans=0.125
2023-11-21 00:30:54,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1283046.6666666667, ans=0.5
2023-11-21 00:30:56,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1283046.6666666667, ans=0.1
2023-11-21 00:31:17,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1283113.3333333333, ans=0.125
2023-11-21 00:31:19,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1283180.0, ans=0.125
2023-11-21 00:31:19,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1283180.0, ans=0.125
2023-11-21 00:31:20,483 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 100, loss[loss=0.082, simple_loss=0.09404, pruned_loss=0.01704, audio_tagging_loss=0.01795, over 15094.00 frames. ], tot_loss[loss=0.0852, simple_loss=0.09754, pruned_loss=0.01824, audio_tagging_loss=0.01818, over 1210947.31 frames. ], batch size: 57, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:31:27,992 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.701e+01 8.625e+01 9.421e+01 9.913e+01 1.432e+02, threshold=1.884e+02, percent-clipped=0.0
2023-11-21 00:31:47,762 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192500
2023-11-21 00:32:18,915 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.61 vs. limit=15.0
2023-11-21 00:32:24,254 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 150, loss[loss=0.07785, simple_loss=0.1012, pruned_loss=0.01685, audio_tagging_loss=0.01042, over 14482.00 frames. ], tot_loss[loss=0.08277, simple_loss=0.09739, pruned_loss=0.01793, audio_tagging_loss=0.01615, over 1621557.71 frames. ], batch size: 54, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:32:36,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1283580.0, ans=0.1
2023-11-21 00:32:51,326 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192550
2023-11-21 00:32:57,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1283646.6666666667, ans=0.0
2023-11-21 00:33:02,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff2.min_abs, batch_count=1283713.3333333333, ans=0.1
2023-11-21 00:33:06,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1283713.3333333333, ans=0.95
2023-11-21 00:33:08,298 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.23 vs. limit=15.0
2023-11-21 00:33:16,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1283780.0, ans=0.0
2023-11-21 00:33:19,992 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.02 vs. limit=10.0
2023-11-21 00:33:29,858 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 200, loss[loss=0.08059, simple_loss=0.08827, pruned_loss=0.02153, audio_tagging_loss=0.01492, over 15055.00 frames. ], tot_loss[loss=0.08065, simple_loss=0.0968, pruned_loss=0.01795, audio_tagging_loss=0.0143, over 1932549.84 frames. ], batch size: 57, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:33:30,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1283846.6666666667, ans=0.05
2023-11-21 00:33:37,143 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.748e+01 8.165e+01 8.890e+01 9.863e+01 2.020e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-21 00:33:37,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1283846.6666666667, ans=0.95
2023-11-21 00:33:37,837 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.68 vs. limit=15.0
2023-11-21 00:33:56,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192600
2023-11-21 00:34:13,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1284046.6666666667, ans=0.125
2023-11-21 00:34:21,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1284113.3333333333, ans=0.1
2023-11-21 00:34:33,593 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 250, loss[loss=0.09703, simple_loss=0.1284, pruned_loss=0.02382, audio_tagging_loss=0.009013, over 14185.00 frames. ], tot_loss[loss=0.08009, simple_loss=0.09823, pruned_loss=0.01803, audio_tagging_loss=0.01294, over 2178856.73 frames. ], batch size: 54, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:34:47,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1284246.6666666667, ans=0.125
2023-11-21 00:35:00,421 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192650
2023-11-21 00:35:05,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1284313.3333333333, ans=0.125
2023-11-21 00:35:11,116 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.81 vs. limit=15.0
2023-11-21 00:35:27,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1284446.6666666667, ans=0.125
2023-11-21 00:35:37,464 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 300, loss[loss=0.08086, simple_loss=0.1034, pruned_loss=0.02034, audio_tagging_loss=0.008819, over 16139.00 frames. ], tot_loss[loss=0.07964, simple_loss=0.09894, pruned_loss=0.01813, audio_tagging_loss=0.01204, over 2370055.68 frames. ], batch size: 59, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:35:45,418 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 8.137e+01 8.919e+01 9.657e+01 1.268e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-21 00:36:03,713 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192700
2023-11-21 00:36:17,032 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.32 vs. limit=15.0
2023-11-21 00:36:30,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1284780.0, ans=0.125
2023-11-21 00:36:35,095 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.42 vs. limit=22.5
2023-11-21 00:36:40,677 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 350, loss[loss=0.06788, simple_loss=0.08308, pruned_loss=0.01737, audio_tagging_loss=0.008969, over 16032.00 frames. ], tot_loss[loss=0.07933, simple_loss=0.09975, pruned_loss=0.01822, audio_tagging_loss=0.01124, over 2520057.71 frames. ], batch size: 63, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:36:42,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1284846.6666666667, ans=0.2
2023-11-21 00:36:52,492 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.41 vs. limit=22.5
2023-11-21 00:37:07,026 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192750
2023-11-21 00:37:10,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1284980.0, ans=0.125
2023-11-21 00:37:20,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1285046.6666666667, ans=0.125
2023-11-21 00:37:28,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1285046.6666666667, ans=0.125
2023-11-21 00:37:29,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1285046.6666666667, ans=0.125
2023-11-21 00:37:44,035 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 400, loss[loss=0.1022, simple_loss=0.1268, pruned_loss=0.02997, audio_tagging_loss=0.008858, over 14367.00 frames. ], tot_loss[loss=0.07838, simple_loss=0.09924, pruned_loss=0.01801, audio_tagging_loss=0.01075, over 2634704.38 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 32.0
2023-11-21 00:37:44,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1285180.0, ans=0.125
2023-11-21 00:37:51,992 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.176e+01 8.863e+01 9.795e+01 2.108e+02, threshold=1.773e+02, percent-clipped=1.0
2023-11-21 00:38:11,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192800
2023-11-21 00:38:41,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1285446.6666666667, ans=0.125
2023-11-21 00:38:47,678 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 450, loss[loss=0.09371, simple_loss=0.1249, pruned_loss=0.02383, audio_tagging_loss=0.007439, over 16048.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09827, pruned_loss=0.01793, audio_tagging_loss=0.01055, over 2720117.10 frames. ], batch size: 58, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:39:14,785 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192850
2023-11-21 00:39:37,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1285780.0, ans=0.1
2023-11-21 00:39:44,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1285780.0, ans=0.125
2023-11-21 00:39:47,795 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.47 vs. limit=15.0
2023-11-21 00:39:50,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1285846.6666666667, ans=0.0
2023-11-21 00:39:51,828 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 500, loss[loss=0.06514, simple_loss=0.08596, pruned_loss=0.01237, audio_tagging_loss=0.009792, over 14855.00 frames. ], tot_loss[loss=0.07733, simple_loss=0.09813, pruned_loss=0.01791, audio_tagging_loss=0.01036, over 2797058.21 frames. ], batch size: 55, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:39:52,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.65 vs. limit=6.0
2023-11-21 00:39:54,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1285846.6666666667, ans=0.125
2023-11-21 00:40:00,405 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.902e+01 7.999e+01 8.766e+01 9.500e+01 1.232e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 00:40:18,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192900
2023-11-21 00:40:23,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1285980.0, ans=0.0
2023-11-21 00:40:29,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1286046.6666666667, ans=0.0
2023-11-21 00:40:41,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1286113.3333333333, ans=0.125
2023-11-21 00:40:49,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1286113.3333333333, ans=0.0
2023-11-21 00:40:55,083 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 550, loss[loss=0.07479, simple_loss=0.09803, pruned_loss=0.01587, audio_tagging_loss=0.009905, over 16792.00 frames. ], tot_loss[loss=0.07698, simple_loss=0.09808, pruned_loss=0.01772, audio_tagging_loss=0.01023, over 2858254.53 frames. ], batch size: 63, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:41:22,237 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 192950
2023-11-21 00:41:49,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1286446.6666666667, ans=0.2
2023-11-21 00:41:58,729 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 600, loss[loss=0.06292, simple_loss=0.07321, pruned_loss=0.01395, audio_tagging_loss=0.01237, over 14126.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09749, pruned_loss=0.01769, audio_tagging_loss=0.01013, over 2901629.18 frames. ], batch size: 55, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:42:07,185 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.459e+01 7.723e+01 8.573e+01 9.272e+01 1.267e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 00:42:07,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1286513.3333333333, ans=0.0
2023-11-21 00:42:24,971 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193000
2023-11-21 00:42:35,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1286713.3333333333, ans=0.0
2023-11-21 00:42:50,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1286780.0, ans=0.125
2023-11-21 00:42:54,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1286780.0, ans=0.125
2023-11-21 00:43:02,797 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 650, loss[loss=0.07015, simple_loss=0.07974, pruned_loss=0.01807, audio_tagging_loss=0.01221, over 15351.00 frames. ], tot_loss[loss=0.07674, simple_loss=0.09754, pruned_loss=0.01786, audio_tagging_loss=0.01011, over 2932836.68 frames. ], batch size: 59, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:43:29,219 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193050
2023-11-21 00:43:36,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1286980.0, ans=0.0
2023-11-21 00:43:41,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1287046.6666666667, ans=0.0
2023-11-21 00:43:51,147 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0
2023-11-21 00:43:55,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1287113.3333333333, ans=0.0
2023-11-21 00:44:01,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.whiten.whitening_limit, batch_count=1287113.3333333333, ans=12.0
2023-11-21 00:44:06,010 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 700, loss[loss=0.08619, simple_loss=0.1165, pruned_loss=0.01905, audio_tagging_loss=0.008907, over 16338.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09835, pruned_loss=0.01792, audio_tagging_loss=0.01005, over 2964095.04 frames. ], batch size: 59, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:44:15,337 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 7.999e+01 8.466e+01 9.113e+01 1.191e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-21 00:44:28,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1287246.6666666667, ans=0.0
2023-11-21 00:44:32,767 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193100
2023-11-21 00:44:32,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1287313.3333333333, ans=0.125
2023-11-21 00:44:48,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1287380.0, ans=0.1
2023-11-21 00:45:01,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1287446.6666666667, ans=0.0
2023-11-21 00:45:02,617 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.49 vs. limit=22.5
2023-11-21 00:45:08,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1287446.6666666667, ans=0.125
2023-11-21 00:45:10,281 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 750, loss[loss=0.05635, simple_loss=0.06591, pruned_loss=0.01214, audio_tagging_loss=0.01126, over 14113.00 frames. ], tot_loss[loss=0.07678, simple_loss=0.09763, pruned_loss=0.01788, audio_tagging_loss=0.01008, over 2989576.53 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:45:12,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1287513.3333333333, ans=0.125
2023-11-21 00:45:26,413 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.52 vs. limit=12.0
2023-11-21 00:45:30,422 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.38 vs. limit=6.0
2023-11-21 00:45:37,840 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193150
2023-11-21 00:45:55,414 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.88 vs. limit=12.0
2023-11-21 00:45:57,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1287713.3333333333, ans=0.5
2023-11-21 00:46:14,624 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 800, loss[loss=0.07209, simple_loss=0.09265, pruned_loss=0.01534, audio_tagging_loss=0.01043, over 15428.00 frames. ], tot_loss[loss=0.07723, simple_loss=0.09844, pruned_loss=0.01792, audio_tagging_loss=0.01009, over 3004113.69 frames. ], batch size: 57, lr: 4.10e-03, grad_scale: 32.0
2023-11-21 00:46:24,328 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.140e+01 8.941e+01 9.385e+01 1.183e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-21 00:46:30,017 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.99 vs. limit=6.0
2023-11-21 00:46:42,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193200
2023-11-21 00:46:52,724 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.46 vs. limit=15.0
2023-11-21 00:47:19,553 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 850, loss[loss=0.05371, simple_loss=0.06352, pruned_loss=0.01125, audio_tagging_loss=0.0107, over 16402.00 frames. ], tot_loss[loss=0.07757, simple_loss=0.09906, pruned_loss=0.01799, audio_tagging_loss=0.01006, over 3020908.16 frames. ], batch size: 62, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:47:22,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1288180.0, ans=0.2
2023-11-21 00:47:25,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1288180.0, ans=0.1
2023-11-21 00:47:37,005 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.43 vs. limit=15.0
2023-11-21 00:47:37,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1288246.6666666667, ans=0.0
2023-11-21 00:47:42,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1288246.6666666667, ans=0.0
2023-11-21 00:47:45,985 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193250
2023-11-21 00:47:50,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=1288313.3333333333, ans=10.0
2023-11-21 00:47:57,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1288380.0, ans=0.125
2023-11-21 00:47:57,850 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.11 vs. limit=6.0
2023-11-21 00:47:58,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1288380.0, ans=0.1
2023-11-21 00:48:04,298 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.21 vs. limit=15.0
2023-11-21 00:48:12,982 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.74 vs. limit=12.0
2023-11-21 00:48:13,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1288446.6666666667, ans=0.1
2023-11-21 00:48:23,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.39 vs. limit=15.0
2023-11-21 00:48:23,681 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 900, loss[loss=0.04935, simple_loss=0.05546, pruned_loss=0.009262, audio_tagging_loss=0.01235, over 14739.00 frames. ], tot_loss[loss=0.0781, simple_loss=0.09981, pruned_loss=0.01817, audio_tagging_loss=0.01003, over 3034406.90 frames. ], batch size: 57, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:48:29,201 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.24 vs. limit=15.0
2023-11-21 00:48:33,408 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.920e+01 8.040e+01 8.568e+01 9.329e+01 1.303e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 00:48:35,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1288580.0, ans=0.0
2023-11-21 00:48:37,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1288580.0, ans=0.0
2023-11-21 00:48:38,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1288580.0, ans=0.125
2023-11-21 00:48:46,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1288580.0, ans=0.1
2023-11-21 00:48:50,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193300
2023-11-21 00:48:56,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1288646.6666666667, ans=0.125
2023-11-21 00:48:58,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1288646.6666666667, ans=0.0
2023-11-21 00:49:00,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1288646.6666666667, ans=0.125
2023-11-21 00:49:06,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1288713.3333333333, ans=0.09899494936611666
2023-11-21 00:49:27,456 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 950, loss[loss=0.08468, simple_loss=0.1103, pruned_loss=0.02101, audio_tagging_loss=0.008542, over 14986.00 frames. ], tot_loss[loss=0.07773, simple_loss=0.09942, pruned_loss=0.01812, audio_tagging_loss=0.009891, over 3034847.71 frames. ], batch size: 56, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:49:32,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1288846.6666666667, ans=0.125
2023-11-21 00:49:40,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1288913.3333333333, ans=0.125
2023-11-21 00:49:52,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1288980.0, ans=0.125
2023-11-21 00:49:55,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193350
2023-11-21 00:50:04,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1288980.0, ans=0.125
2023-11-21 00:50:09,744 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.69 vs. limit=22.5
2023-11-21 00:50:32,434 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1000, loss[loss=0.106, simple_loss=0.1452, pruned_loss=0.02722, audio_tagging_loss=0.006194, over 14870.00 frames. ], tot_loss[loss=0.07758, simple_loss=0.09918, pruned_loss=0.01813, audio_tagging_loss=0.009855, over 3029596.88 frames. ], batch size: 53, lr: 4.10e-03, grad_scale: 16.0
2023-11-21 00:50:42,908 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.002e+01 8.372e+01 8.934e+01 9.683e+01 2.784e+02, threshold=1.787e+02, percent-clipped=1.0
2023-11-21 00:50:49,407 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.93 vs. limit=12.0
2023-11-21 00:50:51,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1289246.6666666667, ans=0.125
2023-11-21 00:50:59,523 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:50:59,592 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193400
2023-11-21 00:50:59,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1289313.3333333333, ans=0.125
2023-11-21 00:51:11,848 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.92 vs. limit=12.0
2023-11-21 00:51:38,127 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1050, loss[loss=0.07974, simple_loss=0.1015, pruned_loss=0.01795, audio_tagging_loss=0.01102, over 15797.00 frames. ], tot_loss[loss=0.07679, simple_loss=0.098, pruned_loss=0.01791, audio_tagging_loss=0.009881, over 3032159.22 frames. ], batch size: 59, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 00:52:05,446 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193450
2023-11-21 00:52:10,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1289646.6666666667, ans=0.125
2023-11-21 00:52:25,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1289713.3333333333, ans=0.0
2023-11-21 00:52:38,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1289780.0, ans=0.125
2023-11-21 00:52:39,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1289780.0, ans=0.0
2023-11-21 00:52:41,790 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1100, loss[loss=0.08934, simple_loss=0.1082, pruned_loss=0.02635, audio_tagging_loss=0.008869, over 16249.00 frames. ], tot_loss[loss=0.07647, simple_loss=0.09788, pruned_loss=0.01783, audio_tagging_loss=0.009693, over 3035017.88 frames. ], batch size: 59, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 00:52:44,395 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:52:47,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1289846.6666666667, ans=0.09899494936611666
2023-11-21 00:52:49,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1289846.6666666667, ans=0.125
2023-11-21 00:52:52,953 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.642e+01 8.168e+01 8.741e+01 9.235e+01 1.137e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 00:52:55,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1289913.3333333333, ans=15.0
2023-11-21 00:53:09,389 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193500
2023-11-21 00:53:19,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1290046.6666666667, ans=0.125
2023-11-21 00:53:46,398 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1150, loss[loss=0.05703, simple_loss=0.06602, pruned_loss=0.01287, audio_tagging_loss=0.01115, over 14325.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09593, pruned_loss=0.01746, audio_tagging_loss=0.009779, over 3034291.82 frames. ], batch size: 55, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 00:53:53,054 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.81 vs. limit=15.0
2023-11-21 00:53:57,064 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=5.04 vs. limit=5.0
2023-11-21 00:54:01,579 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.48 vs. limit=15.0
2023-11-21 00:54:04,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1290246.6666666667, ans=0.0
2023-11-21 00:54:13,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193550
2023-11-21 00:54:23,443 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.30 vs. limit=10.0
2023-11-21 00:54:27,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1290380.0, ans=0.1
2023-11-21 00:54:29,744 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.88 vs. limit=15.0
2023-11-21 00:54:39,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1290446.6666666667, ans=0.04949747468305833
2023-11-21 00:54:51,246 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1200, loss[loss=0.08098, simple_loss=0.09596, pruned_loss=0.02209, audio_tagging_loss=0.01092, over 15070.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09657, pruned_loss=0.01751, audio_tagging_loss=0.009732, over 3038553.50 frames. ], batch size: 57, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:55:00,968 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 7.968e+01 8.692e+01 9.308e+01 1.785e+02, threshold=1.738e+02, percent-clipped=1.0
2023-11-21 00:55:18,160 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193600
2023-11-21 00:55:39,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.26 vs. limit=12.0
2023-11-21 00:55:52,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1290780.0, ans=0.125
2023-11-21 00:55:55,031 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1250, loss[loss=0.08232, simple_loss=0.1025, pruned_loss=0.0202, audio_tagging_loss=0.01086, over 15044.00 frames. ], tot_loss[loss=0.07527, simple_loss=0.09606, pruned_loss=0.01744, audio_tagging_loss=0.009799, over 3039121.47 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:56:13,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1290913.3333333333, ans=0.07
2023-11-21 00:56:22,014 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193650
2023-11-21 00:56:41,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1291046.6666666667, ans=0.2
2023-11-21 00:56:49,209 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.61 vs. limit=15.0
2023-11-21 00:56:59,685 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1300, loss[loss=0.08868, simple_loss=0.1184, pruned_loss=0.02168, audio_tagging_loss=0.007793, over 15043.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09584, pruned_loss=0.0176, audio_tagging_loss=0.009824, over 3037699.98 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:57:09,537 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.443e+01 7.829e+01 8.638e+01 9.326e+01 1.163e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 00:57:19,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1291246.6666666667, ans=0.0
2023-11-21 00:57:26,948 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193700
2023-11-21 00:58:03,422 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1350, loss[loss=0.06953, simple_loss=0.0915, pruned_loss=0.01439, audio_tagging_loss=0.009397, over 14784.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.09647, pruned_loss=0.01764, audio_tagging_loss=0.00978, over 3042463.68 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:58:04,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1291513.3333333333, ans=0.125
2023-11-21 00:58:30,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193750
2023-11-21 00:58:32,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1291646.6666666667, ans=0.125
2023-11-21 00:58:49,293 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 00:59:05,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1291780.0, ans=0.2
2023-11-21 00:59:07,697 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1400, loss[loss=0.07888, simple_loss=0.1098, pruned_loss=0.01789, audio_tagging_loss=0.006115, over 15346.00 frames. ], tot_loss[loss=0.07557, simple_loss=0.09602, pruned_loss=0.01768, audio_tagging_loss=0.009883, over 3042079.88 frames. ], batch size: 54, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 00:59:12,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1291846.6666666667, ans=0.1
2023-11-21 00:59:18,069 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.121e+01 7.749e+01 8.445e+01 9.345e+01 1.750e+02, threshold=1.689e+02, percent-clipped=1.0
2023-11-21 00:59:34,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193800
2023-11-21 00:59:49,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1292046.6666666667, ans=0.0
2023-11-21 00:59:54,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1292046.6666666667, ans=0.2
2023-11-21 01:00:05,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1292113.3333333333, ans=0.0
2023-11-21 01:00:12,440 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1450, loss[loss=0.07278, simple_loss=0.1005, pruned_loss=0.01423, audio_tagging_loss=0.008319, over 15954.00 frames. ], tot_loss[loss=0.07635, simple_loss=0.09678, pruned_loss=0.01813, audio_tagging_loss=0.00983, over 3041131.19 frames. ], batch size: 59, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:00:20,169 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:00:22,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1292180.0, ans=0.1
2023-11-21 01:00:26,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1292246.6666666667, ans=0.0
2023-11-21 01:00:26,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.34 vs. limit=15.0
2023-11-21 01:00:36,671 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.09 vs. limit=6.0
2023-11-21 01:00:38,787 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193850
2023-11-21 01:00:48,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.57 vs. limit=6.0
2023-11-21 01:00:51,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1292380.0, ans=0.2
2023-11-21 01:01:02,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1292446.6666666667, ans=0.0
2023-11-21 01:01:10,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1292446.6666666667, ans=0.125
2023-11-21 01:01:16,055 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1500, loss[loss=0.09043, simple_loss=0.1179, pruned_loss=0.02143, audio_tagging_loss=0.01003, over 14439.00 frames. ], tot_loss[loss=0.07752, simple_loss=0.09851, pruned_loss=0.01843, audio_tagging_loss=0.009834, over 3043873.29 frames. ], batch size: 53, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:01:20,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1292513.3333333333, ans=0.2
2023-11-21 01:01:20,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1292513.3333333333, ans=0.125
2023-11-21 01:01:27,685 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 8.416e+01 9.067e+01 9.838e+01 1.487e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-21 01:01:31,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1292580.0, ans=0.1
2023-11-21 01:01:42,968 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193900
2023-11-21 01:01:46,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1292646.6666666667, ans=0.0
2023-11-21 01:01:49,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1292646.6666666667, ans=0.1
2023-11-21 01:01:53,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1292646.6666666667, ans=0.125
2023-11-21 01:02:13,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1292780.0, ans=0.0
2023-11-21 01:02:17,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1292780.0, ans=0.1
2023-11-21 01:02:18,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1292780.0, ans=0.1
2023-11-21 01:02:20,419 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1550, loss[loss=0.07529, simple_loss=0.09328, pruned_loss=0.02016, audio_tagging_loss=0.00849, over 15083.00 frames. ], tot_loss[loss=0.07771, simple_loss=0.09905, pruned_loss=0.01838, audio_tagging_loss=0.009803, over 3039108.23 frames. ], batch size: 57, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:02:21,357 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.48 vs. limit=10.0
2023-11-21 01:02:33,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1292913.3333333333, ans=0.1
2023-11-21 01:02:36,342 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.97 vs. limit=15.0
2023-11-21 01:02:36,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1292913.3333333333, ans=0.0
2023-11-21 01:02:46,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1292980.0, ans=0.0
2023-11-21 01:02:46,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1292980.0, ans=0.125
2023-11-21 01:02:47,743 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 193950
2023-11-21 01:02:52,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1292980.0, ans=0.125
2023-11-21 01:02:58,750 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.43 vs. limit=22.5
2023-11-21 01:03:04,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1293046.6666666667, ans=0.0
2023-11-21 01:03:04,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1293046.6666666667, ans=0.0
2023-11-21 01:03:08,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1293046.6666666667, ans=0.125
2023-11-21 01:03:25,755 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1600, loss[loss=0.06904, simple_loss=0.09281, pruned_loss=0.01258, audio_tagging_loss=0.01006, over 15241.00 frames. ], tot_loss[loss=0.07758, simple_loss=0.0988, pruned_loss=0.01829, audio_tagging_loss=0.009885, over 3035007.88 frames. ], batch size: 57, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:03:36,991 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.454e+01 8.126e+01 8.877e+01 9.641e+01 1.501e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 01:03:51,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1293313.3333333333, ans=0.5
2023-11-21 01:03:52,661 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194000
2023-11-21 01:03:56,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1293313.3333333333, ans=0.2
2023-11-21 01:04:00,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1293313.3333333333, ans=0.0
2023-11-21 01:04:24,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1293446.6666666667, ans=0.1
2023-11-21 01:04:30,483 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1650, loss[loss=0.07963, simple_loss=0.1056, pruned_loss=0.01774, audio_tagging_loss=0.009104, over 16157.00 frames. ], tot_loss[loss=0.07747, simple_loss=0.09893, pruned_loss=0.0182, audio_tagging_loss=0.009801, over 3037956.44 frames. ], batch size: 62, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:04:39,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1293513.3333333333, ans=0.125
2023-11-21 01:04:48,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1293580.0, ans=0.125
2023-11-21 01:04:57,581 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194050
2023-11-21 01:05:00,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.24 vs. limit=12.0
2023-11-21 01:05:15,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1293713.3333333333, ans=0.125
2023-11-21 01:05:35,457 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1700, loss[loss=0.07329, simple_loss=0.09614, pruned_loss=0.01792, audio_tagging_loss=0.007303, over 14995.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.09838, pruned_loss=0.01798, audio_tagging_loss=0.009928, over 3035451.02 frames. ], batch size: 58, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:05:39,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1293846.6666666667, ans=0.125
2023-11-21 01:05:45,813 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.81 vs. limit=15.0
2023-11-21 01:05:46,333 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.167e+01 8.614e+01 9.209e+01 1.216e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 01:06:02,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.53 vs. limit=22.5
2023-11-21 01:06:02,879 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194100
2023-11-21 01:06:17,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1294046.6666666667, ans=0.125
2023-11-21 01:06:32,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1294113.3333333333, ans=0.125
2023-11-21 01:06:34,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1294113.3333333333, ans=0.125
2023-11-21 01:06:40,571 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1750, loss[loss=0.07008, simple_loss=0.09213, pruned_loss=0.01747, audio_tagging_loss=0.006541, over 14421.00 frames. ], tot_loss[loss=0.07664, simple_loss=0.09776, pruned_loss=0.01789, audio_tagging_loss=0.009873, over 3032415.74 frames. ], batch size: 54, lr: 4.09e-03, grad_scale: 32.0
2023-11-21 01:06:44,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1294180.0, ans=0.125
2023-11-21 01:06:54,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1294246.6666666667, ans=0.0
2023-11-21 01:07:07,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194150
2023-11-21 01:07:17,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1294380.0, ans=0.125
2023-11-21 01:07:44,359 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1800, loss[loss=0.08144, simple_loss=0.1178, pruned_loss=0.01707, audio_tagging_loss=0.005464, over 15826.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09827, pruned_loss=0.01795, audio_tagging_loss=0.009832, over 3034712.20 frames. ], batch size: 56, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:07:57,255 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.340e+01 8.982e+01 9.918e+01 1.410e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-21 01:07:57,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1294580.0, ans=0.125
2023-11-21 01:08:05,125 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.80 vs. limit=15.0
2023-11-21 01:08:10,793 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194200
2023-11-21 01:08:25,855 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.27 vs. limit=12.0
2023-11-21 01:08:31,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.98 vs. limit=15.0
2023-11-21 01:08:44,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1294780.0, ans=0.0
2023-11-21 01:08:48,999 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1850, loss[loss=0.1002, simple_loss=0.123, pruned_loss=0.02856, audio_tagging_loss=0.0101, over 15573.00 frames. ], tot_loss[loss=0.07679, simple_loss=0.09803, pruned_loss=0.01793, audio_tagging_loss=0.009853, over 3036149.18 frames. ], batch size: 58, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:09:04,579 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.64 vs. limit=22.5
2023-11-21 01:09:16,935 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194250
2023-11-21 01:09:18,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1294980.0, ans=0.125
2023-11-21 01:09:22,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1294980.0, ans=0.1
2023-11-21 01:09:24,012 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.73 vs. limit=5.0
2023-11-21 01:09:42,184 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:09:50,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1295113.3333333333, ans=0.0
2023-11-21 01:09:51,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1295180.0, ans=10.0
2023-11-21 01:09:52,825 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1900, loss[loss=0.08831, simple_loss=0.1081, pruned_loss=0.02511, audio_tagging_loss=0.009171, over 14937.00 frames. ], tot_loss[loss=0.07662, simple_loss=0.09746, pruned_loss=0.01798, audio_tagging_loss=0.009908, over 3034047.61 frames. ], batch size: 54, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:10:06,890 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.112e+01 7.944e+01 8.677e+01 9.299e+01 1.429e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 01:10:20,494 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0
2023-11-21 01:10:21,080 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194300
2023-11-21 01:10:32,659 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.90 vs. limit=15.0
2023-11-21 01:10:39,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1295380.0, ans=0.04949747468305833
2023-11-21 01:10:45,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1295446.6666666667, ans=0.125
2023-11-21 01:10:53,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1295446.6666666667, ans=0.0
2023-11-21 01:10:56,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1295446.6666666667, ans=0.125
2023-11-21 01:10:58,330 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 1950, loss[loss=0.0682, simple_loss=0.08684, pruned_loss=0.01514, audio_tagging_loss=0.009644, over 14213.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09783, pruned_loss=0.01804, audio_tagging_loss=0.009879, over 3041875.82 frames. ], batch size: 54, lr: 4.09e-03, grad_scale: 16.0
2023-11-21 01:11:06,470 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.92 vs. limit=22.5
2023-11-21 01:11:15,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1295580.0, ans=0.125
2023-11-21 01:11:19,398 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.78 vs. limit=15.0
2023-11-21 01:11:24,866 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194350
2023-11-21 01:11:26,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1295646.6666666667, ans=0.2
2023-11-21 01:11:52,752 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.78 vs. limit=15.0
2023-11-21 01:12:02,481 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2000, loss[loss=0.06607, simple_loss=0.07896, pruned_loss=0.01525, audio_tagging_loss=0.01134, over 14782.00 frames. ], tot_loss[loss=0.07663, simple_loss=0.0974, pruned_loss=0.01803, audio_tagging_loss=0.009905, over 3039762.32 frames. ], batch size: 58, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:12:10,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1295846.6666666667, ans=0.0
2023-11-21 01:12:14,655 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.274e+01 9.033e+01 9.827e+01 1.298e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 01:12:25,090 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.28 vs. limit=15.0
2023-11-21 01:12:29,408 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194400
2023-11-21 01:12:43,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1296046.6666666667, ans=0.1
2023-11-21 01:12:54,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1296113.3333333333, ans=0.125
2023-11-21 01:12:56,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1296113.3333333333, ans=0.125
2023-11-21 01:12:58,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1296113.3333333333, ans=0.0
2023-11-21 01:13:01,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1296113.3333333333, ans=0.125
2023-11-21 01:13:06,228 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2050, loss[loss=0.0791, simple_loss=0.1036, pruned_loss=0.0198, audio_tagging_loss=0.00748, over 15040.00 frames. ], tot_loss[loss=0.07657, simple_loss=0.09734, pruned_loss=0.01805, audio_tagging_loss=0.009852, over 3036794.85 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:13:09,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1296180.0, ans=0.1
2023-11-21 01:13:16,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1296180.0, ans=0.125
2023-11-21 01:13:19,129 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.14 vs. limit=22.5
2023-11-21 01:13:27,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1296246.6666666667, ans=0.125
2023-11-21 01:13:33,599 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194450
2023-11-21 01:13:43,670 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.87 vs. limit=15.0
2023-11-21 01:13:50,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1296380.0, ans=0.125
2023-11-21 01:14:10,856 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2100, loss[loss=0.06751, simple_loss=0.1007, pruned_loss=0.01064, audio_tagging_loss=0.006511, over 14837.00 frames. ], tot_loss[loss=0.07666, simple_loss=0.09773, pruned_loss=0.01806, audio_tagging_loss=0.009734, over 3040034.71 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:14:23,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.69 vs. limit=15.0
2023-11-21 01:14:24,177 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 8.053e+01 8.707e+01 9.415e+01 1.178e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-21 01:14:25,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1296580.0, ans=0.0
2023-11-21 01:14:30,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1296580.0, ans=0.125
2023-11-21 01:14:33,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1296580.0, ans=0.0
2023-11-21 01:14:37,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194500
2023-11-21 01:15:15,738 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2150, loss[loss=0.06925, simple_loss=0.08753, pruned_loss=0.01546, audio_tagging_loss=0.01002, over 15487.00 frames. ], tot_loss[loss=0.0766, simple_loss=0.09804, pruned_loss=0.01786, audio_tagging_loss=0.009711, over 3044454.94 frames. ], batch size: 61, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:15:18,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1296846.6666666667, ans=0.125
2023-11-21 01:15:42,472 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194550
2023-11-21 01:15:52,156 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:15:54,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1297046.6666666667, ans=0.125
2023-11-21 01:16:18,802 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2200, loss[loss=0.06804, simple_loss=0.09047, pruned_loss=0.01353, audio_tagging_loss=0.009275, over 15486.00 frames. ], tot_loss[loss=0.07705, simple_loss=0.09871, pruned_loss=0.01799, audio_tagging_loss=0.009703, over 3043347.48 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:16:21,968 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.68 vs. limit=15.0
2023-11-21 01:16:32,226 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 8.162e+01 8.764e+01 9.413e+01 1.161e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 01:16:32,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1297246.6666666667, ans=0.07
2023-11-21 01:16:45,742 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194600
2023-11-21 01:16:59,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1297380.0, ans=0.0
2023-11-21 01:17:04,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1297380.0, ans=0.05
2023-11-21 01:17:23,476 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2250, loss[loss=0.08027, simple_loss=0.09602, pruned_loss=0.01958, audio_tagging_loss=0.01268, over 13563.00 frames. ], tot_loss[loss=0.07716, simple_loss=0.09872, pruned_loss=0.01805, audio_tagging_loss=0.009753, over 3037883.61 frames. ], batch size: 53, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:17:50,765 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194650
2023-11-21 01:18:15,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1297780.0, ans=0.2
2023-11-21 01:18:22,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1297780.0, ans=0.2
2023-11-21 01:18:27,471 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2300, loss[loss=0.06168, simple_loss=0.07452, pruned_loss=0.01489, audio_tagging_loss=0.00953, over 16073.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09927, pruned_loss=0.01816, audio_tagging_loss=0.009797, over 3045397.09 frames. ], batch size: 61, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:18:39,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1297913.3333333333, ans=0.125
2023-11-21 01:18:41,609 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.557e+01 8.363e+01 8.884e+01 1.011e+02 1.300e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 01:18:47,333 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.77 vs. limit=15.0
2023-11-21 01:18:55,308 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194700
2023-11-21 01:18:57,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1297980.0, ans=0.125
2023-11-21 01:19:23,584 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:19:28,040 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.08 vs. limit=15.0
2023-11-21 01:19:32,001 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2350, loss[loss=0.07356, simple_loss=0.09643, pruned_loss=0.01697, audio_tagging_loss=0.008375, over 15160.00 frames. ], tot_loss[loss=0.07725, simple_loss=0.0988, pruned_loss=0.01799, audio_tagging_loss=0.00986, over 3043484.34 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:19:49,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten.whitening_limit, batch_count=1298246.6666666667, ans=15.0
2023-11-21 01:19:55,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1298246.6666666667, ans=0.125
2023-11-21 01:19:59,052 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194750
2023-11-21 01:20:03,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1298313.3333333333, ans=0.2
2023-11-21 01:20:16,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1298380.0, ans=0.05
2023-11-21 01:20:16,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1298380.0, ans=0.05
2023-11-21 01:20:21,729 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.57 vs. limit=15.0
2023-11-21 01:20:22,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.17 vs. limit=15.0
2023-11-21 01:20:36,895 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2400, loss[loss=0.08074, simple_loss=0.1054, pruned_loss=0.01722, audio_tagging_loss=0.0108, over 14415.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09835, pruned_loss=0.01785, audio_tagging_loss=0.009983, over 3043643.37 frames. ], batch size: 55, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:20:50,277 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.420e+01 8.164e+01 8.845e+01 9.386e+01 1.132e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 01:20:59,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.79 vs. limit=15.0
2023-11-21 01:21:03,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194800
2023-11-21 01:21:28,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1298780.0, ans=0.0
2023-11-21 01:21:31,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1298780.0, ans=0.2
2023-11-21 01:21:33,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1298780.0, ans=0.125
2023-11-21 01:21:37,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1298780.0, ans=0.0
2023-11-21 01:21:40,674 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2450, loss[loss=0.05936, simple_loss=0.08022, pruned_loss=0.01012, audio_tagging_loss=0.009134, over 15645.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.09833, pruned_loss=0.01788, audio_tagging_loss=0.01005, over 3045166.56 frames. ], batch size: 58, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:22:08,402 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194850
2023-11-21 01:22:11,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1298980.0, ans=0.07
2023-11-21 01:22:11,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1298980.0, ans=0.07
2023-11-21 01:22:12,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1298980.0, ans=0.5
2023-11-21 01:22:24,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1299046.6666666667, ans=0.035
2023-11-21 01:22:34,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1299113.3333333333, ans=0.2
2023-11-21 01:22:39,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1299113.3333333333, ans=0.125
2023-11-21 01:22:45,482 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2500, loss[loss=0.0737, simple_loss=0.09297, pruned_loss=0.01603, audio_tagging_loss=0.01118, over 14809.00 frames. ], tot_loss[loss=0.07742, simple_loss=0.09849, pruned_loss=0.0181, audio_tagging_loss=0.01007, over 3038414.99 frames. ], batch size: 57, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:23:01,472 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.181e+01 8.676e+01 9.218e+01 1.349e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 01:23:11,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1299313.3333333333, ans=0.125
2023-11-21 01:23:12,669 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194900
2023-11-21 01:23:22,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.40 vs. limit=10.0
2023-11-21 01:23:44,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1299446.6666666667, ans=0.1
2023-11-21 01:23:50,125 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2550, loss[loss=0.05999, simple_loss=0.07634, pruned_loss=0.01373, audio_tagging_loss=0.008099, over 13825.00 frames. ], tot_loss[loss=0.07699, simple_loss=0.09813, pruned_loss=0.01797, audio_tagging_loss=0.009946, over 3040315.88 frames. ], batch size: 55, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:23:56,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1299513.3333333333, ans=0.125
2023-11-21 01:24:01,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1299580.0, ans=0.125
2023-11-21 01:24:14,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1299646.6666666667, ans=0.125
2023-11-21 01:24:16,650 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 194950
2023-11-21 01:24:29,869 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.48 vs. limit=15.0
2023-11-21 01:24:52,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1299846.6666666667, ans=0.07
2023-11-21 01:24:53,754 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2600, loss[loss=0.08231, simple_loss=0.1136, pruned_loss=0.01764, audio_tagging_loss=0.00788, over 15812.00 frames. ], tot_loss[loss=0.07663, simple_loss=0.09783, pruned_loss=0.01788, audio_tagging_loss=0.009839, over 3034793.42 frames. ], batch size: 59, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:24:59,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1299846.6666666667, ans=0.0
2023-11-21 01:25:05,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1299913.3333333333, ans=0.015
2023-11-21 01:25:09,168 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.407e+01 7.907e+01 8.660e+01 9.294e+01 1.206e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 01:25:14,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1299913.3333333333, ans=0.125
2023-11-21 01:25:16,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1299913.3333333333, ans=0.0
2023-11-21 01:25:20,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195000
2023-11-21 01:25:35,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1300046.6666666667, ans=0.0
2023-11-21 01:25:47,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1300113.3333333333, ans=0.125
2023-11-21 01:25:49,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1300113.3333333333, ans=0.0
2023-11-21 01:25:58,425 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2650, loss[loss=0.05849, simple_loss=0.07128, pruned_loss=0.01352, audio_tagging_loss=0.009334, over 14710.00 frames. ], tot_loss[loss=0.07648, simple_loss=0.09774, pruned_loss=0.01788, audio_tagging_loss=0.009738, over 3038976.95 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:26:07,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1300180.0, ans=0.2
2023-11-21 01:26:16,230 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.73 vs. limit=6.0
2023-11-21 01:26:19,089 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.78 vs. limit=15.0
2023-11-21 01:26:20,550 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=15.0
2023-11-21 01:26:26,004 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195050
2023-11-21 01:26:30,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1300313.3333333333, ans=0.125
2023-11-21 01:26:35,187 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.17 vs. limit=22.5
2023-11-21 01:26:36,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1300380.0, ans=0.025
2023-11-21 01:26:39,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1300380.0, ans=0.125
2023-11-21 01:26:50,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1300446.6666666667, ans=0.1
2023-11-21 01:26:55,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1300446.6666666667, ans=0.125
2023-11-21 01:27:01,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1300446.6666666667, ans=0.125
2023-11-21 01:27:03,571 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2700, loss[loss=0.07487, simple_loss=0.09229, pruned_loss=0.01402, audio_tagging_loss=0.0147, over 15752.00 frames. ], tot_loss[loss=0.07601, simple_loss=0.09728, pruned_loss=0.01758, audio_tagging_loss=0.009787, over 3038251.18 frames. ], batch size: 61, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:27:10,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1300513.3333333333, ans=0.0
2023-11-21 01:27:13,446 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.12 vs. limit=15.0
2023-11-21 01:27:18,976 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 7.841e+01 8.793e+01 9.412e+01 1.255e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 01:27:22,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1300580.0, ans=0.125
2023-11-21 01:27:25,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1300580.0, ans=0.125
2023-11-21 01:27:30,625 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195100
2023-11-21 01:27:39,878 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.14 vs. limit=15.0
2023-11-21 01:28:01,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1300780.0, ans=0.125
2023-11-21 01:28:08,082 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2750, loss[loss=0.07255, simple_loss=0.09143, pruned_loss=0.01763, audio_tagging_loss=0.009213, over 15571.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09688, pruned_loss=0.01772, audio_tagging_loss=0.009742, over 3037674.11 frames. ], batch size: 57, lr: 4.08e-03, grad_scale: 16.0
2023-11-21 01:28:09,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1300846.6666666667, ans=0.125
2023-11-21 01:28:15,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1300846.6666666667, ans=0.0
2023-11-21 01:28:19,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1300913.3333333333, ans=0.035
2023-11-21 01:28:34,593 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195150
2023-11-21 01:28:47,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1301046.6666666667, ans=0.1
2023-11-21 01:28:53,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1301046.6666666667, ans=0.0
2023-11-21 01:29:02,050 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:29:04,747 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.43 vs. limit=15.0
2023-11-21 01:29:12,440 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2800, loss[loss=0.07602, simple_loss=0.1072, pruned_loss=0.01457, audio_tagging_loss=0.007854, over 16486.00 frames. ], tot_loss[loss=0.07579, simple_loss=0.09688, pruned_loss=0.01764, audio_tagging_loss=0.009713, over 3042521.14 frames. ], batch size: 60, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:29:12,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1301180.0, ans=0.2
2023-11-21 01:29:16,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1301180.0, ans=0.125
2023-11-21 01:29:26,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1301246.6666666667, ans=0.0
2023-11-21 01:29:27,663 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.332e+01 8.111e+01 8.659e+01 9.368e+01 1.203e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 01:29:35,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1301246.6666666667, ans=0.1
2023-11-21 01:29:39,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195200
2023-11-21 01:29:50,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1301380.0, ans=0.125
2023-11-21 01:30:14,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1301513.3333333333, ans=0.0
2023-11-21 01:30:16,067 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2850, loss[loss=0.06297, simple_loss=0.07919, pruned_loss=0.01353, audio_tagging_loss=0.009844, over 14139.00 frames. ], tot_loss[loss=0.07634, simple_loss=0.09769, pruned_loss=0.01788, audio_tagging_loss=0.009617, over 3042683.58 frames. ], batch size: 53, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:30:43,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195250
2023-11-21 01:30:53,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1301713.3333333333, ans=0.125
2023-11-21 01:30:55,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1301713.3333333333, ans=0.1
2023-11-21 01:30:56,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1301713.3333333333, ans=0.1
2023-11-21 01:30:58,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.58 vs. limit=22.5
2023-11-21 01:31:03,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1301713.3333333333, ans=0.1
2023-11-21 01:31:18,113 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.53 vs. limit=15.0
2023-11-21 01:31:21,336 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2900, loss[loss=0.06427, simple_loss=0.08121, pruned_loss=0.01143, audio_tagging_loss=0.01223, over 14478.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.09694, pruned_loss=0.01775, audio_tagging_loss=0.009709, over 3049370.41 frames. ], batch size: 56, lr: 4.08e-03, grad_scale: 32.0
2023-11-21 01:31:36,835 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.167e+01 8.153e+01 9.015e+01 9.723e+01 1.263e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-21 01:31:48,105 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195300
2023-11-21 01:31:54,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1301980.0, ans=0.125
2023-11-21 01:31:56,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1301980.0, ans=0.0
2023-11-21 01:31:59,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1302046.6666666667, ans=0.2
2023-11-21 01:32:07,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1302046.6666666667, ans=0.0
2023-11-21 01:32:18,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1302113.3333333333, ans=0.125
2023-11-21 01:32:26,138 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 2950, loss[loss=0.08921, simple_loss=0.115, pruned_loss=0.02211, audio_tagging_loss=0.009572, over 15969.00 frames. ], tot_loss[loss=0.07537, simple_loss=0.09616, pruned_loss=0.01755, audio_tagging_loss=0.009741, over 3054408.39 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:32:28,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1302180.0, ans=0.0
2023-11-21 01:32:31,955 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.49 vs. limit=12.0
2023-11-21 01:32:38,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1302246.6666666667, ans=0.125
2023-11-21 01:32:53,745 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195350
2023-11-21 01:33:03,937 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.29 vs. limit=12.0
2023-11-21 01:33:29,901 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3000, loss[loss=0.07649, simple_loss=0.1127, pruned_loss=0.01289, audio_tagging_loss=0.00724, over 14761.00 frames. ], tot_loss[loss=0.07618, simple_loss=0.09734, pruned_loss=0.01769, audio_tagging_loss=0.009822, over 3050090.08 frames. ], batch size: 53, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:33:29,905 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 01:33:54,190 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.1749, 2.2997, 4.9989, 2.6820], device='cuda:0')
2023-11-21 01:33:58,354 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.2313, 3.1101, 2.7483, 2.8312, 3.4206, 3.4402, 3.0091, 3.5889],
       device='cuda:0')
2023-11-21 01:34:10,814 INFO [train_asr.py:1253] (0/4) Epoch 17, validation: loss=0.06009, simple_loss=0.05276, pruned_loss=0.005332, audio_tagging_loss=0.02838, over 4681554.00 frames. 
2023-11-21 01:34:10,815 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 01:34:27,240 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.276e+01 8.086e+01 8.879e+01 9.716e+01 1.329e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 01:34:28,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1302580.0, ans=0.0
2023-11-21 01:34:32,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1302580.0, ans=0.0
2023-11-21 01:34:37,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195400
2023-11-21 01:34:49,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1302713.3333333333, ans=0.5
2023-11-21 01:34:55,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1302713.3333333333, ans=0.07
2023-11-21 01:35:15,257 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3050, loss[loss=0.07878, simple_loss=0.09817, pruned_loss=0.01789, audio_tagging_loss=0.0118, over 15711.00 frames. ], tot_loss[loss=0.0765, simple_loss=0.09785, pruned_loss=0.01773, audio_tagging_loss=0.009844, over 3048556.86 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:35:20,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1302846.6666666667, ans=0.2
2023-11-21 01:35:21,781 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.92 vs. limit=15.0
2023-11-21 01:35:37,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1302913.3333333333, ans=0.125
2023-11-21 01:35:41,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.27 vs. limit=15.0
2023-11-21 01:35:41,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195450
2023-11-21 01:35:52,280 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:35:56,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=15.0
2023-11-21 01:35:58,285 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.31 vs. limit=22.5
2023-11-21 01:36:02,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1303046.6666666667, ans=0.125
2023-11-21 01:36:04,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1303046.6666666667, ans=0.04949747468305833
2023-11-21 01:36:11,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1303113.3333333333, ans=0.125
2023-11-21 01:36:14,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1303113.3333333333, ans=0.0
2023-11-21 01:36:18,595 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3100, loss[loss=0.06498, simple_loss=0.08073, pruned_loss=0.01433, audio_tagging_loss=0.01029, over 14991.00 frames. ], tot_loss[loss=0.07636, simple_loss=0.09726, pruned_loss=0.0177, audio_tagging_loss=0.01002, over 3043778.49 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:36:25,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1303180.0, ans=0.0
2023-11-21 01:36:36,454 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.588e+01 8.198e+01 8.813e+01 9.373e+01 1.166e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 01:36:44,393 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.44 vs. limit=22.5
2023-11-21 01:36:46,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195500
2023-11-21 01:36:51,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1303313.3333333333, ans=0.025
2023-11-21 01:36:53,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1303313.3333333333, ans=0.125
2023-11-21 01:36:57,015 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:36:57,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1303380.0, ans=0.0
2023-11-21 01:36:59,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1303380.0, ans=0.125
2023-11-21 01:37:14,829 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:37:23,951 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3150, loss[loss=0.07469, simple_loss=0.1076, pruned_loss=0.01303, audio_tagging_loss=0.007862, over 16465.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.09758, pruned_loss=0.01758, audio_tagging_loss=0.01005, over 3043162.71 frames. ], batch size: 60, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:37:50,717 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195550
2023-11-21 01:38:19,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1303780.0, ans=0.0
2023-11-21 01:38:28,818 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3200, loss[loss=0.08714, simple_loss=0.1079, pruned_loss=0.02141, audio_tagging_loss=0.01177, over 13455.00 frames. ], tot_loss[loss=0.07744, simple_loss=0.09924, pruned_loss=0.01779, audio_tagging_loss=0.01003, over 3045406.98 frames. ], batch size: 53, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:38:36,962 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-21 01:38:42,546 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:38:44,769 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.303e+01 8.338e+01 8.946e+01 9.583e+01 3.907e+02, threshold=1.789e+02, percent-clipped=1.0
2023-11-21 01:38:52,405 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.64 vs. limit=10.0
2023-11-21 01:38:55,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195600
2023-11-21 01:39:03,275 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.09 vs. limit=22.5
2023-11-21 01:39:32,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1304180.0, ans=0.125
2023-11-21 01:39:33,048 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3250, loss[loss=0.08803, simple_loss=0.1103, pruned_loss=0.02361, audio_tagging_loss=0.00929, over 14977.00 frames. ], tot_loss[loss=0.07783, simple_loss=0.09977, pruned_loss=0.0179, audio_tagging_loss=0.01004, over 3046097.05 frames. ], batch size: 56, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:39:34,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1304180.0, ans=0.125
2023-11-21 01:39:51,873 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.98 vs. limit=22.5
2023-11-21 01:39:52,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1304246.6666666667, ans=0.09899494936611666
2023-11-21 01:40:00,426 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195650
2023-11-21 01:40:32,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1304446.6666666667, ans=0.125
2023-11-21 01:40:37,908 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3300, loss[loss=0.06365, simple_loss=0.06838, pruned_loss=0.01736, audio_tagging_loss=0.0121, over 16274.00 frames. ], tot_loss[loss=0.07746, simple_loss=0.09894, pruned_loss=0.01788, audio_tagging_loss=0.01011, over 3048078.45 frames. ], batch size: 63, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:40:51,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1304580.0, ans=0.07
2023-11-21 01:40:56,299 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.228e+01 8.124e+01 8.901e+01 9.520e+01 1.377e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 01:41:05,092 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195700
2023-11-21 01:41:24,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1304713.3333333333, ans=0.0
2023-11-21 01:41:41,960 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3350, loss[loss=0.05181, simple_loss=0.06391, pruned_loss=0.007419, audio_tagging_loss=0.01244, over 14837.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.09856, pruned_loss=0.01798, audio_tagging_loss=0.01004, over 3048824.42 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:42:05,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten.whitening_limit, batch_count=1304913.3333333333, ans=15.0
2023-11-21 01:42:06,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1304980.0, ans=0.0
2023-11-21 01:42:08,303 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195750
2023-11-21 01:42:16,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1304980.0, ans=0.125
2023-11-21 01:42:42,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1305113.3333333333, ans=0.2
2023-11-21 01:42:45,582 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3400, loss[loss=0.08279, simple_loss=0.1137, pruned_loss=0.019, audio_tagging_loss=0.006932, over 16225.00 frames. ], tot_loss[loss=0.07691, simple_loss=0.09825, pruned_loss=0.01793, audio_tagging_loss=0.009855, over 3043070.71 frames. ], batch size: 59, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:43:00,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1305246.6666666667, ans=0.125
2023-11-21 01:43:03,984 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 8.029e+01 8.814e+01 9.645e+01 1.986e+02, threshold=1.763e+02, percent-clipped=1.0
2023-11-21 01:43:04,484 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:43:04,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1305246.6666666667, ans=0.0
2023-11-21 01:43:05,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.47 vs. limit=10.0
2023-11-21 01:43:12,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195800
2023-11-21 01:43:25,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1305380.0, ans=0.0
2023-11-21 01:43:34,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1305380.0, ans=10.0
2023-11-21 01:43:48,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1305446.6666666667, ans=0.1
2023-11-21 01:43:51,463 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3450, loss[loss=0.0679, simple_loss=0.07805, pruned_loss=0.01646, audio_tagging_loss=0.01242, over 14575.00 frames. ], tot_loss[loss=0.07731, simple_loss=0.09905, pruned_loss=0.01807, audio_tagging_loss=0.009709, over 3052475.87 frames. ], batch size: 58, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:43:54,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1305513.3333333333, ans=0.125
2023-11-21 01:44:07,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1305580.0, ans=0.125
2023-11-21 01:44:14,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1305580.0, ans=0.125
2023-11-21 01:44:18,457 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195850
2023-11-21 01:44:35,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1305713.3333333333, ans=0.2
2023-11-21 01:44:36,093 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.28 vs. limit=15.0
2023-11-21 01:44:42,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1305780.0, ans=0.0
2023-11-21 01:44:52,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1305780.0, ans=0.125
2023-11-21 01:44:56,179 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3500, loss[loss=0.1053, simple_loss=0.1346, pruned_loss=0.03329, audio_tagging_loss=0.004702, over 14994.00 frames. ], tot_loss[loss=0.07694, simple_loss=0.09854, pruned_loss=0.01803, audio_tagging_loss=0.009634, over 3055740.36 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:45:13,925 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.440e+01 8.307e+01 8.800e+01 9.894e+01 1.360e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 01:45:23,397 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195900
2023-11-21 01:45:29,980 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:45:48,590 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.30 vs. limit=6.0
2023-11-21 01:45:55,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1306113.3333333333, ans=0.2
2023-11-21 01:46:00,105 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.78 vs. limit=15.0
2023-11-21 01:46:00,383 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3550, loss[loss=0.06566, simple_loss=0.09195, pruned_loss=0.00973, audio_tagging_loss=0.009958, over 15738.00 frames. ], tot_loss[loss=0.07718, simple_loss=0.09904, pruned_loss=0.01812, audio_tagging_loss=0.009537, over 3051747.24 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:46:06,077 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.39 vs. limit=10.0
2023-11-21 01:46:27,490 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 195950
2023-11-21 01:46:53,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1306446.6666666667, ans=0.1
2023-11-21 01:46:59,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1306446.6666666667, ans=0.1
2023-11-21 01:47:04,449 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3600, loss[loss=0.09147, simple_loss=0.1214, pruned_loss=0.02152, audio_tagging_loss=0.009235, over 15990.00 frames. ], tot_loss[loss=0.07679, simple_loss=0.09845, pruned_loss=0.01805, audio_tagging_loss=0.009515, over 3045431.38 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:47:22,589 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.078e+01 8.527e+01 9.382e+01 1.423e+02, threshold=1.705e+02, percent-clipped=0.0
2023-11-21 01:47:24,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1306580.0, ans=0.0
2023-11-21 01:47:27,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1306580.0, ans=0.2
2023-11-21 01:47:31,151 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196000
2023-11-21 01:47:32,597 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-196000.pt
2023-11-21 01:47:36,759 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.44 vs. limit=15.0
2023-11-21 01:47:41,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1306646.6666666667, ans=0.0
2023-11-21 01:47:52,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=1306713.3333333333, ans=10.0
2023-11-21 01:48:12,009 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3650, loss[loss=0.08115, simple_loss=0.1014, pruned_loss=0.02103, audio_tagging_loss=0.009428, over 15904.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09859, pruned_loss=0.01822, audio_tagging_loss=0.009601, over 3039712.08 frames. ], batch size: 61, lr: 4.07e-03, grad_scale: 32.0
2023-11-21 01:48:35,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1306913.3333333333, ans=0.125
2023-11-21 01:48:36,912 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.61 vs. limit=15.0
2023-11-21 01:48:38,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196050
2023-11-21 01:48:39,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1306980.0, ans=0.2
2023-11-21 01:48:54,369 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.48 vs. limit=22.5
2023-11-21 01:49:15,775 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3700, loss[loss=0.0817, simple_loss=0.1088, pruned_loss=0.01959, audio_tagging_loss=0.007736, over 15343.00 frames. ], tot_loss[loss=0.07778, simple_loss=0.09966, pruned_loss=0.01849, audio_tagging_loss=0.009461, over 3041947.14 frames. ], batch size: 57, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:49:19,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1307180.0, ans=0.07
2023-11-21 01:49:36,155 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.153e+01 8.365e+01 9.163e+01 1.051e+02 1.464e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-21 01:49:40,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1307246.6666666667, ans=0.1
2023-11-21 01:49:42,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1307313.3333333333, ans=0.125
2023-11-21 01:49:43,735 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196100
2023-11-21 01:49:49,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1307313.3333333333, ans=0.125
2023-11-21 01:50:17,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1307446.6666666667, ans=0.125
2023-11-21 01:50:18,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1307446.6666666667, ans=0.125
2023-11-21 01:50:21,119 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3750, loss[loss=0.05808, simple_loss=0.08127, pruned_loss=0.01022, audio_tagging_loss=0.007223, over 14615.00 frames. ], tot_loss[loss=0.07789, simple_loss=0.1, pruned_loss=0.01842, audio_tagging_loss=0.009454, over 3043653.87 frames. ], batch size: 55, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:50:48,182 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196150
2023-11-21 01:51:04,877 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 01:51:05,800 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:51:09,726 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.54 vs. limit=22.5
2023-11-21 01:51:25,133 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3800, loss[loss=0.07914, simple_loss=0.09738, pruned_loss=0.02011, audio_tagging_loss=0.01035, over 15884.00 frames. ], tot_loss[loss=0.07757, simple_loss=0.09926, pruned_loss=0.01828, audio_tagging_loss=0.009662, over 3051535.15 frames. ], batch size: 60, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:51:44,164 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.062e+01 7.915e+01 8.603e+01 9.483e+01 1.210e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 01:51:52,368 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196200
2023-11-21 01:52:14,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1308046.6666666667, ans=0.0
2023-11-21 01:52:17,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1308113.3333333333, ans=0.0
2023-11-21 01:52:29,389 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3850, loss[loss=0.08381, simple_loss=0.1091, pruned_loss=0.01842, audio_tagging_loss=0.01087, over 14944.00 frames. ], tot_loss[loss=0.0773, simple_loss=0.099, pruned_loss=0.01804, audio_tagging_loss=0.009755, over 3052800.90 frames. ], batch size: 54, lr: 4.07e-03, grad_scale: 16.0
2023-11-21 01:52:34,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1308180.0, ans=0.125
2023-11-21 01:52:38,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1308180.0, ans=0.1
2023-11-21 01:52:43,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1308246.6666666667, ans=0.2
2023-11-21 01:52:43,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1308246.6666666667, ans=0.0
2023-11-21 01:52:50,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1308246.6666666667, ans=0.07
2023-11-21 01:52:56,926 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196250
2023-11-21 01:53:15,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1308380.0, ans=0.125
2023-11-21 01:53:19,747 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.56 vs. limit=15.0
2023-11-21 01:53:33,575 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3900, loss[loss=0.08926, simple_loss=0.1174, pruned_loss=0.02026, audio_tagging_loss=0.01029, over 16343.00 frames. ], tot_loss[loss=0.07781, simple_loss=0.09985, pruned_loss=0.01811, audio_tagging_loss=0.009776, over 3048326.69 frames. ], batch size: 61, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 01:53:33,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1308513.3333333333, ans=0.125
2023-11-21 01:53:34,132 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.48 vs. limit=6.0
2023-11-21 01:53:46,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1308580.0, ans=0.125
2023-11-21 01:53:52,217 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.909e+01 8.160e+01 8.803e+01 9.797e+01 1.334e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 01:54:00,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196300
2023-11-21 01:54:06,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1308646.6666666667, ans=0.125
2023-11-21 01:54:09,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1308646.6666666667, ans=0.125
2023-11-21 01:54:23,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1308780.0, ans=0.125
2023-11-21 01:54:37,453 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 3950, loss[loss=0.05744, simple_loss=0.0614, pruned_loss=0.01478, audio_tagging_loss=0.01195, over 14477.00 frames. ], tot_loss[loss=0.07723, simple_loss=0.09895, pruned_loss=0.01789, audio_tagging_loss=0.009859, over 3043566.77 frames. ], batch size: 56, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 01:54:45,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1308846.6666666667, ans=0.0
2023-11-21 01:54:50,937 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.23 vs. limit=15.0
2023-11-21 01:55:03,850 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196350
2023-11-21 01:55:10,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1308980.0, ans=0.125
2023-11-21 01:55:23,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1309046.6666666667, ans=0.0
2023-11-21 01:55:29,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1309113.3333333333, ans=0.0
2023-11-21 01:55:29,776 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.74 vs. limit=22.5
2023-11-21 01:55:35,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1309113.3333333333, ans=0.125
2023-11-21 01:55:41,772 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4000, loss[loss=0.06257, simple_loss=0.08443, pruned_loss=0.01181, audio_tagging_loss=0.008542, over 14767.00 frames. ], tot_loss[loss=0.07776, simple_loss=0.09933, pruned_loss=0.01821, audio_tagging_loss=0.009884, over 3046385.13 frames. ], batch size: 54, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 01:55:57,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1309246.6666666667, ans=0.1
2023-11-21 01:56:00,792 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.301e+01 8.874e+01 9.780e+01 1.152e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 01:56:09,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196400
2023-11-21 01:56:35,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1309446.6666666667, ans=0.2
2023-11-21 01:56:45,661 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4050, loss[loss=0.08242, simple_loss=0.1104, pruned_loss=0.02013, audio_tagging_loss=0.007063, over 15215.00 frames. ], tot_loss[loss=0.07778, simple_loss=0.09911, pruned_loss=0.01823, audio_tagging_loss=0.009989, over 3047151.29 frames. ], batch size: 54, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 01:56:49,898 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:56:57,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1309513.3333333333, ans=0.0
2023-11-21 01:57:13,977 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196450
2023-11-21 01:57:29,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1309713.3333333333, ans=10.0
2023-11-21 01:57:51,779 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4100, loss[loss=0.08157, simple_loss=0.1106, pruned_loss=0.01893, audio_tagging_loss=0.007326, over 15288.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09856, pruned_loss=0.0181, audio_tagging_loss=0.009991, over 3043173.89 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 01:58:10,136 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.51 vs. limit=15.0
2023-11-21 01:58:10,691 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.112e+01 8.710e+01 9.440e+01 1.199e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 01:58:18,126 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196500
2023-11-21 01:58:27,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1309980.0, ans=0.0
2023-11-21 01:58:32,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1310046.6666666667, ans=0.0
2023-11-21 01:58:40,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1310046.6666666667, ans=0.0
2023-11-21 01:58:41,705 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.04 vs. limit=22.5
2023-11-21 01:58:56,165 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4150, loss[loss=0.0802, simple_loss=0.1083, pruned_loss=0.0163, audio_tagging_loss=0.009745, over 15366.00 frames. ], tot_loss[loss=0.0771, simple_loss=0.09824, pruned_loss=0.01807, audio_tagging_loss=0.009913, over 3049923.38 frames. ], batch size: 55, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 01:59:23,465 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196550
2023-11-21 01:59:24,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1310313.3333333333, ans=0.125
2023-11-21 01:59:26,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1310313.3333333333, ans=0.0
2023-11-21 01:59:26,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1310313.3333333333, ans=0.0
2023-11-21 01:59:43,285 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 01:59:48,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1310446.6666666667, ans=0.0
2023-11-21 02:00:00,216 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4200, loss[loss=0.1033, simple_loss=0.1338, pruned_loss=0.02917, audio_tagging_loss=0.007255, over 16067.00 frames. ], tot_loss[loss=0.07639, simple_loss=0.09779, pruned_loss=0.01775, audio_tagging_loss=0.009743, over 3044598.13 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:00:11,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1310513.3333333333, ans=0.125
2023-11-21 02:00:12,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1310580.0, ans=0.125
2023-11-21 02:00:20,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1310580.0, ans=0.1
2023-11-21 02:00:21,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1310580.0, ans=0.125
2023-11-21 02:00:21,786 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.242e+01 8.811e+01 9.654e+01 1.216e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 02:00:23,267 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:00:25,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1310646.6666666667, ans=0.0
2023-11-21 02:00:25,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1310646.6666666667, ans=0.125
2023-11-21 02:00:28,540 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196600
2023-11-21 02:00:42,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1310713.3333333333, ans=0.07
2023-11-21 02:01:05,046 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4250, loss[loss=0.08329, simple_loss=0.106, pruned_loss=0.01913, audio_tagging_loss=0.01116, over 15220.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09776, pruned_loss=0.0179, audio_tagging_loss=0.009629, over 3043093.52 frames. ], batch size: 56, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:01:21,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1310913.3333333333, ans=0.125
2023-11-21 02:01:32,420 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196650
2023-11-21 02:01:41,545 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0
2023-11-21 02:01:42,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1311046.6666666667, ans=0.0
2023-11-21 02:02:03,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1311113.3333333333, ans=0.125
2023-11-21 02:02:09,889 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4300, loss[loss=0.06778, simple_loss=0.08164, pruned_loss=0.01476, audio_tagging_loss=0.0122, over 15648.00 frames. ], tot_loss[loss=0.0774, simple_loss=0.0993, pruned_loss=0.01819, audio_tagging_loss=0.009564, over 3042392.42 frames. ], batch size: 59, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:02:27,754 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.35 vs. limit=22.5
2023-11-21 02:02:29,314 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.098e+01 8.656e+01 9.268e+01 1.139e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 02:02:36,094 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196700
2023-11-21 02:02:47,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1311380.0, ans=0.1
2023-11-21 02:02:47,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1311380.0, ans=0.125
2023-11-21 02:02:52,340 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.37 vs. limit=15.0
2023-11-21 02:02:58,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1311380.0, ans=0.0
2023-11-21 02:02:58,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1311380.0, ans=0.125
2023-11-21 02:03:05,638 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.89 vs. limit=15.0
2023-11-21 02:03:13,563 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4350, loss[loss=0.09332, simple_loss=0.1117, pruned_loss=0.02125, audio_tagging_loss=0.0162, over 14977.00 frames. ], tot_loss[loss=0.07698, simple_loss=0.09879, pruned_loss=0.018, audio_tagging_loss=0.009586, over 3042093.18 frames. ], batch size: 53, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:03:21,480 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.66 vs. limit=22.5
2023-11-21 02:03:31,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1311580.0, ans=0.0
2023-11-21 02:03:38,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1311646.6666666667, ans=0.1
2023-11-21 02:03:40,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196750
2023-11-21 02:03:49,155 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.12 vs. limit=15.0
2023-11-21 02:03:55,303 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=12.0
2023-11-21 02:03:56,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1311713.3333333333, ans=0.125
2023-11-21 02:04:17,759 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4400, loss[loss=0.07441, simple_loss=0.09073, pruned_loss=0.01967, audio_tagging_loss=0.009379, over 15888.00 frames. ], tot_loss[loss=0.07672, simple_loss=0.09839, pruned_loss=0.01796, audio_tagging_loss=0.009573, over 3048024.36 frames. ], batch size: 59, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 02:04:18,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=1311846.6666666667, ans=15.0
2023-11-21 02:04:36,882 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.05 vs. limit=6.0
2023-11-21 02:04:38,517 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 7.827e+01 8.393e+01 8.998e+01 1.090e+02, threshold=1.679e+02, percent-clipped=0.0
2023-11-21 02:04:44,781 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196800
2023-11-21 02:04:51,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1311980.0, ans=0.125
2023-11-21 02:05:05,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1312046.6666666667, ans=0.125
2023-11-21 02:05:22,849 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4450, loss[loss=0.08329, simple_loss=0.09726, pruned_loss=0.02482, audio_tagging_loss=0.00984, over 15505.00 frames. ], tot_loss[loss=0.07688, simple_loss=0.09888, pruned_loss=0.01797, audio_tagging_loss=0.009469, over 3042464.10 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 02:05:49,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196850
2023-11-21 02:05:57,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1312313.3333333333, ans=0.2
2023-11-21 02:05:59,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1312380.0, ans=0.0
2023-11-21 02:06:15,141 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.83 vs. limit=15.0
2023-11-21 02:06:17,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1312446.6666666667, ans=0.2
2023-11-21 02:06:18,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1312446.6666666667, ans=0.0
2023-11-21 02:06:20,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1312446.6666666667, ans=0.125
2023-11-21 02:06:26,541 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4500, loss[loss=0.06445, simple_loss=0.07938, pruned_loss=0.0126, audio_tagging_loss=0.01216, over 15083.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09934, pruned_loss=0.01805, audio_tagging_loss=0.009421, over 3047489.47 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:06:31,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1312513.3333333333, ans=0.125
2023-11-21 02:06:36,923 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.60 vs. limit=10.0
2023-11-21 02:06:48,063 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.230e+01 8.666e+01 9.490e+01 1.272e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 02:06:52,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1312646.6666666667, ans=0.0
2023-11-21 02:06:53,037 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196900
2023-11-21 02:06:57,719 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0
2023-11-21 02:07:30,281 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4550, loss[loss=0.08114, simple_loss=0.1072, pruned_loss=0.01601, audio_tagging_loss=0.01151, over 15426.00 frames. ], tot_loss[loss=0.07682, simple_loss=0.09902, pruned_loss=0.01783, audio_tagging_loss=0.009479, over 3050718.29 frames. ], batch size: 56, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:07:56,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1312980.0, ans=0.125
2023-11-21 02:07:57,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 196950
2023-11-21 02:08:19,066 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 02:08:33,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1313180.0, ans=0.125
2023-11-21 02:08:34,855 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4600, loss[loss=0.0684, simple_loss=0.07904, pruned_loss=0.01501, audio_tagging_loss=0.01386, over 16817.00 frames. ], tot_loss[loss=0.07654, simple_loss=0.09829, pruned_loss=0.01778, audio_tagging_loss=0.009612, over 3053639.52 frames. ], batch size: 67, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:08:35,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1313180.0, ans=0.1
2023-11-21 02:08:47,143 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.34 vs. limit=22.5
2023-11-21 02:08:56,246 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.061e+01 8.654e+01 9.730e+01 1.161e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 02:09:01,181 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197000
2023-11-21 02:09:06,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1313313.3333333333, ans=0.0
2023-11-21 02:09:12,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1313380.0, ans=0.2
2023-11-21 02:09:38,903 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4650, loss[loss=0.08027, simple_loss=0.1019, pruned_loss=0.02048, audio_tagging_loss=0.008839, over 14296.00 frames. ], tot_loss[loss=0.07644, simple_loss=0.0977, pruned_loss=0.01781, audio_tagging_loss=0.009771, over 3051832.68 frames. ], batch size: 53, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:09:44,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1313513.3333333333, ans=0.125
2023-11-21 02:09:44,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1313513.3333333333, ans=0.125
2023-11-21 02:09:56,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1313580.0, ans=0.125
2023-11-21 02:10:03,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1313646.6666666667, ans=0.125
2023-11-21 02:10:03,982 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.49 vs. limit=10.0
2023-11-21 02:10:05,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197050
2023-11-21 02:10:06,617 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=13.14 vs. limit=15.0
2023-11-21 02:10:31,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1313780.0, ans=0.04949747468305833
2023-11-21 02:10:36,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1313780.0, ans=0.04949747468305833
2023-11-21 02:10:42,957 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4700, loss[loss=0.06534, simple_loss=0.08469, pruned_loss=0.01367, audio_tagging_loss=0.009327, over 15594.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09668, pruned_loss=0.01757, audio_tagging_loss=0.009901, over 3048815.14 frames. ], batch size: 59, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:11:04,738 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.068e+01 8.673e+01 9.283e+01 1.102e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 02:11:08,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1313980.0, ans=15.0
2023-11-21 02:11:09,813 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197100
2023-11-21 02:11:47,005 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4750, loss[loss=0.07444, simple_loss=0.09554, pruned_loss=0.01383, audio_tagging_loss=0.01284, over 15247.00 frames. ], tot_loss[loss=0.07674, simple_loss=0.09786, pruned_loss=0.01788, audio_tagging_loss=0.009934, over 3044762.95 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 16.0
2023-11-21 02:11:56,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1314180.0, ans=0.1
2023-11-21 02:12:13,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197150
2023-11-21 02:12:46,366 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0
2023-11-21 02:12:50,792 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4800, loss[loss=0.06494, simple_loss=0.08552, pruned_loss=0.01151, audio_tagging_loss=0.01068, over 15486.00 frames. ], tot_loss[loss=0.07656, simple_loss=0.09774, pruned_loss=0.01773, audio_tagging_loss=0.00996, over 3042276.91 frames. ], batch size: 58, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 02:13:03,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1314580.0, ans=0.025
2023-11-21 02:13:09,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1314580.0, ans=10.0
2023-11-21 02:13:12,702 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 8.143e+01 8.825e+01 9.696e+01 1.559e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-21 02:13:13,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1314580.0, ans=0.07
2023-11-21 02:13:18,350 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197200
2023-11-21 02:13:26,782 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.22 vs. limit=15.0
2023-11-21 02:13:39,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1314713.3333333333, ans=0.1
2023-11-21 02:13:47,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1314780.0, ans=0.1
2023-11-21 02:13:49,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1314780.0, ans=0.2
2023-11-21 02:13:55,409 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4850, loss[loss=0.07281, simple_loss=0.0836, pruned_loss=0.02154, audio_tagging_loss=0.009468, over 14710.00 frames. ], tot_loss[loss=0.07696, simple_loss=0.09818, pruned_loss=0.01776, audio_tagging_loss=0.01011, over 3040991.52 frames. ], batch size: 57, lr: 4.06e-03, grad_scale: 32.0
2023-11-21 02:13:57,276 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.05 vs. limit=6.0
2023-11-21 02:14:06,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1314846.6666666667, ans=0.125
2023-11-21 02:14:11,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1314913.3333333333, ans=0.0
2023-11-21 02:14:17,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1314913.3333333333, ans=0.125
2023-11-21 02:14:22,856 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197250
2023-11-21 02:14:25,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1314980.0, ans=0.1
2023-11-21 02:14:39,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1315046.6666666667, ans=0.125
2023-11-21 02:14:53,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1315113.3333333333, ans=0.125
2023-11-21 02:14:59,950 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4900, loss[loss=0.077, simple_loss=0.1014, pruned_loss=0.01913, audio_tagging_loss=0.007163, over 15057.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.09653, pruned_loss=0.01724, audio_tagging_loss=0.01011, over 3038410.41 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:15:21,163 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.125e+01 8.648e+01 9.394e+01 1.252e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 02:15:22,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1315246.6666666667, ans=0.125
2023-11-21 02:15:26,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197300
2023-11-21 02:15:29,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1315313.3333333333, ans=0.0
2023-11-21 02:15:33,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1315313.3333333333, ans=0.0
2023-11-21 02:15:33,322 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.42 vs. limit=22.5
2023-11-21 02:15:34,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1315313.3333333333, ans=0.2
2023-11-21 02:15:39,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1315380.0, ans=0.025
2023-11-21 02:16:03,695 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 4950, loss[loss=0.1046, simple_loss=0.1327, pruned_loss=0.02923, audio_tagging_loss=0.008997, over 14381.00 frames. ], tot_loss[loss=0.07571, simple_loss=0.09692, pruned_loss=0.01739, audio_tagging_loss=0.009858, over 3042357.69 frames. ], batch size: 53, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:16:07,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1315513.3333333333, ans=0.125
2023-11-21 02:16:08,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1315513.3333333333, ans=0.125
2023-11-21 02:16:15,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1315580.0, ans=0.015
2023-11-21 02:16:20,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1315580.0, ans=0.0
2023-11-21 02:16:21,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1315580.0, ans=0.125
2023-11-21 02:16:31,051 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197350
2023-11-21 02:17:02,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1315780.0, ans=0.125
2023-11-21 02:17:05,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1315780.0, ans=0.2
2023-11-21 02:17:07,627 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5000, loss[loss=0.06604, simple_loss=0.08623, pruned_loss=0.01258, audio_tagging_loss=0.01035, over 15451.00 frames. ], tot_loss[loss=0.07652, simple_loss=0.09829, pruned_loss=0.01772, audio_tagging_loss=0.009656, over 3046251.53 frames. ], batch size: 58, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:17:07,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1315846.6666666667, ans=0.125
2023-11-21 02:17:28,475 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.282e-01
2023-11-21 02:17:29,319 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.592e+01 8.139e+01 8.846e+01 9.798e+01 1.314e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 02:17:34,895 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197400
2023-11-21 02:17:36,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1315980.0, ans=0.0
2023-11-21 02:17:49,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1316046.6666666667, ans=0.1
2023-11-21 02:17:49,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1316046.6666666667, ans=0.0
2023-11-21 02:17:51,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1316046.6666666667, ans=0.1
2023-11-21 02:17:56,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1316046.6666666667, ans=0.125
2023-11-21 02:18:10,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1316113.3333333333, ans=0.125
2023-11-21 02:18:12,076 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5050, loss[loss=0.0807, simple_loss=0.1081, pruned_loss=0.01844, audio_tagging_loss=0.008223, over 14117.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09777, pruned_loss=0.01766, audio_tagging_loss=0.009623, over 3042506.56 frames. ], batch size: 54, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:18:27,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1316246.6666666667, ans=0.04949747468305833
2023-11-21 02:18:36,601 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.38 vs. limit=15.0
2023-11-21 02:18:38,393 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197450
2023-11-21 02:18:50,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1316380.0, ans=0.125
2023-11-21 02:18:52,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1316380.0, ans=0.125
2023-11-21 02:18:52,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1316380.0, ans=0.125
2023-11-21 02:18:53,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1316380.0, ans=0.0
2023-11-21 02:19:00,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.29 vs. limit=22.5
2023-11-21 02:19:16,006 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5100, loss[loss=0.0743, simple_loss=0.09857, pruned_loss=0.01409, audio_tagging_loss=0.01093, over 15983.00 frames. ], tot_loss[loss=0.076, simple_loss=0.09755, pruned_loss=0.01761, audio_tagging_loss=0.009621, over 3041972.25 frames. ], batch size: 60, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:19:21,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1316513.3333333333, ans=0.1
2023-11-21 02:19:28,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1316580.0, ans=0.1
2023-11-21 02:19:37,077 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.469e+01 7.939e+01 8.776e+01 9.689e+01 1.456e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 02:19:42,565 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197500
2023-11-21 02:19:52,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1316713.3333333333, ans=0.1
2023-11-21 02:20:18,825 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5150, loss[loss=0.09058, simple_loss=0.1236, pruned_loss=0.01949, audio_tagging_loss=0.009276, over 16224.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09716, pruned_loss=0.01761, audio_tagging_loss=0.009642, over 3037021.51 frames. ], batch size: 58, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:20:26,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1316846.6666666667, ans=0.125
2023-11-21 02:20:30,260 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=15.24 vs. limit=15.0
2023-11-21 02:20:46,686 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197550
2023-11-21 02:20:57,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1317046.6666666667, ans=0.2
2023-11-21 02:21:22,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1317180.0, ans=0.0
2023-11-21 02:21:22,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1317180.0, ans=0.0
2023-11-21 02:21:23,752 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5200, loss[loss=0.06374, simple_loss=0.08054, pruned_loss=0.01314, audio_tagging_loss=0.01033, over 14178.00 frames. ], tot_loss[loss=0.07588, simple_loss=0.09733, pruned_loss=0.01759, audio_tagging_loss=0.009632, over 3043939.89 frames. ], batch size: 54, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:21:35,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1317246.6666666667, ans=0.125
2023-11-21 02:21:44,686 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.249e+01 8.824e+01 9.467e+01 1.175e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-21 02:21:49,711 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197600
2023-11-21 02:22:01,784 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.75 vs. limit=8.0
2023-11-21 02:22:13,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1317446.6666666667, ans=0.125
2023-11-21 02:22:25,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1317446.6666666667, ans=0.0
2023-11-21 02:22:27,584 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5250, loss[loss=0.07132, simple_loss=0.08147, pruned_loss=0.02019, audio_tagging_loss=0.01039, over 15008.00 frames. ], tot_loss[loss=0.07658, simple_loss=0.09848, pruned_loss=0.0178, audio_tagging_loss=0.009539, over 3045020.82 frames. ], batch size: 61, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:22:53,806 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.27 vs. limit=15.0
2023-11-21 02:22:54,336 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197650
2023-11-21 02:23:30,862 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5300, loss[loss=0.09066, simple_loss=0.1178, pruned_loss=0.02501, audio_tagging_loss=0.006745, over 14768.00 frames. ], tot_loss[loss=0.07748, simple_loss=0.09962, pruned_loss=0.01819, audio_tagging_loss=0.009479, over 3046342.17 frames. ], batch size: 53, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:23:38,911 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.30 vs. limit=15.0
2023-11-21 02:23:42,812 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:23:45,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1317913.3333333333, ans=0.125
2023-11-21 02:23:53,267 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.065e+01 8.722e+01 9.594e+01 1.316e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 02:23:58,317 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197700
2023-11-21 02:24:03,082 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.18 vs. limit=15.0
2023-11-21 02:24:28,819 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=12.95 vs. limit=15.0
2023-11-21 02:24:34,806 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5350, loss[loss=0.07767, simple_loss=0.1036, pruned_loss=0.01794, audio_tagging_loss=0.007915, over 15426.00 frames. ], tot_loss[loss=0.07775, simple_loss=0.09988, pruned_loss=0.01824, audio_tagging_loss=0.009572, over 3031969.70 frames. ], batch size: 56, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:24:52,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1318246.6666666667, ans=0.2
2023-11-21 02:24:58,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1318246.6666666667, ans=0.125
2023-11-21 02:25:02,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197750
2023-11-21 02:25:34,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1318446.6666666667, ans=0.125
2023-11-21 02:25:36,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1318446.6666666667, ans=0.2
2023-11-21 02:25:38,682 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5400, loss[loss=0.08747, simple_loss=0.1003, pruned_loss=0.02417, audio_tagging_loss=0.01317, over 14523.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09893, pruned_loss=0.01805, audio_tagging_loss=0.009636, over 3033619.73 frames. ], batch size: 53, lr: 4.05e-03, grad_scale: 32.0
2023-11-21 02:25:40,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1318513.3333333333, ans=0.07
2023-11-21 02:25:54,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1318580.0, ans=0.125
2023-11-21 02:25:59,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1318580.0, ans=0.1
2023-11-21 02:26:00,538 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.677e+01 8.214e+01 8.750e+01 9.356e+01 1.169e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 02:26:04,384 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197800
2023-11-21 02:26:25,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1318713.3333333333, ans=0.0
2023-11-21 02:26:26,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1318713.3333333333, ans=0.125
2023-11-21 02:26:41,189 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=15.57 vs. limit=15.0
2023-11-21 02:26:41,772 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5450, loss[loss=0.05838, simple_loss=0.07129, pruned_loss=0.01205, audio_tagging_loss=0.01068, over 14455.00 frames. ], tot_loss[loss=0.07684, simple_loss=0.09849, pruned_loss=0.01794, audio_tagging_loss=0.009661, over 3036804.98 frames. ], batch size: 55, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:27:09,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197850
2023-11-21 02:27:27,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1319046.6666666667, ans=0.1
2023-11-21 02:27:44,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1319180.0, ans=0.125
2023-11-21 02:27:45,095 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5500, loss[loss=0.07308, simple_loss=0.1008, pruned_loss=0.01473, audio_tagging_loss=0.00794, over 16401.00 frames. ], tot_loss[loss=0.07648, simple_loss=0.09817, pruned_loss=0.0177, audio_tagging_loss=0.009689, over 3038935.98 frames. ], batch size: 58, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:27:45,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1319180.0, ans=0.0
2023-11-21 02:27:58,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1319246.6666666667, ans=0.07
2023-11-21 02:28:04,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1319246.6666666667, ans=0.2
2023-11-21 02:28:08,771 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.510e+01 8.155e+01 8.783e+01 9.511e+01 1.245e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 02:28:09,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1319246.6666666667, ans=0.125
2023-11-21 02:28:12,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197900
2023-11-21 02:28:35,470 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.77 vs. limit=15.0
2023-11-21 02:28:41,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1319446.6666666667, ans=0.1
2023-11-21 02:28:49,923 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5550, loss[loss=0.08558, simple_loss=0.09889, pruned_loss=0.02626, audio_tagging_loss=0.009873, over 16503.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.09828, pruned_loss=0.01782, audio_tagging_loss=0.00981, over 3042616.42 frames. ], batch size: 62, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:28:54,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1319513.3333333333, ans=0.125
2023-11-21 02:28:54,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1319513.3333333333, ans=0.05
2023-11-21 02:29:15,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 197950
2023-11-21 02:29:22,543 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.11 vs. limit=15.0
2023-11-21 02:29:23,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1319646.6666666667, ans=0.2
2023-11-21 02:29:28,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1319713.3333333333, ans=0.0
2023-11-21 02:29:33,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1319713.3333333333, ans=0.125
2023-11-21 02:29:39,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1319780.0, ans=0.1
2023-11-21 02:29:50,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1319780.0, ans=0.0
2023-11-21 02:29:50,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1319780.0, ans=0.125
2023-11-21 02:29:52,785 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5600, loss[loss=0.07903, simple_loss=0.09807, pruned_loss=0.02138, audio_tagging_loss=0.008621, over 13767.00 frames. ], tot_loss[loss=0.07675, simple_loss=0.09833, pruned_loss=0.01771, audio_tagging_loss=0.009873, over 3038381.18 frames. ], batch size: 54, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:30:01,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1319846.6666666667, ans=0.1
2023-11-21 02:30:08,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1319913.3333333333, ans=0.05
2023-11-21 02:30:13,700 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0
2023-11-21 02:30:17,158 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.123e+01 8.788e+01 9.495e+01 1.515e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 02:30:17,488 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:30:18,807 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:30:19,714 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198000
2023-11-21 02:30:26,189 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.57 vs. limit=12.0
2023-11-21 02:30:29,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1319980.0, ans=0.125
2023-11-21 02:30:37,140 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.31 vs. limit=22.5
2023-11-21 02:30:37,689 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 02:30:55,891 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5650, loss[loss=0.07642, simple_loss=0.1002, pruned_loss=0.02009, audio_tagging_loss=0.006203, over 15422.00 frames. ], tot_loss[loss=0.07667, simple_loss=0.09827, pruned_loss=0.01765, audio_tagging_loss=0.009883, over 3052338.12 frames. ], batch size: 57, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:31:13,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1320246.6666666667, ans=0.025
2023-11-21 02:31:14,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1320246.6666666667, ans=0.125
2023-11-21 02:31:16,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1320246.6666666667, ans=0.1
2023-11-21 02:31:18,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1320246.6666666667, ans=10.0
2023-11-21 02:31:18,169 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.49 vs. limit=15.0
2023-11-21 02:31:23,719 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198050
2023-11-21 02:31:25,526 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.82 vs. limit=15.0
2023-11-21 02:31:37,633 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.50 vs. limit=10.0
2023-11-21 02:31:41,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1320380.0, ans=0.125
2023-11-21 02:31:59,784 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5700, loss[loss=0.07505, simple_loss=0.09486, pruned_loss=0.01771, audio_tagging_loss=0.009916, over 15160.00 frames. ], tot_loss[loss=0.07664, simple_loss=0.09822, pruned_loss=0.01758, audio_tagging_loss=0.009956, over 3058710.18 frames. ], batch size: 56, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:32:08,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1320513.3333333333, ans=0.125
2023-11-21 02:32:13,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1320580.0, ans=0.125
2023-11-21 02:32:23,971 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.139e+01 8.085e+01 8.943e+01 9.516e+01 1.355e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-21 02:32:26,614 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198100
2023-11-21 02:32:46,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1320713.3333333333, ans=0.0
2023-11-21 02:33:00,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1320780.0, ans=0.125
2023-11-21 02:33:03,888 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5750, loss[loss=0.07494, simple_loss=0.09689, pruned_loss=0.01637, audio_tagging_loss=0.01013, over 15089.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09892, pruned_loss=0.0177, audio_tagging_loss=0.009753, over 3056525.16 frames. ], batch size: 58, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:33:24,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1320913.3333333333, ans=0.125
2023-11-21 02:33:24,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1320913.3333333333, ans=0.1
2023-11-21 02:33:30,763 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198150
2023-11-21 02:33:40,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1320980.0, ans=0.125
2023-11-21 02:34:06,830 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5800, loss[loss=0.08748, simple_loss=0.109, pruned_loss=0.02074, audio_tagging_loss=0.01225, over 15558.00 frames. ], tot_loss[loss=0.07623, simple_loss=0.09768, pruned_loss=0.01762, audio_tagging_loss=0.009768, over 3054299.16 frames. ], batch size: 56, lr: 4.05e-03, grad_scale: 16.0
2023-11-21 02:34:08,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1321180.0, ans=0.025
2023-11-21 02:34:14,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1321180.0, ans=0.1
2023-11-21 02:34:19,395 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.16 vs. limit=15.0
2023-11-21 02:34:20,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1321246.6666666667, ans=0.125
2023-11-21 02:34:31,409 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.619e+01 8.028e+01 8.750e+01 9.322e+01 1.418e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 02:34:34,760 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198200
2023-11-21 02:34:50,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1321380.0, ans=0.125
2023-11-21 02:34:58,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1321446.6666666667, ans=0.125
2023-11-21 02:35:11,418 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5850, loss[loss=0.06865, simple_loss=0.09205, pruned_loss=0.01553, audio_tagging_loss=0.007096, over 14042.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09731, pruned_loss=0.0176, audio_tagging_loss=0.009731, over 3059906.18 frames. ], batch size: 54, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:35:26,497 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.61 vs. limit=10.0
2023-11-21 02:35:38,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198250
2023-11-21 02:35:38,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1321646.6666666667, ans=0.0
2023-11-21 02:35:41,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.13 vs. limit=15.0
2023-11-21 02:35:48,620 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0
2023-11-21 02:35:58,199 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.46 vs. limit=15.0
2023-11-21 02:36:10,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1321780.0, ans=0.0
2023-11-21 02:36:14,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1321846.6666666667, ans=10.0
2023-11-21 02:36:15,433 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5900, loss[loss=0.06778, simple_loss=0.08899, pruned_loss=0.01393, audio_tagging_loss=0.009358, over 13616.00 frames. ], tot_loss[loss=0.07521, simple_loss=0.09649, pruned_loss=0.01725, audio_tagging_loss=0.009721, over 3051717.11 frames. ], batch size: 53, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:36:24,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1321846.6666666667, ans=0.0
2023-11-21 02:36:29,328 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.71 vs. limit=15.0
2023-11-21 02:36:39,030 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.990e+01 8.061e+01 8.738e+01 9.398e+01 2.090e+02, threshold=1.748e+02, percent-clipped=1.0
2023-11-21 02:36:41,049 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.68 vs. limit=12.0
2023-11-21 02:36:41,581 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198300
2023-11-21 02:37:02,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1322046.6666666667, ans=0.125
2023-11-21 02:37:18,298 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 5950, loss[loss=0.06053, simple_loss=0.06775, pruned_loss=0.01221, audio_tagging_loss=0.01445, over 14044.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.0966, pruned_loss=0.01719, audio_tagging_loss=0.009572, over 3052985.72 frames. ], batch size: 55, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:37:30,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1322246.6666666667, ans=0.1
2023-11-21 02:37:45,637 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198350
2023-11-21 02:37:50,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1322313.3333333333, ans=0.0
2023-11-21 02:38:03,577 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.58 vs. limit=6.0
2023-11-21 02:38:12,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1322446.6666666667, ans=0.2
2023-11-21 02:38:22,699 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6000, loss[loss=0.08143, simple_loss=0.1018, pruned_loss=0.02115, audio_tagging_loss=0.009369, over 16092.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09685, pruned_loss=0.01724, audio_tagging_loss=0.00954, over 3051447.19 frames. ], batch size: 60, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:38:22,702 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 02:38:45,945 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.1402, 3.8042, 4.1099, 3.5344, 4.0526, 4.0065, 3.8020, 3.7561],
       device='cuda:0')
2023-11-21 02:38:47,328 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.5552, 2.9989, 3.3862, 3.1333], device='cuda:0')
2023-11-21 02:39:01,760 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.7594, 4.0329, 3.6271, 3.0645], device='cuda:0')
2023-11-21 02:39:04,193 INFO [train_asr.py:1253] (0/4) Epoch 17, validation: loss=0.06056, simple_loss=0.05273, pruned_loss=0.005281, audio_tagging_loss=0.02892, over 4681554.00 frames. 
2023-11-21 02:39:04,194 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 02:39:05,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1322513.3333333333, ans=0.125
2023-11-21 02:39:06,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1322513.3333333333, ans=0.0
2023-11-21 02:39:28,369 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.213e+01 8.088e+01 8.691e+01 9.569e+01 1.376e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 02:39:31,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198400
2023-11-21 02:39:50,594 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 02:40:08,191 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6050, loss[loss=0.077, simple_loss=0.1088, pruned_loss=0.01658, audio_tagging_loss=0.006006, over 15632.00 frames. ], tot_loss[loss=0.07567, simple_loss=0.09757, pruned_loss=0.01741, audio_tagging_loss=0.009478, over 3054251.49 frames. ], batch size: 58, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:40:12,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1322846.6666666667, ans=0.125
2023-11-21 02:40:16,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1322846.6666666667, ans=0.125
2023-11-21 02:40:18,308 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.89 vs. limit=15.0
2023-11-21 02:40:22,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1322913.3333333333, ans=0.125
2023-11-21 02:40:34,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1322980.0, ans=0.0
2023-11-21 02:40:35,361 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198450
2023-11-21 02:40:46,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1323046.6666666667, ans=0.125
2023-11-21 02:41:12,329 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6100, loss[loss=0.06847, simple_loss=0.08787, pruned_loss=0.0144, audio_tagging_loss=0.01014, over 15746.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09769, pruned_loss=0.01753, audio_tagging_loss=0.00944, over 3057244.84 frames. ], batch size: 58, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:41:16,807 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.16 vs. limit=15.0
2023-11-21 02:41:28,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1323246.6666666667, ans=0.0
2023-11-21 02:41:28,259 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.53 vs. limit=15.0
2023-11-21 02:41:33,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1323246.6666666667, ans=0.0
2023-11-21 02:41:36,814 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.811e+01 8.186e+01 8.873e+01 9.701e+01 1.239e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 02:41:39,353 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198500
2023-11-21 02:42:03,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1323446.6666666667, ans=0.125
2023-11-21 02:42:05,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1323446.6666666667, ans=0.125
2023-11-21 02:42:16,029 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6150, loss[loss=0.05509, simple_loss=0.07298, pruned_loss=0.009348, audio_tagging_loss=0.009254, over 14450.00 frames. ], tot_loss[loss=0.0762, simple_loss=0.09802, pruned_loss=0.0177, audio_tagging_loss=0.009492, over 3058386.92 frames. ], batch size: 55, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:42:19,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1323513.3333333333, ans=0.1
2023-11-21 02:42:24,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1323513.3333333333, ans=0.1
2023-11-21 02:42:24,236 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:42:28,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1323580.0, ans=0.2
2023-11-21 02:42:30,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1323580.0, ans=0.125
2023-11-21 02:42:43,486 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198550
2023-11-21 02:42:47,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1323646.6666666667, ans=0.0
2023-11-21 02:42:59,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1323713.3333333333, ans=0.0
2023-11-21 02:43:01,373 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.38 vs. limit=15.0
2023-11-21 02:43:04,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1323713.3333333333, ans=0.0
2023-11-21 02:43:11,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1323780.0, ans=0.0
2023-11-21 02:43:19,591 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6200, loss[loss=0.0789, simple_loss=0.09859, pruned_loss=0.01804, audio_tagging_loss=0.01157, over 15360.00 frames. ], tot_loss[loss=0.07633, simple_loss=0.09814, pruned_loss=0.01772, audio_tagging_loss=0.009545, over 3059394.01 frames. ], batch size: 58, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:43:45,294 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.093e+01 8.158e+01 8.699e+01 9.470e+01 1.213e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 02:43:47,239 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198600
2023-11-21 02:43:59,092 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.78 vs. limit=12.0
2023-11-21 02:44:11,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1324113.3333333333, ans=0.125
2023-11-21 02:44:13,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1324113.3333333333, ans=0.1
2023-11-21 02:44:24,703 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6250, loss[loss=0.06378, simple_loss=0.07824, pruned_loss=0.01496, audio_tagging_loss=0.0097, over 14867.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09715, pruned_loss=0.01778, audio_tagging_loss=0.009712, over 3055009.84 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:44:47,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1324246.6666666667, ans=0.2
2023-11-21 02:44:50,988 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198650
2023-11-21 02:44:57,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1324313.3333333333, ans=0.125
2023-11-21 02:45:10,187 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.00 vs. limit=22.5
2023-11-21 02:45:12,458 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:45:13,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1324380.0, ans=0.1
2023-11-21 02:45:19,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1324446.6666666667, ans=0.125
2023-11-21 02:45:20,025 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.07 vs. limit=10.0
2023-11-21 02:45:28,195 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6300, loss[loss=0.07824, simple_loss=0.09317, pruned_loss=0.01978, audio_tagging_loss=0.01187, over 14996.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09635, pruned_loss=0.01766, audio_tagging_loss=0.009949, over 3053422.88 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:45:34,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1324513.3333333333, ans=0.035
2023-11-21 02:45:35,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1324513.3333333333, ans=0.125
2023-11-21 02:45:39,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1324580.0, ans=0.0
2023-11-21 02:45:49,130 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.56 vs. limit=6.0
2023-11-21 02:45:53,588 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.806e+01 8.207e+01 8.844e+01 9.523e+01 1.211e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 02:45:55,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198700
2023-11-21 02:46:01,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1324646.6666666667, ans=0.125
2023-11-21 02:46:01,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1324646.6666666667, ans=0.0
2023-11-21 02:46:04,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1324646.6666666667, ans=0.125
2023-11-21 02:46:31,994 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6350, loss[loss=0.08786, simple_loss=0.113, pruned_loss=0.0244, audio_tagging_loss=0.006953, over 15235.00 frames. ], tot_loss[loss=0.07557, simple_loss=0.09613, pruned_loss=0.01746, audio_tagging_loss=0.01005, over 3058103.42 frames. ], batch size: 54, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:46:59,713 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198750
2023-11-21 02:47:05,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1324980.0, ans=0.0
2023-11-21 02:47:10,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1325046.6666666667, ans=0.0
2023-11-21 02:47:15,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1325046.6666666667, ans=0.125
2023-11-21 02:47:24,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1325113.3333333333, ans=0.5
2023-11-21 02:47:37,359 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6400, loss[loss=0.0624, simple_loss=0.08147, pruned_loss=0.01134, audio_tagging_loss=0.01032, over 16481.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09672, pruned_loss=0.01758, audio_tagging_loss=0.01012, over 3050476.09 frames. ], batch size: 62, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:48:02,549 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.929e+01 8.215e+01 8.788e+01 9.513e+01 1.963e+02, threshold=1.758e+02, percent-clipped=1.0
2023-11-21 02:48:03,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198800
2023-11-21 02:48:04,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1325313.3333333333, ans=0.125
2023-11-21 02:48:14,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1325380.0, ans=0.1
2023-11-21 02:48:24,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1325380.0, ans=0.2
2023-11-21 02:48:30,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1325446.6666666667, ans=0.0
2023-11-21 02:48:33,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1325446.6666666667, ans=0.125
2023-11-21 02:48:39,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1325446.6666666667, ans=0.1
2023-11-21 02:48:41,908 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6450, loss[loss=0.09578, simple_loss=0.1323, pruned_loss=0.02337, audio_tagging_loss=0.006247, over 15702.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.09691, pruned_loss=0.01759, audio_tagging_loss=0.01009, over 3045571.97 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:48:54,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1325580.0, ans=0.125
2023-11-21 02:48:55,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1325580.0, ans=0.125
2023-11-21 02:49:07,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198850
2023-11-21 02:49:15,972 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.58 vs. limit=15.0
2023-11-21 02:49:22,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1325713.3333333333, ans=0.0
2023-11-21 02:49:25,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1325713.3333333333, ans=0.125
2023-11-21 02:49:45,159 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6500, loss[loss=0.07349, simple_loss=0.09518, pruned_loss=0.01871, audio_tagging_loss=0.007188, over 15263.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09686, pruned_loss=0.01757, audio_tagging_loss=0.009992, over 3041535.52 frames. ], batch size: 57, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:49:48,004 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:49:49,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1325846.6666666667, ans=0.0
2023-11-21 02:50:10,556 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.752e+01 8.009e+01 8.684e+01 9.310e+01 1.221e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 02:50:11,883 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198900
2023-11-21 02:50:35,911 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 02:50:48,733 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6550, loss[loss=0.05631, simple_loss=0.07287, pruned_loss=0.01043, audio_tagging_loss=0.00945, over 15783.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09711, pruned_loss=0.01773, audio_tagging_loss=0.009881, over 3046293.58 frames. ], batch size: 60, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:50:50,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1326180.0, ans=0.0
2023-11-21 02:50:53,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1326180.0, ans=0.0
2023-11-21 02:50:59,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1326180.0, ans=0.0
2023-11-21 02:51:11,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1326246.6666666667, ans=0.125
2023-11-21 02:51:15,810 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 198950
2023-11-21 02:51:15,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1326313.3333333333, ans=0.015
2023-11-21 02:51:20,204 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.80 vs. limit=15.0
2023-11-21 02:51:52,726 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6600, loss[loss=0.07077, simple_loss=0.0865, pruned_loss=0.01634, audio_tagging_loss=0.01118, over 16530.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09732, pruned_loss=0.01774, audio_tagging_loss=0.009745, over 3047660.61 frames. ], batch size: 63, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:52:14,930 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.52 vs. limit=15.0
2023-11-21 02:52:17,923 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.950e+01 8.252e+01 8.813e+01 9.521e+01 1.214e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 02:52:19,341 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199000
2023-11-21 02:52:22,791 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.97 vs. limit=15.0
2023-11-21 02:52:35,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1326713.3333333333, ans=0.0
2023-11-21 02:52:42,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1326713.3333333333, ans=0.5
2023-11-21 02:52:49,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1326780.0, ans=0.0
2023-11-21 02:52:50,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1326780.0, ans=0.09899494936611666
2023-11-21 02:52:51,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1326780.0, ans=0.125
2023-11-21 02:52:57,311 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6650, loss[loss=0.08377, simple_loss=0.1034, pruned_loss=0.01969, audio_tagging_loss=0.01239, over 15590.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.09678, pruned_loss=0.01777, audio_tagging_loss=0.009772, over 3045356.28 frames. ], batch size: 59, lr: 4.04e-03, grad_scale: 32.0
2023-11-21 02:53:25,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199050
2023-11-21 02:53:30,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1326980.0, ans=0.125
2023-11-21 02:53:51,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1327113.3333333333, ans=0.2
2023-11-21 02:53:52,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.61 vs. limit=15.0
2023-11-21 02:53:58,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1327113.3333333333, ans=0.0
2023-11-21 02:54:01,150 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6700, loss[loss=0.06633, simple_loss=0.08946, pruned_loss=0.01367, audio_tagging_loss=0.007922, over 16621.00 frames. ], tot_loss[loss=0.07587, simple_loss=0.09683, pruned_loss=0.0177, audio_tagging_loss=0.009756, over 3044496.61 frames. ], batch size: 60, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:54:17,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1327246.6666666667, ans=0.125
2023-11-21 02:54:23,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1327246.6666666667, ans=0.125
2023-11-21 02:54:23,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1327246.6666666667, ans=0.2
2023-11-21 02:54:24,049 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.47 vs. limit=15.0
2023-11-21 02:54:29,385 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.948e+01 7.825e+01 8.466e+01 9.524e+01 1.131e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-21 02:54:29,538 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199100
2023-11-21 02:54:32,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1327313.3333333333, ans=0.125
2023-11-21 02:54:38,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1327313.3333333333, ans=0.125
2023-11-21 02:55:06,867 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6750, loss[loss=0.06098, simple_loss=0.07517, pruned_loss=0.01335, audio_tagging_loss=0.01004, over 13487.00 frames. ], tot_loss[loss=0.07586, simple_loss=0.09673, pruned_loss=0.01775, audio_tagging_loss=0.009749, over 3039844.58 frames. ], batch size: 52, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:55:11,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1327513.3333333333, ans=0.125
2023-11-21 02:55:32,929 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199150
2023-11-21 02:55:49,507 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.90 vs. limit=15.0
2023-11-21 02:56:04,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1327780.0, ans=0.2
2023-11-21 02:56:10,696 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6800, loss[loss=0.09969, simple_loss=0.1321, pruned_loss=0.02662, audio_tagging_loss=0.007024, over 15648.00 frames. ], tot_loss[loss=0.07621, simple_loss=0.09752, pruned_loss=0.01781, audio_tagging_loss=0.009645, over 3050651.05 frames. ], batch size: 56, lr: 4.04e-03, grad_scale: 16.0
2023-11-21 02:56:11,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1327846.6666666667, ans=0.2
2023-11-21 02:56:37,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199200
2023-11-21 02:56:38,913 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.071e+01 8.833e+01 9.608e+01 1.176e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 02:57:00,660 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.22 vs. limit=10.0
2023-11-21 02:57:09,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1328113.3333333333, ans=0.125
2023-11-21 02:57:14,635 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6850, loss[loss=0.08524, simple_loss=0.1159, pruned_loss=0.0203, audio_tagging_loss=0.006986, over 16187.00 frames. ], tot_loss[loss=0.07595, simple_loss=0.09734, pruned_loss=0.0176, audio_tagging_loss=0.009685, over 3052294.70 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 02:57:35,302 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.22 vs. limit=15.0
2023-11-21 02:57:37,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1328246.6666666667, ans=0.125
2023-11-21 02:57:39,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.94 vs. limit=22.5
2023-11-21 02:57:42,696 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199250
2023-11-21 02:57:48,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1328313.3333333333, ans=0.0
2023-11-21 02:57:49,314 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.32 vs. limit=15.0
2023-11-21 02:58:13,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1328446.6666666667, ans=0.5
2023-11-21 02:58:19,394 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6900, loss[loss=0.0741, simple_loss=0.09654, pruned_loss=0.0151, audio_tagging_loss=0.01074, over 15618.00 frames. ], tot_loss[loss=0.07586, simple_loss=0.09751, pruned_loss=0.0175, audio_tagging_loss=0.0096, over 3047509.38 frames. ], batch size: 60, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 02:58:22,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1328513.3333333333, ans=0.0
2023-11-21 02:58:46,585 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199300
2023-11-21 02:58:47,623 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 7.846e+01 8.588e+01 9.245e+01 1.215e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-21 02:58:54,537 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.51 vs. limit=6.0
2023-11-21 02:59:03,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1328713.3333333333, ans=0.125
2023-11-21 02:59:08,606 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 02:59:12,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.60 vs. limit=22.5
2023-11-21 02:59:13,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.54 vs. limit=15.0
2023-11-21 02:59:23,850 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 6950, loss[loss=0.0803, simple_loss=0.09987, pruned_loss=0.02211, audio_tagging_loss=0.008247, over 15267.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.09767, pruned_loss=0.01746, audio_tagging_loss=0.009633, over 3046079.47 frames. ], batch size: 60, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 02:59:30,904 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.28 vs. limit=15.0
2023-11-21 02:59:31,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1328846.6666666667, ans=0.125
2023-11-21 02:59:35,470 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.065e-02
2023-11-21 02:59:39,409 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.17 vs. limit=22.5
2023-11-21 02:59:51,130 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199350
2023-11-21 02:59:54,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1328980.0, ans=0.2
2023-11-21 03:00:02,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1329046.6666666667, ans=0.0
2023-11-21 03:00:06,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1329046.6666666667, ans=0.1
2023-11-21 03:00:17,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1329113.3333333333, ans=0.125
2023-11-21 03:00:18,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1329113.3333333333, ans=0.0
2023-11-21 03:00:27,835 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7000, loss[loss=0.1049, simple_loss=0.1435, pruned_loss=0.02574, audio_tagging_loss=0.007374, over 14363.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09755, pruned_loss=0.01761, audio_tagging_loss=0.00965, over 3040968.55 frames. ], batch size: 52, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:00:35,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1329180.0, ans=0.125
2023-11-21 03:00:44,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1329246.6666666667, ans=0.1
2023-11-21 03:00:53,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1329313.3333333333, ans=0.0
2023-11-21 03:00:55,574 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199400
2023-11-21 03:00:57,291 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.260e+01 8.901e+01 9.434e+01 1.193e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 03:01:21,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1329446.6666666667, ans=0.0
2023-11-21 03:01:30,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1329446.6666666667, ans=0.5
2023-11-21 03:01:32,980 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7050, loss[loss=0.07135, simple_loss=0.09226, pruned_loss=0.01493, audio_tagging_loss=0.0103, over 14281.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09697, pruned_loss=0.01744, audio_tagging_loss=0.009731, over 3043148.18 frames. ], batch size: 53, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:01:33,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1329513.3333333333, ans=0.125
2023-11-21 03:01:39,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1329513.3333333333, ans=0.1
2023-11-21 03:01:45,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1329580.0, ans=0.015
2023-11-21 03:01:50,881 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.05 vs. limit=15.0
2023-11-21 03:01:55,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1329580.0, ans=0.125
2023-11-21 03:02:00,157 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199450
2023-11-21 03:02:19,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1329713.3333333333, ans=0.0
2023-11-21 03:02:38,008 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7100, loss[loss=0.07285, simple_loss=0.08596, pruned_loss=0.01852, audio_tagging_loss=0.01135, over 15758.00 frames. ], tot_loss[loss=0.07577, simple_loss=0.09721, pruned_loss=0.01735, audio_tagging_loss=0.009811, over 3050227.37 frames. ], batch size: 60, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:02:39,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1329846.6666666667, ans=0.09899494936611666
2023-11-21 03:02:40,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1329846.6666666667, ans=0.125
2023-11-21 03:02:42,231 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:02:49,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1329913.3333333333, ans=15.0
2023-11-21 03:02:57,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1329913.3333333333, ans=0.0
2023-11-21 03:03:05,039 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199500
2023-11-21 03:03:06,068 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 7.897e+01 8.673e+01 9.642e+01 1.180e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 03:03:23,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1330046.6666666667, ans=0.2
2023-11-21 03:03:25,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1330046.6666666667, ans=0.0
2023-11-21 03:03:33,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1330113.3333333333, ans=0.125
2023-11-21 03:03:38,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1330113.3333333333, ans=0.1
2023-11-21 03:03:39,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1330113.3333333333, ans=0.09899494936611666
2023-11-21 03:03:42,084 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7150, loss[loss=0.09432, simple_loss=0.1215, pruned_loss=0.02338, audio_tagging_loss=0.01021, over 15326.00 frames. ], tot_loss[loss=0.07601, simple_loss=0.09781, pruned_loss=0.01737, audio_tagging_loss=0.009728, over 3045535.60 frames. ], batch size: 57, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:03:48,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.27 vs. limit=22.5
2023-11-21 03:03:54,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1330246.6666666667, ans=0.125
2023-11-21 03:04:09,801 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199550
2023-11-21 03:04:31,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1330380.0, ans=0.1
2023-11-21 03:04:38,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1330446.6666666667, ans=0.125
2023-11-21 03:04:46,665 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7200, loss[loss=0.0933, simple_loss=0.1238, pruned_loss=0.01999, audio_tagging_loss=0.01139, over 15939.00 frames. ], tot_loss[loss=0.07602, simple_loss=0.09774, pruned_loss=0.01735, audio_tagging_loss=0.009797, over 3043200.82 frames. ], batch size: 57, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:05:00,957 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:05:01,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1330580.0, ans=0.125
2023-11-21 03:05:09,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1330580.0, ans=0.125
2023-11-21 03:05:13,785 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199600
2023-11-21 03:05:16,288 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 7.951e+01 8.767e+01 9.502e+01 1.229e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 03:05:16,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1330646.6666666667, ans=10.0
2023-11-21 03:05:19,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1330646.6666666667, ans=0.05
2023-11-21 03:05:25,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1330713.3333333333, ans=0.125
2023-11-21 03:05:32,757 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.11 vs. limit=15.0
2023-11-21 03:05:37,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.29 vs. limit=10.0
2023-11-21 03:05:38,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1330780.0, ans=0.0
2023-11-21 03:05:44,925 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=22.23 vs. limit=22.5
2023-11-21 03:05:50,867 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7250, loss[loss=0.07002, simple_loss=0.08197, pruned_loss=0.01613, audio_tagging_loss=0.01291, over 15148.00 frames. ], tot_loss[loss=0.07605, simple_loss=0.09745, pruned_loss=0.01735, audio_tagging_loss=0.00998, over 3041107.10 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:06:17,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199650
2023-11-21 03:06:22,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1330980.0, ans=0.0
2023-11-21 03:06:53,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1331180.0, ans=0.125
2023-11-21 03:06:54,326 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7300, loss[loss=0.07829, simple_loss=0.1111, pruned_loss=0.01491, audio_tagging_loss=0.007828, over 14895.00 frames. ], tot_loss[loss=0.07567, simple_loss=0.09697, pruned_loss=0.01729, audio_tagging_loss=0.009894, over 3042275.07 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:07:16,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1331246.6666666667, ans=0.0
2023-11-21 03:07:21,002 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199700
2023-11-21 03:07:22,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1331313.3333333333, ans=0.125
2023-11-21 03:07:23,853 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.086e+01 8.011e+01 8.733e+01 9.607e+01 1.282e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 03:07:43,135 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.79 vs. limit=12.0
2023-11-21 03:07:58,470 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7350, loss[loss=0.07563, simple_loss=0.09437, pruned_loss=0.02064, audio_tagging_loss=0.007806, over 14348.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09663, pruned_loss=0.01734, audio_tagging_loss=0.009802, over 3041015.33 frames. ], batch size: 54, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:08:10,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1331580.0, ans=0.125
2023-11-21 03:08:18,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1331580.0, ans=0.1
2023-11-21 03:08:24,795 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199750
2023-11-21 03:08:24,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1331646.6666666667, ans=0.07
2023-11-21 03:08:27,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1331646.6666666667, ans=0.0
2023-11-21 03:08:41,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1331713.3333333333, ans=0.2
2023-11-21 03:08:46,492 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.49 vs. limit=22.5
2023-11-21 03:08:50,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1331780.0, ans=0.0
2023-11-21 03:09:02,769 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7400, loss[loss=0.08071, simple_loss=0.09606, pruned_loss=0.02202, audio_tagging_loss=0.01067, over 14382.00 frames. ], tot_loss[loss=0.07672, simple_loss=0.09875, pruned_loss=0.01773, audio_tagging_loss=0.009613, over 3046805.65 frames. ], batch size: 55, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:09:27,673 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.29 vs. limit=15.0
2023-11-21 03:09:28,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1331980.0, ans=0.125
2023-11-21 03:09:29,542 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199800
2023-11-21 03:09:32,471 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.394e+01 8.010e+01 8.731e+01 9.621e+01 1.537e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 03:09:41,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1332046.6666666667, ans=0.125
2023-11-21 03:09:43,579 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.68 vs. limit=15.0
2023-11-21 03:09:54,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1332113.3333333333, ans=0.035
2023-11-21 03:09:59,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1332113.3333333333, ans=0.0
2023-11-21 03:10:06,602 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7450, loss[loss=0.08484, simple_loss=0.1098, pruned_loss=0.02296, audio_tagging_loss=0.006968, over 15636.00 frames. ], tot_loss[loss=0.0767, simple_loss=0.09895, pruned_loss=0.01774, audio_tagging_loss=0.009488, over 3048362.49 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:10:19,797 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.65 vs. limit=10.0
2023-11-21 03:10:24,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1332246.6666666667, ans=0.125
2023-11-21 03:10:32,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1332313.3333333333, ans=0.1
2023-11-21 03:10:33,744 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199850
2023-11-21 03:10:44,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1332380.0, ans=0.1
2023-11-21 03:10:52,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1332380.0, ans=0.2
2023-11-21 03:11:06,850 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=12.89 vs. limit=15.0
2023-11-21 03:11:10,952 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7500, loss[loss=0.07138, simple_loss=0.08933, pruned_loss=0.01859, audio_tagging_loss=0.008118, over 15550.00 frames. ], tot_loss[loss=0.07551, simple_loss=0.09702, pruned_loss=0.01751, audio_tagging_loss=0.009485, over 3044429.46 frames. ], batch size: 59, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:11:13,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1332513.3333333333, ans=0.2
2023-11-21 03:11:25,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1332580.0, ans=0.05
2023-11-21 03:11:37,147 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199900
2023-11-21 03:11:38,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1332646.6666666667, ans=0.125
2023-11-21 03:11:39,368 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.979e+01 8.350e+01 8.922e+01 9.540e+01 1.272e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-21 03:11:46,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1332713.3333333333, ans=0.125
2023-11-21 03:11:47,155 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.37 vs. limit=15.0
2023-11-21 03:11:55,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1332713.3333333333, ans=0.125
2023-11-21 03:12:01,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1332780.0, ans=0.0
2023-11-21 03:12:14,028 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7550, loss[loss=0.05931, simple_loss=0.08615, pruned_loss=0.007875, audio_tagging_loss=0.008367, over 15920.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09673, pruned_loss=0.01752, audio_tagging_loss=0.009489, over 3042186.65 frames. ], batch size: 58, lr: 4.03e-03, grad_scale: 16.0
2023-11-21 03:12:19,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1332846.6666666667, ans=0.2
2023-11-21 03:12:40,947 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 199950
2023-11-21 03:12:46,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1332980.0, ans=0.1
2023-11-21 03:12:46,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.26 vs. limit=15.0
2023-11-21 03:12:57,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1333046.6666666667, ans=0.125
2023-11-21 03:13:13,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1333113.3333333333, ans=0.125
2023-11-21 03:13:14,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1333113.3333333333, ans=0.0
2023-11-21 03:13:17,623 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7600, loss[loss=0.08003, simple_loss=0.09766, pruned_loss=0.01881, audio_tagging_loss=0.01238, over 13828.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.09691, pruned_loss=0.01753, audio_tagging_loss=0.009558, over 3047399.59 frames. ], batch size: 53, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:13:29,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1333246.6666666667, ans=0.07
2023-11-21 03:13:42,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1333313.3333333333, ans=0.0
2023-11-21 03:13:42,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1333313.3333333333, ans=0.125
2023-11-21 03:13:44,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1333313.3333333333, ans=0.125
2023-11-21 03:13:45,190 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200000
2023-11-21 03:13:46,713 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-200000.pt
2023-11-21 03:13:50,908 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.857e+01 8.060e+01 8.807e+01 9.387e+01 1.277e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 03:13:53,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1333313.3333333333, ans=0.0
2023-11-21 03:13:56,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1333313.3333333333, ans=0.1
2023-11-21 03:14:25,292 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7650, loss[loss=0.07885, simple_loss=0.1022, pruned_loss=0.0194, audio_tagging_loss=0.008347, over 14855.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09637, pruned_loss=0.01733, audio_tagging_loss=0.009584, over 3051431.81 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:14:50,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1333646.6666666667, ans=0.125
2023-11-21 03:14:52,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200050
2023-11-21 03:15:08,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1333713.3333333333, ans=0.0
2023-11-21 03:15:09,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1333713.3333333333, ans=0.0
2023-11-21 03:15:12,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1333713.3333333333, ans=0.0
2023-11-21 03:15:29,949 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7700, loss[loss=0.07864, simple_loss=0.1035, pruned_loss=0.0187, audio_tagging_loss=0.008175, over 14129.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09777, pruned_loss=0.01766, audio_tagging_loss=0.009484, over 3044233.59 frames. ], batch size: 56, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:15:34,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.97 vs. limit=10.0
2023-11-21 03:15:55,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200100
2023-11-21 03:15:58,171 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.520e+01 8.098e+01 8.698e+01 9.708e+01 1.361e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 03:16:07,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1334046.6666666667, ans=0.2
2023-11-21 03:16:11,134 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.63 vs. limit=15.0
2023-11-21 03:16:15,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1334046.6666666667, ans=0.125
2023-11-21 03:16:22,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.48 vs. limit=15.0
2023-11-21 03:16:23,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1334113.3333333333, ans=0.125
2023-11-21 03:16:26,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1334113.3333333333, ans=0.125
2023-11-21 03:16:33,107 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7750, loss[loss=0.1071, simple_loss=0.1424, pruned_loss=0.02604, audio_tagging_loss=0.009873, over 15959.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09756, pruned_loss=0.01769, audio_tagging_loss=0.009623, over 3040513.89 frames. ], batch size: 60, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:16:54,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1334246.6666666667, ans=0.125
2023-11-21 03:17:00,613 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200150
2023-11-21 03:17:01,218 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.11 vs. limit=10.0
2023-11-21 03:17:08,495 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0
2023-11-21 03:17:16,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1334380.0, ans=0.1
2023-11-21 03:17:36,758 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7800, loss[loss=0.09871, simple_loss=0.1204, pruned_loss=0.02958, audio_tagging_loss=0.008956, over 14578.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09858, pruned_loss=0.01801, audio_tagging_loss=0.009708, over 3038803.34 frames. ], batch size: 55, lr: 4.03e-03, grad_scale: 32.0
2023-11-21 03:17:47,817 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.33 vs. limit=22.5
2023-11-21 03:18:04,180 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.04 vs. limit=22.5
2023-11-21 03:18:04,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200200
2023-11-21 03:18:07,466 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.821e+01 7.982e+01 8.620e+01 9.218e+01 1.876e+02, threshold=1.724e+02, percent-clipped=1.0
2023-11-21 03:18:16,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1334713.3333333333, ans=0.025
2023-11-21 03:18:18,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1334713.3333333333, ans=0.04949747468305833
2023-11-21 03:18:42,254 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7850, loss[loss=0.09161, simple_loss=0.1236, pruned_loss=0.02514, audio_tagging_loss=0.004695, over 14343.00 frames. ], tot_loss[loss=0.07673, simple_loss=0.09811, pruned_loss=0.01791, audio_tagging_loss=0.009764, over 3044433.36 frames. ], batch size: 53, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:18:47,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1334846.6666666667, ans=0.0
2023-11-21 03:19:05,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1334913.3333333333, ans=0.2
2023-11-21 03:19:08,031 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=7.98 vs. limit=12.0
2023-11-21 03:19:08,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200250
2023-11-21 03:19:12,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1334980.0, ans=0.125
2023-11-21 03:19:39,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1335113.3333333333, ans=0.2
2023-11-21 03:19:43,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1335113.3333333333, ans=0.125
2023-11-21 03:19:46,659 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7900, loss[loss=0.08702, simple_loss=0.1163, pruned_loss=0.02084, audio_tagging_loss=0.008023, over 14430.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.09807, pruned_loss=0.01789, audio_tagging_loss=0.009849, over 3045062.57 frames. ], batch size: 53, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:20:13,767 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200300
2023-11-21 03:20:16,601 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.124e+01 8.233e+01 9.036e+01 9.766e+01 1.372e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 03:20:34,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1335380.0, ans=0.2
2023-11-21 03:20:43,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1335446.6666666667, ans=0.125
2023-11-21 03:20:49,717 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 7950, loss[loss=0.06056, simple_loss=0.07346, pruned_loss=0.01172, audio_tagging_loss=0.01211, over 14753.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.097, pruned_loss=0.01765, audio_tagging_loss=0.01001, over 3049977.21 frames. ], batch size: 57, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:21:05,450 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 03:21:14,932 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.30 vs. limit=15.0
2023-11-21 03:21:17,886 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200350
2023-11-21 03:21:41,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1335780.0, ans=0.0
2023-11-21 03:21:54,638 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8000, loss[loss=0.06028, simple_loss=0.07884, pruned_loss=0.01213, audio_tagging_loss=0.00873, over 14846.00 frames. ], tot_loss[loss=0.07626, simple_loss=0.09701, pruned_loss=0.01771, audio_tagging_loss=0.01005, over 3045722.58 frames. ], batch size: 58, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:22:05,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1335846.6666666667, ans=0.125
2023-11-21 03:22:09,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1335913.3333333333, ans=0.0
2023-11-21 03:22:11,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1335913.3333333333, ans=0.0
2023-11-21 03:22:21,541 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200400
2023-11-21 03:22:24,185 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.612e+01 7.996e+01 8.659e+01 9.313e+01 1.449e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 03:22:42,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1336046.6666666667, ans=0.04949747468305833
2023-11-21 03:22:59,644 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8050, loss[loss=0.06085, simple_loss=0.07613, pruned_loss=0.01231, audio_tagging_loss=0.01047, over 15141.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.097, pruned_loss=0.01766, audio_tagging_loss=0.01003, over 3044130.63 frames. ], batch size: 57, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:23:05,145 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.84 vs. limit=10.0
2023-11-21 03:23:26,474 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200450
2023-11-21 03:23:32,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1336313.3333333333, ans=0.0
2023-11-21 03:23:51,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1336446.6666666667, ans=0.2
2023-11-21 03:23:53,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1336446.6666666667, ans=0.125
2023-11-21 03:24:01,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1336513.3333333333, ans=0.1
2023-11-21 03:24:02,442 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8100, loss[loss=0.0752, simple_loss=0.1041, pruned_loss=0.01627, audio_tagging_loss=0.006898, over 15263.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09732, pruned_loss=0.01779, audio_tagging_loss=0.009947, over 3037626.47 frames. ], batch size: 55, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:24:05,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1336513.3333333333, ans=0.1
2023-11-21 03:24:06,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1336513.3333333333, ans=0.0
2023-11-21 03:24:06,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1336513.3333333333, ans=0.125
2023-11-21 03:24:06,761 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.31 vs. limit=6.0
2023-11-21 03:24:09,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1336513.3333333333, ans=0.1
2023-11-21 03:24:29,782 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200500
2023-11-21 03:24:32,117 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.373e+01 8.135e+01 8.794e+01 9.348e+01 1.385e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 03:24:34,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1336646.6666666667, ans=0.2
2023-11-21 03:25:06,602 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8150, loss[loss=0.06802, simple_loss=0.08517, pruned_loss=0.01634, audio_tagging_loss=0.009098, over 13739.00 frames. ], tot_loss[loss=0.07639, simple_loss=0.09731, pruned_loss=0.0179, audio_tagging_loss=0.009831, over 3041402.47 frames. ], batch size: 54, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:25:07,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1336846.6666666667, ans=0.125
2023-11-21 03:25:18,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1336913.3333333333, ans=0.125
2023-11-21 03:25:33,585 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200550
2023-11-21 03:25:37,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1336980.0, ans=10.0
2023-11-21 03:26:11,141 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8200, loss[loss=0.08157, simple_loss=0.1031, pruned_loss=0.0188, audio_tagging_loss=0.01124, over 14705.00 frames. ], tot_loss[loss=0.07726, simple_loss=0.09873, pruned_loss=0.0182, audio_tagging_loss=0.009701, over 3053903.17 frames. ], batch size: 55, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:26:11,181 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 03:26:11,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1337180.0, ans=0.125
2023-11-21 03:26:20,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1337180.0, ans=0.125
2023-11-21 03:26:37,478 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200600
2023-11-21 03:26:40,661 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.735e+01 8.383e+01 9.269e+01 1.042e+02 1.722e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-21 03:27:15,023 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8250, loss[loss=0.08292, simple_loss=0.112, pruned_loss=0.0209, audio_tagging_loss=0.00604, over 15420.00 frames. ], tot_loss[loss=0.07718, simple_loss=0.09879, pruned_loss=0.01823, audio_tagging_loss=0.009548, over 3053165.62 frames. ], batch size: 58, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:27:15,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1337513.3333333333, ans=0.125
2023-11-21 03:27:26,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1337580.0, ans=0.125
2023-11-21 03:27:41,836 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200650
2023-11-21 03:27:42,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1337646.6666666667, ans=0.125
2023-11-21 03:27:50,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1337646.6666666667, ans=0.0
2023-11-21 03:27:54,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1337713.3333333333, ans=0.125
2023-11-21 03:28:04,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1337713.3333333333, ans=0.1
2023-11-21 03:28:09,440 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.26 vs. limit=15.0
2023-11-21 03:28:19,779 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8300, loss[loss=0.07946, simple_loss=0.1085, pruned_loss=0.01514, audio_tagging_loss=0.01006, over 15695.00 frames. ], tot_loss[loss=0.07664, simple_loss=0.09813, pruned_loss=0.01799, audio_tagging_loss=0.009582, over 3051658.24 frames. ], batch size: 61, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:28:28,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1337846.6666666667, ans=0.0
2023-11-21 03:28:46,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200700
2023-11-21 03:28:50,272 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.519e+01 8.192e+01 8.783e+01 9.537e+01 1.149e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 03:29:01,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.88 vs. limit=15.0
2023-11-21 03:29:07,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1338046.6666666667, ans=0.0
2023-11-21 03:29:13,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1338113.3333333333, ans=0.2
2023-11-21 03:29:15,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1338113.3333333333, ans=0.0
2023-11-21 03:29:20,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1338113.3333333333, ans=0.125
2023-11-21 03:29:23,824 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8350, loss[loss=0.07779, simple_loss=0.0922, pruned_loss=0.01996, audio_tagging_loss=0.01173, over 13784.00 frames. ], tot_loss[loss=0.07645, simple_loss=0.09768, pruned_loss=0.01797, audio_tagging_loss=0.009634, over 3049529.05 frames. ], batch size: 52, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:29:50,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200750
2023-11-21 03:30:20,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1338446.6666666667, ans=0.125
2023-11-21 03:30:27,688 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8400, loss[loss=0.08394, simple_loss=0.109, pruned_loss=0.02155, audio_tagging_loss=0.007917, over 15204.00 frames. ], tot_loss[loss=0.07618, simple_loss=0.09744, pruned_loss=0.01787, audio_tagging_loss=0.009591, over 3046306.60 frames. ], batch size: 55, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:30:31,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1338513.3333333333, ans=0.1
2023-11-21 03:30:54,122 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200800
2023-11-21 03:30:55,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1338646.6666666667, ans=0.125
2023-11-21 03:30:57,992 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.656e+01 8.229e+01 8.784e+01 9.362e+01 1.197e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 03:31:18,224 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.66 vs. limit=15.0
2023-11-21 03:31:30,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1338846.6666666667, ans=0.1
2023-11-21 03:31:31,834 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8450, loss[loss=0.06067, simple_loss=0.08309, pruned_loss=0.008515, audio_tagging_loss=0.0106, over 16099.00 frames. ], tot_loss[loss=0.07579, simple_loss=0.09689, pruned_loss=0.01769, audio_tagging_loss=0.009652, over 3039688.97 frames. ], batch size: 62, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:31:40,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1338846.6666666667, ans=0.0
2023-11-21 03:31:58,166 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200850
2023-11-21 03:32:00,656 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.04 vs. limit=10.0
2023-11-21 03:32:01,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1338980.0, ans=0.125
2023-11-21 03:32:26,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1339113.3333333333, ans=0.125
2023-11-21 03:32:30,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1339113.3333333333, ans=0.125
2023-11-21 03:32:35,649 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8500, loss[loss=0.07387, simple_loss=0.09235, pruned_loss=0.01925, audio_tagging_loss=0.008449, over 15937.00 frames. ], tot_loss[loss=0.07597, simple_loss=0.09703, pruned_loss=0.01778, audio_tagging_loss=0.009681, over 3044861.64 frames. ], batch size: 61, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:32:45,490 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.55 vs. limit=15.0
2023-11-21 03:32:46,380 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:32:53,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1339246.6666666667, ans=0.125
2023-11-21 03:33:02,936 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200900
2023-11-21 03:33:06,650 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 7.935e+01 8.661e+01 9.459e+01 1.270e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 03:33:18,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1339380.0, ans=0.125
2023-11-21 03:33:35,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1339446.6666666667, ans=0.125
2023-11-21 03:33:39,578 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8550, loss[loss=0.1021, simple_loss=0.1267, pruned_loss=0.02502, audio_tagging_loss=0.01373, over 16176.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09778, pruned_loss=0.01789, audio_tagging_loss=0.009735, over 3047448.56 frames. ], batch size: 61, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:33:49,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1339513.3333333333, ans=15.0
2023-11-21 03:34:05,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1339646.6666666667, ans=0.125
2023-11-21 03:34:06,582 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 200950
2023-11-21 03:34:23,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1339713.3333333333, ans=0.125
2023-11-21 03:34:33,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1339780.0, ans=0.0
2023-11-21 03:34:43,518 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8600, loss[loss=0.07183, simple_loss=0.09602, pruned_loss=0.01494, audio_tagging_loss=0.008879, over 15565.00 frames. ], tot_loss[loss=0.07648, simple_loss=0.09791, pruned_loss=0.01781, audio_tagging_loss=0.009717, over 3048327.96 frames. ], batch size: 56, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:34:51,129 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.28 vs. limit=15.0
2023-11-21 03:35:03,848 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.54 vs. limit=5.0
2023-11-21 03:35:10,387 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201000
2023-11-21 03:35:10,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1339980.0, ans=0.2
2023-11-21 03:35:14,246 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.482e+01 7.981e+01 8.707e+01 9.381e+01 1.149e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-21 03:35:31,551 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.30 vs. limit=15.0
2023-11-21 03:35:35,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1340113.3333333333, ans=0.0
2023-11-21 03:35:47,420 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8650, loss[loss=0.05964, simple_loss=0.07105, pruned_loss=0.01293, audio_tagging_loss=0.01118, over 14508.00 frames. ], tot_loss[loss=0.07633, simple_loss=0.09767, pruned_loss=0.01772, audio_tagging_loss=0.009776, over 3050405.82 frames. ], batch size: 57, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:36:01,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1340246.6666666667, ans=0.1
2023-11-21 03:36:06,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1340246.6666666667, ans=0.125
2023-11-21 03:36:14,573 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201050
2023-11-21 03:36:18,698 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.72 vs. limit=12.0
2023-11-21 03:36:37,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1340446.6666666667, ans=0.0
2023-11-21 03:36:42,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1340446.6666666667, ans=0.125
2023-11-21 03:36:50,948 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8700, loss[loss=0.0945, simple_loss=0.1176, pruned_loss=0.02674, audio_tagging_loss=0.00895, over 16324.00 frames. ], tot_loss[loss=0.07698, simple_loss=0.09875, pruned_loss=0.01789, audio_tagging_loss=0.009724, over 3052041.62 frames. ], batch size: 61, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:37:02,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1340580.0, ans=0.125
2023-11-21 03:37:06,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1340580.0, ans=0.125
2023-11-21 03:37:14,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1340580.0, ans=0.125
2023-11-21 03:37:17,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201100
2023-11-21 03:37:22,702 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.048e+01 8.240e+01 8.992e+01 1.008e+02 1.229e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-21 03:37:30,093 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.70 vs. limit=15.0
2023-11-21 03:37:39,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1340713.3333333333, ans=0.125
2023-11-21 03:37:49,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1340780.0, ans=0.1
2023-11-21 03:37:54,266 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8750, loss[loss=0.08848, simple_loss=0.1068, pruned_loss=0.02569, audio_tagging_loss=0.009406, over 16195.00 frames. ], tot_loss[loss=0.07779, simple_loss=0.09974, pruned_loss=0.01818, audio_tagging_loss=0.009738, over 3042861.63 frames. ], batch size: 59, lr: 4.02e-03, grad_scale: 16.0
2023-11-21 03:38:08,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1340913.3333333333, ans=0.0
2023-11-21 03:38:21,161 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201150
2023-11-21 03:38:22,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1340980.0, ans=0.125
2023-11-21 03:38:31,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1341046.6666666667, ans=0.125
2023-11-21 03:38:58,555 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8800, loss[loss=0.08971, simple_loss=0.1167, pruned_loss=0.02341, audio_tagging_loss=0.007974, over 15106.00 frames. ], tot_loss[loss=0.07775, simple_loss=0.09953, pruned_loss=0.01814, audio_tagging_loss=0.00985, over 3047051.64 frames. ], batch size: 56, lr: 4.02e-03, grad_scale: 32.0
2023-11-21 03:38:58,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1341180.0, ans=0.125
2023-11-21 03:39:04,400 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.52 vs. limit=22.5
2023-11-21 03:39:16,912 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.35 vs. limit=15.0
2023-11-21 03:39:24,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201200
2023-11-21 03:39:27,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1341313.3333333333, ans=0.125
2023-11-21 03:39:30,382 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.246e+01 8.266e+01 8.872e+01 9.572e+01 1.229e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 03:39:46,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1341380.0, ans=0.125
2023-11-21 03:40:02,356 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8850, loss[loss=0.09425, simple_loss=0.1216, pruned_loss=0.02297, audio_tagging_loss=0.0105, over 15269.00 frames. ], tot_loss[loss=0.07818, simple_loss=0.1001, pruned_loss=0.01827, audio_tagging_loss=0.009834, over 3048746.13 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:40:13,277 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 03:40:21,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1341580.0, ans=0.0
2023-11-21 03:40:29,554 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201250
2023-11-21 03:41:05,637 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8900, loss[loss=0.08348, simple_loss=0.1009, pruned_loss=0.02222, audio_tagging_loss=0.0108, over 14715.00 frames. ], tot_loss[loss=0.07798, simple_loss=0.09985, pruned_loss=0.01831, audio_tagging_loss=0.009741, over 3046271.66 frames. ], batch size: 54, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:41:26,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1341913.3333333333, ans=0.1
2023-11-21 03:41:33,481 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201300
2023-11-21 03:41:38,118 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.632e+01 8.120e+01 8.967e+01 9.567e+01 1.214e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 03:41:40,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1341980.0, ans=0.125
2023-11-21 03:41:52,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1342046.6666666667, ans=0.125
2023-11-21 03:42:07,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1342113.3333333333, ans=0.04949747468305833
2023-11-21 03:42:11,254 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 8950, loss[loss=0.09019, simple_loss=0.1217, pruned_loss=0.02122, audio_tagging_loss=0.008123, over 15483.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.0998, pruned_loss=0.01834, audio_tagging_loss=0.009564, over 3047891.42 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:42:15,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1342180.0, ans=0.125
2023-11-21 03:42:37,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201350
2023-11-21 03:42:47,621 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.00 vs. limit=12.0
2023-11-21 03:43:05,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1342446.6666666667, ans=0.0
2023-11-21 03:43:13,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1342513.3333333333, ans=0.0
2023-11-21 03:43:13,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1342513.3333333333, ans=0.125
2023-11-21 03:43:14,488 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9000, loss[loss=0.08181, simple_loss=0.09176, pruned_loss=0.02625, audio_tagging_loss=0.00969, over 14807.00 frames. ], tot_loss[loss=0.078, simple_loss=0.1002, pruned_loss=0.01842, audio_tagging_loss=0.009495, over 3045203.83 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:43:14,491 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 03:43:55,728 INFO [train_asr.py:1253] (0/4) Epoch 17, validation: loss=0.06143, simple_loss=0.05268, pruned_loss=0.005433, audio_tagging_loss=0.02966, over 4681554.00 frames. 
2023-11-21 03:43:55,728 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 03:43:57,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1342513.3333333333, ans=0.1
2023-11-21 03:43:59,600 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.19 vs. limit=12.0
2023-11-21 03:44:23,565 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201400
2023-11-21 03:44:28,635 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.286e+01 9.090e+01 9.750e+01 1.340e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-21 03:44:34,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1342713.3333333333, ans=0.09899494936611666
2023-11-21 03:44:36,831 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-21 03:45:00,657 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9050, loss[loss=0.06785, simple_loss=0.09162, pruned_loss=0.01203, audio_tagging_loss=0.01001, over 15050.00 frames. ], tot_loss[loss=0.07737, simple_loss=0.09922, pruned_loss=0.01829, audio_tagging_loss=0.009473, over 3044040.10 frames. ], batch size: 56, lr: 4.01e-03, grad_scale: 16.0
2023-11-21 03:45:11,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1342846.6666666667, ans=0.125
2023-11-21 03:45:26,926 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201450
2023-11-21 03:45:31,048 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=12.0
2023-11-21 03:45:31,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1342980.0, ans=0.2
2023-11-21 03:45:34,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1342980.0, ans=0.125
2023-11-21 03:46:02,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1343113.3333333333, ans=0.07
2023-11-21 03:46:04,623 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9100, loss[loss=0.06743, simple_loss=0.09049, pruned_loss=0.01205, audio_tagging_loss=0.01013, over 14558.00 frames. ], tot_loss[loss=0.0766, simple_loss=0.09825, pruned_loss=0.01799, audio_tagging_loss=0.00949, over 3039676.08 frames. ], batch size: 53, lr: 4.01e-03, grad_scale: 16.0
2023-11-21 03:46:04,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1343180.0, ans=0.1
2023-11-21 03:46:25,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1343246.6666666667, ans=0.0
2023-11-21 03:46:32,193 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201500
2023-11-21 03:46:38,211 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.193e+01 8.891e+01 9.703e+01 1.287e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 03:46:50,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1343380.0, ans=0.0
2023-11-21 03:47:02,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1343446.6666666667, ans=0.0
2023-11-21 03:47:06,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1343446.6666666667, ans=0.1
2023-11-21 03:47:08,813 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9150, loss[loss=0.07925, simple_loss=0.09173, pruned_loss=0.0219, audio_tagging_loss=0.01148, over 14102.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09799, pruned_loss=0.01781, audio_tagging_loss=0.009584, over 3041242.67 frames. ], batch size: 55, lr: 4.01e-03, grad_scale: 16.0
2023-11-21 03:47:10,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1343513.3333333333, ans=0.0
2023-11-21 03:47:17,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1343513.3333333333, ans=0.1
2023-11-21 03:47:34,522 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0
2023-11-21 03:47:35,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1343646.6666666667, ans=0.125
2023-11-21 03:47:36,987 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201550
2023-11-21 03:47:43,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1343646.6666666667, ans=0.04949747468305833
2023-11-21 03:47:56,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1343713.3333333333, ans=0.0
2023-11-21 03:47:57,163 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.02 vs. limit=15.0
2023-11-21 03:48:04,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1343780.0, ans=0.0
2023-11-21 03:48:08,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1343780.0, ans=0.125
2023-11-21 03:48:13,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1343846.6666666667, ans=0.125
2023-11-21 03:48:13,911 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9200, loss[loss=0.06659, simple_loss=0.07587, pruned_loss=0.01615, audio_tagging_loss=0.0125, over 14757.00 frames. ], tot_loss[loss=0.07621, simple_loss=0.09774, pruned_loss=0.01772, audio_tagging_loss=0.009624, over 3043443.11 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:48:19,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1343846.6666666667, ans=0.1
2023-11-21 03:48:20,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1343846.6666666667, ans=0.2
2023-11-21 03:48:21,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1343846.6666666667, ans=0.125
2023-11-21 03:48:22,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.05 vs. limit=15.0
2023-11-21 03:48:40,526 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201600
2023-11-21 03:48:46,815 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.667e+01 8.039e+01 8.659e+01 9.327e+01 1.552e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 03:48:54,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1344046.6666666667, ans=0.1
2023-11-21 03:48:57,437 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.74 vs. limit=15.0
2023-11-21 03:49:15,746 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.53 vs. limit=15.0
2023-11-21 03:49:16,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1344113.3333333333, ans=0.125
2023-11-21 03:49:18,530 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9250, loss[loss=0.07835, simple_loss=0.1054, pruned_loss=0.01617, audio_tagging_loss=0.009498, over 15784.00 frames. ], tot_loss[loss=0.0765, simple_loss=0.09831, pruned_loss=0.0178, audio_tagging_loss=0.009546, over 3045419.73 frames. ], batch size: 57, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:49:45,602 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201650
2023-11-21 03:49:47,440 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.25 vs. limit=10.0
2023-11-21 03:49:59,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1344380.0, ans=0.125
2023-11-21 03:50:05,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1344380.0, ans=0.015
2023-11-21 03:50:05,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1344380.0, ans=0.125
2023-11-21 03:50:21,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1344513.3333333333, ans=0.0
2023-11-21 03:50:22,240 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9300, loss[loss=0.05793, simple_loss=0.06832, pruned_loss=0.01301, audio_tagging_loss=0.01077, over 15301.00 frames. ], tot_loss[loss=0.07585, simple_loss=0.09744, pruned_loss=0.01746, audio_tagging_loss=0.009677, over 3048815.90 frames. ], batch size: 59, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:50:37,551 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.23 vs. limit=12.0
2023-11-21 03:50:42,928 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:50:42,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1344580.0, ans=0.125
2023-11-21 03:50:46,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1344580.0, ans=0.07
2023-11-21 03:50:46,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1344580.0, ans=0.0
2023-11-21 03:50:50,185 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201700
2023-11-21 03:50:56,658 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.829e+01 7.965e+01 8.483e+01 9.250e+01 1.325e+02, threshold=1.697e+02, percent-clipped=0.0
2023-11-21 03:51:01,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.36 vs. limit=15.0
2023-11-21 03:51:09,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.54 vs. limit=10.0
2023-11-21 03:51:10,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1344713.3333333333, ans=0.125
2023-11-21 03:51:19,972 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:51:27,526 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9350, loss[loss=0.07391, simple_loss=0.09797, pruned_loss=0.01612, audio_tagging_loss=0.008803, over 15801.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09696, pruned_loss=0.01751, audio_tagging_loss=0.009733, over 3047214.23 frames. ], batch size: 61, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:51:29,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1344846.6666666667, ans=0.125
2023-11-21 03:51:45,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1344913.3333333333, ans=0.0
2023-11-21 03:51:52,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1344980.0, ans=0.1
2023-11-21 03:51:54,709 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201750
2023-11-21 03:51:59,726 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 03:52:13,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1345046.6666666667, ans=0.07
2023-11-21 03:52:32,194 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9400, loss[loss=0.07411, simple_loss=0.09979, pruned_loss=0.01348, audio_tagging_loss=0.01074, over 17178.00 frames. ], tot_loss[loss=0.07549, simple_loss=0.0965, pruned_loss=0.0174, audio_tagging_loss=0.009838, over 3050310.46 frames. ], batch size: 62, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:52:42,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1345180.0, ans=0.2
2023-11-21 03:52:57,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1345313.3333333333, ans=0.0
2023-11-21 03:52:58,695 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201800
2023-11-21 03:53:04,951 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.174e+01 8.028e+01 8.710e+01 9.500e+01 1.230e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 03:53:21,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1345380.0, ans=0.125
2023-11-21 03:53:31,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1345446.6666666667, ans=0.0
2023-11-21 03:53:33,232 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 03:53:35,701 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9450, loss[loss=0.06596, simple_loss=0.09004, pruned_loss=0.01274, audio_tagging_loss=0.00821, over 15996.00 frames. ], tot_loss[loss=0.07552, simple_loss=0.09625, pruned_loss=0.01742, audio_tagging_loss=0.009971, over 3049001.52 frames. ], batch size: 60, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:53:39,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1345513.3333333333, ans=0.125
2023-11-21 03:53:41,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1345513.3333333333, ans=0.125
2023-11-21 03:54:03,465 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201850
2023-11-21 03:54:10,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1345646.6666666667, ans=0.125
2023-11-21 03:54:23,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1345713.3333333333, ans=0.125
2023-11-21 03:54:40,220 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9500, loss[loss=0.07534, simple_loss=0.09919, pruned_loss=0.01598, audio_tagging_loss=0.009771, over 15606.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.09664, pruned_loss=0.01744, audio_tagging_loss=0.009892, over 3053693.01 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:54:50,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1345846.6666666667, ans=0.125
2023-11-21 03:55:04,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1345980.0, ans=0.0
2023-11-21 03:55:06,962 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201900
2023-11-21 03:55:09,912 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.75 vs. limit=15.0
2023-11-21 03:55:12,978 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.020e+01 8.201e+01 8.880e+01 9.737e+01 1.239e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 03:55:14,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1345980.0, ans=0.0
2023-11-21 03:55:16,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1346046.6666666667, ans=0.2
2023-11-21 03:55:21,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1346046.6666666667, ans=0.125
2023-11-21 03:55:38,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1346113.3333333333, ans=0.125
2023-11-21 03:55:42,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1346180.0, ans=0.1
2023-11-21 03:55:44,097 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9550, loss[loss=0.06093, simple_loss=0.07492, pruned_loss=0.01133, audio_tagging_loss=0.01214, over 15020.00 frames. ], tot_loss[loss=0.07561, simple_loss=0.09655, pruned_loss=0.01733, audio_tagging_loss=0.01001, over 3047048.50 frames. ], batch size: 58, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:55:46,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1346180.0, ans=0.125
2023-11-21 03:56:10,926 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 201950
2023-11-21 03:56:17,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1346313.3333333333, ans=0.1
2023-11-21 03:56:30,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1346380.0, ans=0.1
2023-11-21 03:56:46,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1346446.6666666667, ans=0.2
2023-11-21 03:56:47,689 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.51 vs. limit=15.0
2023-11-21 03:56:48,201 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9600, loss[loss=0.06593, simple_loss=0.08379, pruned_loss=0.01272, audio_tagging_loss=0.01132, over 14717.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09632, pruned_loss=0.01723, audio_tagging_loss=0.01003, over 3045110.79 frames. ], batch size: 54, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:56:53,723 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.62 vs. limit=15.0
2023-11-21 03:57:13,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.65 vs. limit=15.0
2023-11-21 03:57:15,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202000
2023-11-21 03:57:21,649 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.303e+01 7.767e+01 8.568e+01 9.180e+01 1.180e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 03:57:31,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1346713.3333333333, ans=0.1
2023-11-21 03:57:38,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1346713.3333333333, ans=0.125
2023-11-21 03:57:41,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1346780.0, ans=0.1
2023-11-21 03:57:45,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1346780.0, ans=0.1
2023-11-21 03:57:53,419 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9650, loss[loss=0.07894, simple_loss=0.1008, pruned_loss=0.0204, audio_tagging_loss=0.008141, over 16855.00 frames. ], tot_loss[loss=0.07516, simple_loss=0.09614, pruned_loss=0.01713, audio_tagging_loss=0.00997, over 3044799.95 frames. ], batch size: 65, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:58:03,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1346846.6666666667, ans=0.125
2023-11-21 03:58:12,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1346913.3333333333, ans=0.125
2023-11-21 03:58:16,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1346913.3333333333, ans=0.0
2023-11-21 03:58:20,091 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202050
2023-11-21 03:58:21,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1346980.0, ans=0.125
2023-11-21 03:58:25,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1346980.0, ans=0.125
2023-11-21 03:58:27,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1346980.0, ans=0.09899494936611666
2023-11-21 03:58:28,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1346980.0, ans=0.0
2023-11-21 03:58:57,081 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9700, loss[loss=0.08202, simple_loss=0.1019, pruned_loss=0.01808, audio_tagging_loss=0.01297, over 14362.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.0975, pruned_loss=0.01742, audio_tagging_loss=0.009726, over 3049592.47 frames. ], batch size: 54, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 03:59:21,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1347313.3333333333, ans=0.125
2023-11-21 03:59:24,412 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202100
2023-11-21 03:59:28,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1347313.3333333333, ans=0.125
2023-11-21 03:59:31,478 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.702e+01 8.071e+01 8.852e+01 9.589e+01 1.175e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 03:59:45,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1347380.0, ans=0.07
2023-11-21 03:59:55,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1347446.6666666667, ans=0.0
2023-11-21 04:00:01,938 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9750, loss[loss=0.08887, simple_loss=0.1119, pruned_loss=0.0252, audio_tagging_loss=0.007705, over 14031.00 frames. ], tot_loss[loss=0.07623, simple_loss=0.09811, pruned_loss=0.01762, audio_tagging_loss=0.009556, over 3051112.72 frames. ], batch size: 54, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 04:00:28,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202150
2023-11-21 04:00:37,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1347646.6666666667, ans=0.125
2023-11-21 04:01:05,913 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9800, loss[loss=0.09708, simple_loss=0.1358, pruned_loss=0.02515, audio_tagging_loss=0.00403, over 15150.00 frames. ], tot_loss[loss=0.07598, simple_loss=0.09761, pruned_loss=0.01765, audio_tagging_loss=0.009525, over 3042672.86 frames. ], batch size: 55, lr: 4.01e-03, grad_scale: 32.0
2023-11-21 04:01:06,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1347846.6666666667, ans=0.0
2023-11-21 04:01:09,794 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.83 vs. limit=15.0
2023-11-21 04:01:24,726 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.77 vs. limit=5.0
2023-11-21 04:01:28,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1347913.3333333333, ans=0.0
2023-11-21 04:01:30,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1347980.0, ans=0.0
2023-11-21 04:01:33,227 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202200
2023-11-21 04:01:33,330 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:01:40,223 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.143e+01 8.218e+01 8.911e+01 9.582e+01 1.351e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-21 04:02:02,347 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:02:10,982 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9850, loss[loss=0.07104, simple_loss=0.08748, pruned_loss=0.01871, audio_tagging_loss=0.008586, over 14987.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.09782, pruned_loss=0.01769, audio_tagging_loss=0.009564, over 3048094.37 frames. ], batch size: 57, lr: 4.00e-03, grad_scale: 32.0
2023-11-21 04:02:23,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1348246.6666666667, ans=0.0
2023-11-21 04:02:23,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1348246.6666666667, ans=0.035
2023-11-21 04:02:36,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1348313.3333333333, ans=0.125
2023-11-21 04:02:36,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1348313.3333333333, ans=0.125
2023-11-21 04:02:38,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202250
2023-11-21 04:03:16,215 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9900, loss[loss=0.06939, simple_loss=0.09328, pruned_loss=0.01327, audio_tagging_loss=0.009486, over 15650.00 frames. ], tot_loss[loss=0.07645, simple_loss=0.09866, pruned_loss=0.0177, audio_tagging_loss=0.009418, over 3048897.56 frames. ], batch size: 57, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:03:19,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1348513.3333333333, ans=0.5
2023-11-21 04:03:44,154 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202300
2023-11-21 04:03:51,370 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.722e+01 8.036e+01 8.846e+01 9.540e+01 1.221e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 04:03:58,775 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.74 vs. limit=15.0
2023-11-21 04:04:00,176 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0
2023-11-21 04:04:02,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1348713.3333333333, ans=0.125
2023-11-21 04:04:21,098 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 9950, loss[loss=0.08425, simple_loss=0.1077, pruned_loss=0.02207, audio_tagging_loss=0.008356, over 15160.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09757, pruned_loss=0.0174, audio_tagging_loss=0.009439, over 3050518.48 frames. ], batch size: 55, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:04:48,413 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202350
2023-11-21 04:04:49,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1348980.0, ans=0.125
2023-11-21 04:04:51,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1348980.0, ans=0.125
2023-11-21 04:04:57,412 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.53 vs. limit=15.0
2023-11-21 04:05:02,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1349046.6666666667, ans=0.2
2023-11-21 04:05:05,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1349046.6666666667, ans=0.125
2023-11-21 04:05:18,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1349113.3333333333, ans=0.2
2023-11-21 04:05:25,849 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10000, loss[loss=0.07835, simple_loss=0.1027, pruned_loss=0.01614, audio_tagging_loss=0.01087, over 15196.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09781, pruned_loss=0.01755, audio_tagging_loss=0.009375, over 3046222.57 frames. ], batch size: 56, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:05:28,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1349180.0, ans=0.1
2023-11-21 04:05:29,137 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.88 vs. limit=15.0
2023-11-21 04:05:39,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1349246.6666666667, ans=0.035
2023-11-21 04:05:39,585 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.35 vs. limit=12.0
2023-11-21 04:05:41,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1349246.6666666667, ans=0.125
2023-11-21 04:05:52,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202400
2023-11-21 04:06:02,544 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.043e+01 8.738e+01 9.707e+01 1.240e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 04:06:08,464 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:06:21,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1349446.6666666667, ans=0.125
2023-11-21 04:06:22,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1349446.6666666667, ans=0.1
2023-11-21 04:06:29,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.58 vs. limit=15.0
2023-11-21 04:06:30,500 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10050, loss[loss=0.08297, simple_loss=0.1089, pruned_loss=0.01543, audio_tagging_loss=0.01307, over 15793.00 frames. ], tot_loss[loss=0.07586, simple_loss=0.09765, pruned_loss=0.01752, audio_tagging_loss=0.009514, over 3052267.63 frames. ], batch size: 58, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:06:30,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1349513.3333333333, ans=0.0
2023-11-21 04:06:45,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1349580.0, ans=0.04949747468305833
2023-11-21 04:06:58,158 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202450
2023-11-21 04:07:01,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1349646.6666666667, ans=0.1
2023-11-21 04:07:22,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1349780.0, ans=0.125
2023-11-21 04:07:34,520 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10100, loss[loss=0.05905, simple_loss=0.07277, pruned_loss=0.01179, audio_tagging_loss=0.01087, over 15999.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09806, pruned_loss=0.01774, audio_tagging_loss=0.009594, over 3052535.20 frames. ], batch size: 62, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:07:53,086 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.18 vs. limit=15.0
2023-11-21 04:08:02,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202500
2023-11-21 04:08:11,188 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.354e+01 8.956e+01 9.533e+01 1.205e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-21 04:08:17,964 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=13.35 vs. limit=15.0
2023-11-21 04:08:24,734 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:08:31,782 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.00 vs. limit=22.5
2023-11-21 04:08:39,860 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10150, loss[loss=0.08396, simple_loss=0.1027, pruned_loss=0.02258, audio_tagging_loss=0.01002, over 15100.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.0989, pruned_loss=0.01798, audio_tagging_loss=0.009616, over 3056004.81 frames. ], batch size: 56, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:08:48,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1350180.0, ans=0.1
2023-11-21 04:09:04,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1350313.3333333333, ans=0.125
2023-11-21 04:09:06,486 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202550
2023-11-21 04:09:07,625 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:09:19,381 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.92 vs. limit=15.0
2023-11-21 04:09:32,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1350446.6666666667, ans=0.0
2023-11-21 04:09:44,015 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10200, loss[loss=0.05884, simple_loss=0.07705, pruned_loss=0.01149, audio_tagging_loss=0.008816, over 15162.00 frames. ], tot_loss[loss=0.07677, simple_loss=0.09855, pruned_loss=0.01773, audio_tagging_loss=0.009767, over 3057716.99 frames. ], batch size: 60, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:10:00,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1350580.0, ans=0.0
2023-11-21 04:10:04,754 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:10:10,926 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202600
2023-11-21 04:10:19,601 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.093e+01 8.508e+01 9.534e+01 1.524e+02, threshold=1.702e+02, percent-clipped=0.0
2023-11-21 04:10:33,247 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.71 vs. limit=15.0
2023-11-21 04:10:34,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1350780.0, ans=0.0
2023-11-21 04:10:46,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1350846.6666666667, ans=0.0
2023-11-21 04:10:47,497 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10250, loss[loss=0.06608, simple_loss=0.07296, pruned_loss=0.01382, audio_tagging_loss=0.01578, over 15842.00 frames. ], tot_loss[loss=0.07627, simple_loss=0.09776, pruned_loss=0.01749, audio_tagging_loss=0.009904, over 3058735.91 frames. ], batch size: 60, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:10:50,712 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.37 vs. limit=15.0
2023-11-21 04:11:07,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1350913.3333333333, ans=0.125
2023-11-21 04:11:07,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1350913.3333333333, ans=0.1
2023-11-21 04:11:14,706 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202650
2023-11-21 04:11:17,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1350980.0, ans=0.125
2023-11-21 04:11:52,701 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10300, loss[loss=0.07585, simple_loss=0.09789, pruned_loss=0.01753, audio_tagging_loss=0.009376, over 15571.00 frames. ], tot_loss[loss=0.07644, simple_loss=0.09772, pruned_loss=0.0176, audio_tagging_loss=0.009984, over 3062244.43 frames. ], batch size: 57, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:11:57,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1351180.0, ans=0.0
2023-11-21 04:12:09,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1351246.6666666667, ans=0.2
2023-11-21 04:12:12,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1351246.6666666667, ans=0.1
2023-11-21 04:12:19,384 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202700
2023-11-21 04:12:21,358 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.72 vs. limit=15.0
2023-11-21 04:12:27,750 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.078e+01 8.129e+01 8.690e+01 9.404e+01 1.408e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 04:12:57,302 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10350, loss[loss=0.06542, simple_loss=0.07504, pruned_loss=0.0112, audio_tagging_loss=0.0167, over 14079.00 frames. ], tot_loss[loss=0.07668, simple_loss=0.09808, pruned_loss=0.01765, audio_tagging_loss=0.009985, over 3063408.54 frames. ], batch size: 55, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:13:06,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1351513.3333333333, ans=0.0
2023-11-21 04:13:07,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1351513.3333333333, ans=0.125
2023-11-21 04:13:11,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1351580.0, ans=0.0
2023-11-21 04:13:17,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1351580.0, ans=0.125
2023-11-21 04:13:17,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.49 vs. limit=15.0
2023-11-21 04:13:24,365 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202750
2023-11-21 04:13:27,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1351646.6666666667, ans=0.125
2023-11-21 04:13:45,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1351713.3333333333, ans=0.125
2023-11-21 04:13:48,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1351780.0, ans=0.0
2023-11-21 04:13:50,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1351780.0, ans=0.2
2023-11-21 04:13:51,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1351780.0, ans=0.0
2023-11-21 04:13:52,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1351780.0, ans=0.125
2023-11-21 04:13:59,704 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.51 vs. limit=15.0
2023-11-21 04:14:01,512 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10400, loss[loss=0.05492, simple_loss=0.06499, pruned_loss=0.009944, audio_tagging_loss=0.01248, over 15467.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09855, pruned_loss=0.01766, audio_tagging_loss=0.01007, over 3061507.73 frames. ], batch size: 59, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:14:04,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1351846.6666666667, ans=0.125
2023-11-21 04:14:23,935 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.36 vs. limit=5.0
2023-11-21 04:14:29,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202800
2023-11-21 04:14:33,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1351980.0, ans=0.2
2023-11-21 04:14:39,608 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.428e+01 7.927e+01 8.563e+01 9.335e+01 1.364e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 04:14:44,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1352046.6666666667, ans=0.2
2023-11-21 04:15:06,479 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10450, loss[loss=0.06933, simple_loss=0.0844, pruned_loss=0.01479, audio_tagging_loss=0.01234, over 15069.00 frames. ], tot_loss[loss=0.07609, simple_loss=0.09712, pruned_loss=0.01745, audio_tagging_loss=0.01009, over 3057991.12 frames. ], batch size: 58, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:15:28,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1352246.6666666667, ans=0.125
2023-11-21 04:15:33,465 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202850
2023-11-21 04:15:43,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1352380.0, ans=0.0
2023-11-21 04:15:44,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1352380.0, ans=0.1
2023-11-21 04:15:47,065 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:16:10,737 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10500, loss[loss=0.06312, simple_loss=0.07574, pruned_loss=0.01356, audio_tagging_loss=0.01169, over 15532.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09743, pruned_loss=0.01758, audio_tagging_loss=0.009952, over 3045412.50 frames. ], batch size: 60, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:16:11,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1352513.3333333333, ans=0.125
2023-11-21 04:16:37,790 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202900
2023-11-21 04:16:50,046 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.399e+01 8.004e+01 8.567e+01 9.229e+01 1.198e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 04:16:57,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1352713.3333333333, ans=0.0
2023-11-21 04:17:10,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.73 vs. limit=15.0
2023-11-21 04:17:16,041 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10550, loss[loss=0.07287, simple_loss=0.09161, pruned_loss=0.01808, audio_tagging_loss=0.008979, over 14631.00 frames. ], tot_loss[loss=0.07611, simple_loss=0.0977, pruned_loss=0.01744, audio_tagging_loss=0.009817, over 3041314.00 frames. ], batch size: 54, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:17:43,225 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 202950
2023-11-21 04:17:45,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1352980.0, ans=0.125
2023-11-21 04:17:53,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1352980.0, ans=0.2
2023-11-21 04:18:03,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1353046.6666666667, ans=0.1
2023-11-21 04:18:21,415 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10600, loss[loss=0.07242, simple_loss=0.09804, pruned_loss=0.01423, audio_tagging_loss=0.009167, over 14398.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.0975, pruned_loss=0.01741, audio_tagging_loss=0.009747, over 3034372.41 frames. ], batch size: 52, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:18:26,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1353180.0, ans=0.1
2023-11-21 04:18:43,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1353246.6666666667, ans=0.035
2023-11-21 04:18:44,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1353246.6666666667, ans=0.125
2023-11-21 04:18:48,719 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203000
2023-11-21 04:18:59,890 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.372e+01 8.133e+01 8.810e+01 9.778e+01 1.215e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 04:19:06,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1353380.0, ans=0.125
2023-11-21 04:19:26,451 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10650, loss[loss=0.06488, simple_loss=0.087, pruned_loss=0.01495, audio_tagging_loss=0.006431, over 15664.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09739, pruned_loss=0.01748, audio_tagging_loss=0.00974, over 3040063.02 frames. ], batch size: 60, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:19:27,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1353513.3333333333, ans=0.125
2023-11-21 04:19:27,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1353513.3333333333, ans=0.09899494936611666
2023-11-21 04:19:30,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1353513.3333333333, ans=0.125
2023-11-21 04:19:53,044 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203050
2023-11-21 04:19:57,909 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.10 vs. limit=22.5
2023-11-21 04:19:59,857 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=10.17 vs. limit=15.0
2023-11-21 04:20:07,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1353713.3333333333, ans=0.0
2023-11-21 04:20:22,023 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.76 vs. limit=22.5
2023-11-21 04:20:26,867 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.50 vs. limit=15.0
2023-11-21 04:20:31,168 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10700, loss[loss=0.07341, simple_loss=0.08665, pruned_loss=0.01775, audio_tagging_loss=0.01233, over 16072.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.09649, pruned_loss=0.01729, audio_tagging_loss=0.009816, over 3038293.38 frames. ], batch size: 63, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:20:33,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.98 vs. limit=6.0
2023-11-21 04:20:35,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1353846.6666666667, ans=0.2
2023-11-21 04:20:37,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1353846.6666666667, ans=0.125
2023-11-21 04:20:55,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1353913.3333333333, ans=0.09899494936611666
2023-11-21 04:20:58,630 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203100
2023-11-21 04:21:10,482 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.882e+01 7.991e+01 8.733e+01 9.483e+01 1.206e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 04:21:30,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1354113.3333333333, ans=0.125
2023-11-21 04:21:35,721 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10750, loss[loss=0.07445, simple_loss=0.09472, pruned_loss=0.01866, audio_tagging_loss=0.008429, over 14508.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09561, pruned_loss=0.01692, audio_tagging_loss=0.009778, over 3039065.96 frames. ], batch size: 53, lr: 4.00e-03, grad_scale: 8.0
2023-11-21 04:21:50,688 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.50 vs. limit=15.0
2023-11-21 04:21:52,039 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.15 vs. limit=15.0
2023-11-21 04:21:52,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1354246.6666666667, ans=0.125
2023-11-21 04:21:55,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1354246.6666666667, ans=0.1
2023-11-21 04:21:56,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1354246.6666666667, ans=0.125
2023-11-21 04:22:03,913 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203150
2023-11-21 04:22:07,183 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.34 vs. limit=15.0
2023-11-21 04:22:09,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1354313.3333333333, ans=0.125
2023-11-21 04:22:15,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1354380.0, ans=0.2
2023-11-21 04:22:18,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1354380.0, ans=0.0
2023-11-21 04:22:27,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1354446.6666666667, ans=0.07
2023-11-21 04:22:31,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1354446.6666666667, ans=0.2
2023-11-21 04:22:39,647 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.02 vs. limit=15.0
2023-11-21 04:22:41,291 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10800, loss[loss=0.07884, simple_loss=0.1023, pruned_loss=0.01759, audio_tagging_loss=0.01012, over 15056.00 frames. ], tot_loss[loss=0.07516, simple_loss=0.09651, pruned_loss=0.01715, audio_tagging_loss=0.009758, over 3040398.09 frames. ], batch size: 56, lr: 4.00e-03, grad_scale: 16.0
2023-11-21 04:22:43,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1354513.3333333333, ans=0.2
2023-11-21 04:23:03,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1354580.0, ans=0.125
2023-11-21 04:23:09,452 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203200
2023-11-21 04:23:21,929 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.183e+01 7.867e+01 8.488e+01 9.244e+01 1.126e+02, threshold=1.698e+02, percent-clipped=0.0
2023-11-21 04:23:24,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1354713.3333333333, ans=0.2
2023-11-21 04:23:38,023 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.80 vs. limit=22.5
2023-11-21 04:23:47,614 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10850, loss[loss=0.09197, simple_loss=0.1142, pruned_loss=0.0274, audio_tagging_loss=0.007472, over 14810.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.09703, pruned_loss=0.01723, audio_tagging_loss=0.009652, over 3037415.98 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:23:54,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1354846.6666666667, ans=0.125
2023-11-21 04:24:03,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1354913.3333333333, ans=0.2
2023-11-21 04:24:14,739 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203250
2023-11-21 04:24:23,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1354980.0, ans=0.125
2023-11-21 04:24:30,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1355046.6666666667, ans=0.125
2023-11-21 04:24:45,975 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:24:51,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1355180.0, ans=0.125
2023-11-21 04:24:52,205 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10900, loss[loss=0.09573, simple_loss=0.1192, pruned_loss=0.02717, audio_tagging_loss=0.008955, over 15023.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09706, pruned_loss=0.01739, audio_tagging_loss=0.009663, over 3047907.34 frames. ], batch size: 54, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:24:55,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1355180.0, ans=0.0
2023-11-21 04:25:00,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1355180.0, ans=10.0
2023-11-21 04:25:03,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.86 vs. limit=15.0
2023-11-21 04:25:20,168 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203300
2023-11-21 04:25:27,520 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.43 vs. limit=15.0
2023-11-21 04:25:31,861 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.236e+01 9.164e+01 1.009e+02 1.942e+02, threshold=1.833e+02, percent-clipped=1.0
2023-11-21 04:25:33,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1355380.0, ans=0.0
2023-11-21 04:25:37,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1355380.0, ans=0.125
2023-11-21 04:25:52,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1355446.6666666667, ans=0.125
2023-11-21 04:25:57,471 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 10950, loss[loss=0.08608, simple_loss=0.1094, pruned_loss=0.02337, audio_tagging_loss=0.007996, over 14865.00 frames. ], tot_loss[loss=0.0758, simple_loss=0.09733, pruned_loss=0.01747, audio_tagging_loss=0.009664, over 3047241.73 frames. ], batch size: 54, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:26:06,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1355513.3333333333, ans=0.2
2023-11-21 04:26:23,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.31 vs. limit=10.0
2023-11-21 04:26:23,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1355646.6666666667, ans=0.125
2023-11-21 04:26:24,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203350
2023-11-21 04:26:42,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1355713.3333333333, ans=0.125
2023-11-21 04:27:02,257 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11000, loss[loss=0.07961, simple_loss=0.09706, pruned_loss=0.01815, audio_tagging_loss=0.01293, over 14564.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09688, pruned_loss=0.01752, audio_tagging_loss=0.009749, over 3042080.69 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:27:10,226 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:27:14,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1355913.3333333333, ans=0.125
2023-11-21 04:27:16,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1355913.3333333333, ans=0.0
2023-11-21 04:27:20,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.13 vs. limit=15.0
2023-11-21 04:27:29,715 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203400
2023-11-21 04:27:36,890 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.77 vs. limit=6.0
2023-11-21 04:27:37,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1355980.0, ans=0.2
2023-11-21 04:27:40,975 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.079e+01 8.776e+01 9.579e+01 1.135e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 04:27:41,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1356046.6666666667, ans=0.125
2023-11-21 04:27:44,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1356046.6666666667, ans=0.2
2023-11-21 04:27:53,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1356113.3333333333, ans=0.1
2023-11-21 04:28:06,852 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11050, loss[loss=0.08837, simple_loss=0.09898, pruned_loss=0.02818, audio_tagging_loss=0.0107, over 16591.00 frames. ], tot_loss[loss=0.07669, simple_loss=0.09805, pruned_loss=0.01781, audio_tagging_loss=0.009854, over 3042542.46 frames. ], batch size: 61, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:28:10,079 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.75 vs. limit=15.0
2023-11-21 04:28:24,606 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.23 vs. limit=15.0
2023-11-21 04:28:34,473 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203450
2023-11-21 04:28:59,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1356446.6666666667, ans=0.2
2023-11-21 04:29:08,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1356446.6666666667, ans=0.125
2023-11-21 04:29:11,788 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11100, loss[loss=0.0676, simple_loss=0.0853, pruned_loss=0.01627, audio_tagging_loss=0.008679, over 15393.00 frames. ], tot_loss[loss=0.07657, simple_loss=0.09762, pruned_loss=0.01778, audio_tagging_loss=0.009987, over 3038831.27 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:29:24,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1356580.0, ans=0.125
2023-11-21 04:29:27,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1356580.0, ans=0.0
2023-11-21 04:29:39,520 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203500
2023-11-21 04:29:40,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1356646.6666666667, ans=0.2
2023-11-21 04:29:51,161 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.645e+01 8.437e+01 9.142e+01 9.871e+01 1.258e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-21 04:30:16,990 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11150, loss[loss=0.08802, simple_loss=0.1298, pruned_loss=0.01607, audio_tagging_loss=0.007042, over 16430.00 frames. ], tot_loss[loss=0.07668, simple_loss=0.09752, pruned_loss=0.01776, audio_tagging_loss=0.01016, over 3043580.66 frames. ], batch size: 60, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:30:36,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1356913.3333333333, ans=0.125
2023-11-21 04:30:45,425 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203550
2023-11-21 04:31:02,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1357046.6666666667, ans=0.125
2023-11-21 04:31:02,629 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.96 vs. limit=15.0
2023-11-21 04:31:17,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1357113.3333333333, ans=0.07
2023-11-21 04:31:23,104 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11200, loss[loss=0.05836, simple_loss=0.0696, pruned_loss=0.01034, audio_tagging_loss=0.01323, over 15791.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09692, pruned_loss=0.01744, audio_tagging_loss=0.01024, over 3042873.69 frames. ], batch size: 62, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:31:31,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1357180.0, ans=0.07
2023-11-21 04:31:44,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1357246.6666666667, ans=0.0
2023-11-21 04:31:45,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1357246.6666666667, ans=0.0
2023-11-21 04:31:51,177 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203600
2023-11-21 04:31:54,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1357313.3333333333, ans=0.1
2023-11-21 04:32:00,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1357313.3333333333, ans=0.125
2023-11-21 04:32:01,787 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.13 vs. limit=15.0
2023-11-21 04:32:02,370 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.264e+01 8.069e+01 8.575e+01 9.412e+01 1.593e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 04:32:15,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.71 vs. limit=15.0
2023-11-21 04:32:29,584 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11250, loss[loss=0.05819, simple_loss=0.07145, pruned_loss=0.01381, audio_tagging_loss=0.008663, over 14592.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09694, pruned_loss=0.01756, audio_tagging_loss=0.01022, over 3041052.14 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:32:36,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1357513.3333333333, ans=0.125
2023-11-21 04:32:52,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=22.5
2023-11-21 04:32:56,338 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203650
2023-11-21 04:33:09,198 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.27 vs. limit=6.0
2023-11-21 04:33:13,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.17 vs. limit=22.5
2023-11-21 04:33:17,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1357713.3333333333, ans=0.125
2023-11-21 04:33:21,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1357780.0, ans=0.1
2023-11-21 04:33:34,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1357846.6666666667, ans=0.1
2023-11-21 04:33:35,079 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11300, loss[loss=0.08205, simple_loss=0.1063, pruned_loss=0.01883, audio_tagging_loss=0.01006, over 16095.00 frames. ], tot_loss[loss=0.07649, simple_loss=0.09752, pruned_loss=0.01773, audio_tagging_loss=0.009997, over 3047052.34 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:33:49,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1357913.3333333333, ans=0.125
2023-11-21 04:34:02,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203700
2023-11-21 04:34:05,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1357980.0, ans=0.1
2023-11-21 04:34:09,668 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 04:34:14,173 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.294e+01 8.174e+01 8.923e+01 9.788e+01 1.326e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-21 04:34:37,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1358113.3333333333, ans=0.0
2023-11-21 04:34:39,463 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11350, loss[loss=0.06636, simple_loss=0.08522, pruned_loss=0.01444, audio_tagging_loss=0.00931, over 14552.00 frames. ], tot_loss[loss=0.07678, simple_loss=0.09811, pruned_loss=0.01795, audio_tagging_loss=0.009772, over 3051870.25 frames. ], batch size: 54, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:34:49,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1358180.0, ans=0.09899494936611666
2023-11-21 04:34:49,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1358180.0, ans=0.04949747468305833
2023-11-21 04:34:57,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1358246.6666666667, ans=0.0
2023-11-21 04:35:07,572 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203750
2023-11-21 04:35:23,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1358380.0, ans=0.125
2023-11-21 04:35:42,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1358446.6666666667, ans=0.125
2023-11-21 04:35:46,358 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11400, loss[loss=0.08951, simple_loss=0.1197, pruned_loss=0.02213, audio_tagging_loss=0.007534, over 15112.00 frames. ], tot_loss[loss=0.07679, simple_loss=0.0985, pruned_loss=0.01797, audio_tagging_loss=0.009571, over 3051225.65 frames. ], batch size: 55, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:35:47,026 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.64 vs. limit=10.0
2023-11-21 04:36:08,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1358580.0, ans=0.0
2023-11-21 04:36:13,266 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203800
2023-11-21 04:36:13,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1358646.6666666667, ans=0.125
2023-11-21 04:36:24,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1358713.3333333333, ans=0.04949747468305833
2023-11-21 04:36:24,725 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.066e+01 8.848e+01 9.491e+01 1.619e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 04:36:52,328 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11450, loss[loss=0.04646, simple_loss=0.05667, pruned_loss=0.007345, audio_tagging_loss=0.01079, over 14680.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09895, pruned_loss=0.0179, audio_tagging_loss=0.009428, over 3057251.92 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:37:08,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1358913.3333333333, ans=0.05
2023-11-21 04:37:18,663 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203850
2023-11-21 04:37:19,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.09 vs. limit=15.0
2023-11-21 04:37:24,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1358980.0, ans=0.125
2023-11-21 04:37:51,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1359113.3333333333, ans=0.125
2023-11-21 04:37:55,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1359180.0, ans=0.125
2023-11-21 04:37:56,320 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11500, loss[loss=0.06784, simple_loss=0.08133, pruned_loss=0.01475, audio_tagging_loss=0.01243, over 17199.00 frames. ], tot_loss[loss=0.07673, simple_loss=0.09863, pruned_loss=0.01788, audio_tagging_loss=0.009539, over 3062566.39 frames. ], batch size: 66, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:38:24,029 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203900
2023-11-21 04:38:35,522 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.753e+01 8.131e+01 8.864e+01 9.970e+01 1.435e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-21 04:38:38,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1359380.0, ans=0.0
2023-11-21 04:38:44,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1359380.0, ans=0.07
2023-11-21 04:39:01,272 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11550, loss[loss=0.08415, simple_loss=0.1082, pruned_loss=0.0198, audio_tagging_loss=0.01027, over 14789.00 frames. ], tot_loss[loss=0.07643, simple_loss=0.09822, pruned_loss=0.01772, audio_tagging_loss=0.009596, over 3058843.43 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:39:28,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 203950
2023-11-21 04:39:37,829 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 04:39:41,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1359713.3333333333, ans=0.125
2023-11-21 04:40:00,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1359780.0, ans=0.0
2023-11-21 04:40:05,795 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11600, loss[loss=0.07409, simple_loss=0.08756, pruned_loss=0.01846, audio_tagging_loss=0.01185, over 14116.00 frames. ], tot_loss[loss=0.07706, simple_loss=0.09892, pruned_loss=0.01803, audio_tagging_loss=0.009565, over 3051862.33 frames. ], batch size: 54, lr: 3.99e-03, grad_scale: 32.0
2023-11-21 04:40:22,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1359913.3333333333, ans=0.125
2023-11-21 04:40:32,546 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204000
2023-11-21 04:40:34,071 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-204000.pt
2023-11-21 04:40:51,365 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.912e+01 8.095e+01 8.795e+01 9.597e+01 1.214e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 04:41:01,516 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.33 vs. limit=15.0
2023-11-21 04:41:14,576 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11650, loss[loss=0.07999, simple_loss=0.1026, pruned_loss=0.01965, audio_tagging_loss=0.009054, over 15240.00 frames. ], tot_loss[loss=0.07658, simple_loss=0.09839, pruned_loss=0.01782, audio_tagging_loss=0.00957, over 3055150.17 frames. ], batch size: 58, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:41:38,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1360246.6666666667, ans=0.2
2023-11-21 04:41:42,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204050
2023-11-21 04:42:09,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1360446.6666666667, ans=0.125
2023-11-21 04:42:11,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1360446.6666666667, ans=0.125
2023-11-21 04:42:13,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1360446.6666666667, ans=0.1
2023-11-21 04:42:19,572 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11700, loss[loss=0.06923, simple_loss=0.09494, pruned_loss=0.01284, audio_tagging_loss=0.008918, over 14390.00 frames. ], tot_loss[loss=0.07629, simple_loss=0.098, pruned_loss=0.01769, audio_tagging_loss=0.009602, over 3051525.18 frames. ], batch size: 56, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:42:26,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1360513.3333333333, ans=0.0
2023-11-21 04:42:28,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1360513.3333333333, ans=0.125
2023-11-21 04:42:29,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1360513.3333333333, ans=0.0
2023-11-21 04:42:31,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1360580.0, ans=0.125
2023-11-21 04:42:47,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204100
2023-11-21 04:42:52,935 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.37 vs. limit=15.0
2023-11-21 04:43:00,702 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 7.797e+01 8.480e+01 9.195e+01 1.315e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-21 04:43:17,050 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.74 vs. limit=15.0
2023-11-21 04:43:24,826 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11750, loss[loss=0.07014, simple_loss=0.09097, pruned_loss=0.01391, audio_tagging_loss=0.01074, over 15535.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.09671, pruned_loss=0.01748, audio_tagging_loss=0.009826, over 3057209.19 frames. ], batch size: 57, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:43:38,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1360913.3333333333, ans=0.04949747468305833
2023-11-21 04:43:41,104 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.59 vs. limit=15.0
2023-11-21 04:43:41,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.73 vs. limit=6.0
2023-11-21 04:43:45,992 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.01 vs. limit=15.0
2023-11-21 04:43:51,714 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204150
2023-11-21 04:43:55,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1360980.0, ans=0.1
2023-11-21 04:44:14,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1361046.6666666667, ans=0.125
2023-11-21 04:44:29,400 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11800, loss[loss=0.08112, simple_loss=0.1084, pruned_loss=0.01806, audio_tagging_loss=0.00887, over 14646.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09763, pruned_loss=0.01762, audio_tagging_loss=0.009812, over 3058683.06 frames. ], batch size: 55, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:44:56,649 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204200
2023-11-21 04:45:02,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1361313.3333333333, ans=0.1
2023-11-21 04:45:11,506 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.252e+01 9.002e+01 9.752e+01 1.410e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 04:45:16,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1361380.0, ans=0.125
2023-11-21 04:45:33,875 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11850, loss[loss=0.08591, simple_loss=0.1101, pruned_loss=0.02128, audio_tagging_loss=0.009566, over 14618.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09796, pruned_loss=0.0176, audio_tagging_loss=0.009832, over 3054045.06 frames. ], batch size: 53, lr: 3.99e-03, grad_scale: 16.0
2023-11-21 04:45:39,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1361513.3333333333, ans=0.0
2023-11-21 04:45:45,712 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.17 vs. limit=15.0
2023-11-21 04:45:51,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.82 vs. limit=12.0
2023-11-21 04:45:56,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1361580.0, ans=0.125
2023-11-21 04:46:02,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204250
2023-11-21 04:46:28,199 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.89 vs. limit=15.0
2023-11-21 04:46:39,221 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11900, loss[loss=0.07203, simple_loss=0.09201, pruned_loss=0.01719, audio_tagging_loss=0.008838, over 15476.00 frames. ], tot_loss[loss=0.07596, simple_loss=0.0972, pruned_loss=0.01745, audio_tagging_loss=0.009908, over 3054911.45 frames. ], batch size: 59, lr: 3.98e-03, grad_scale: 16.0
2023-11-21 04:46:40,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1361846.6666666667, ans=0.125
2023-11-21 04:46:52,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1361913.3333333333, ans=0.125
2023-11-21 04:46:59,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1361913.3333333333, ans=0.0
2023-11-21 04:47:06,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204300
2023-11-21 04:47:07,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1361980.0, ans=0.0
2023-11-21 04:47:15,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1361980.0, ans=0.125
2023-11-21 04:47:20,915 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.041e+01 8.628e+01 9.158e+01 1.228e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-21 04:47:26,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1362046.6666666667, ans=0.1
2023-11-21 04:47:27,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1362046.6666666667, ans=0.2
2023-11-21 04:47:44,170 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 11950, loss[loss=0.07848, simple_loss=0.1038, pruned_loss=0.01802, audio_tagging_loss=0.008535, over 15404.00 frames. ], tot_loss[loss=0.07659, simple_loss=0.09804, pruned_loss=0.01763, audio_tagging_loss=0.009944, over 3052975.72 frames. ], batch size: 56, lr: 3.98e-03, grad_scale: 16.0
2023-11-21 04:47:50,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1362180.0, ans=0.0
2023-11-21 04:48:10,713 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204350
2023-11-21 04:48:12,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=24.06 vs. limit=22.5
2023-11-21 04:48:26,512 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.17 vs. limit=22.5
2023-11-21 04:48:28,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1362380.0, ans=0.0
2023-11-21 04:48:45,545 INFO [train_asr.py:1221] (0/4) Epoch 17, batch 12000, loss[loss=0.07849, simple_loss=0.09116, pruned_loss=0.0203, audio_tagging_loss=0.01261, over 13895.00 frames. ], tot_loss[loss=0.07652, simple_loss=0.0979, pruned_loss=0.01761, audio_tagging_loss=0.009959, over 3039399.92 frames. ], batch size: 53, lr: 3.98e-03, grad_scale: 32.0
2023-11-21 04:48:45,548 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 04:49:26,098 INFO [train_asr.py:1253] (0/4) Epoch 17, validation: loss=0.06069, simple_loss=0.05267, pruned_loss=0.005387, audio_tagging_loss=0.02896, over 4681554.00 frames. 
2023-11-21 04:49:26,099 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 04:49:29,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1362513.3333333333, ans=0.125
2023-11-21 04:49:37,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1362580.0, ans=0.0
2023-11-21 04:49:51,180 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204400
2023-11-21 04:49:55,968 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-17.pt
2023-11-21 04:50:33,272 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 0, loss[loss=0.07953, simple_loss=0.08512, pruned_loss=0.01331, audio_tagging_loss=0.02366, over 15102.00 frames. ], tot_loss[loss=0.07953, simple_loss=0.08512, pruned_loss=0.01331, audio_tagging_loss=0.02366, over 15102.00 frames. ], batch size: 57, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:50:33,275 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 04:50:53,801 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7596, 5.7618, 5.8342, 5.8164], device='cuda:0')
2023-11-21 04:51:08,570 INFO [train_asr.py:1253] (0/4) Epoch 18, validation: loss=0.05959, simple_loss=0.05266, pruned_loss=0.005405, audio_tagging_loss=0.02786, over 4681554.00 frames. 
2023-11-21 04:51:08,571 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 04:51:19,307 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.498e+01 8.070e+01 8.803e+01 9.675e+01 1.246e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 04:51:39,680 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.13 vs. limit=22.5
2023-11-21 04:51:54,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1362866.6666666667, ans=0.07
2023-11-21 04:51:54,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1362866.6666666667, ans=0.0
2023-11-21 04:51:59,026 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.14 vs. limit=6.0
2023-11-21 04:52:03,792 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.46 vs. limit=15.0
2023-11-21 04:52:09,457 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204450
2023-11-21 04:52:11,814 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 50, loss[loss=0.09133, simple_loss=0.104, pruned_loss=0.02047, audio_tagging_loss=0.01885, over 15461.00 frames. ], tot_loss[loss=0.08508, simple_loss=0.09693, pruned_loss=0.01745, audio_tagging_loss=0.01916, over 682847.31 frames. ], batch size: 56, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:52:15,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1363000.0, ans=0.125
2023-11-21 04:52:31,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.92 vs. limit=15.0
2023-11-21 04:52:36,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1363133.3333333333, ans=0.0
2023-11-21 04:52:40,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1363133.3333333333, ans=0.125
2023-11-21 04:53:13,391 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204500
2023-11-21 04:53:15,783 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 100, loss[loss=0.06385, simple_loss=0.07347, pruned_loss=0.008685, audio_tagging_loss=0.01843, over 15456.00 frames. ], tot_loss[loss=0.08371, simple_loss=0.09698, pruned_loss=0.01705, audio_tagging_loss=0.01817, over 1202153.82 frames. ], batch size: 58, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:53:25,276 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=10.81 vs. limit=15.0
2023-11-21 04:53:28,009 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.544e+01 9.245e+01 1.008e+02 1.490e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-21 04:53:31,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1363400.0, ans=0.125
2023-11-21 04:53:33,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1363400.0, ans=0.5
2023-11-21 04:53:39,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1363400.0, ans=0.125
2023-11-21 04:54:12,610 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.60 vs. limit=6.0
2023-11-21 04:54:18,667 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204550
2023-11-21 04:54:21,055 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 150, loss[loss=0.05063, simple_loss=0.06211, pruned_loss=0.006239, audio_tagging_loss=0.01334, over 15610.00 frames. ], tot_loss[loss=0.082, simple_loss=0.09757, pruned_loss=0.01729, audio_tagging_loss=0.01593, over 1608413.30 frames. ], batch size: 59, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:54:21,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1363666.6666666667, ans=0.1
2023-11-21 04:54:34,674 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.81 vs. limit=15.0
2023-11-21 04:54:40,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1363733.3333333333, ans=0.125
2023-11-21 04:54:51,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1363800.0, ans=0.125
2023-11-21 04:55:23,552 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204600
2023-11-21 04:55:26,218 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 200, loss[loss=0.06349, simple_loss=0.07663, pruned_loss=0.01098, audio_tagging_loss=0.01419, over 15214.00 frames. ], tot_loss[loss=0.07997, simple_loss=0.09725, pruned_loss=0.01721, audio_tagging_loss=0.01414, over 1931916.43 frames. ], batch size: 56, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:55:37,066 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.030e+01 8.028e+01 8.648e+01 9.287e+01 1.143e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 04:55:46,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=1364066.6666666667, ans=0.05
2023-11-21 04:55:48,004 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.22 vs. limit=10.0
2023-11-21 04:55:54,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1364133.3333333333, ans=0.125
2023-11-21 04:56:02,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1364133.3333333333, ans=0.125
2023-11-21 04:56:02,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1364133.3333333333, ans=0.2
2023-11-21 04:56:26,756 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204650
2023-11-21 04:56:29,165 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 250, loss[loss=0.09333, simple_loss=0.121, pruned_loss=0.02521, audio_tagging_loss=0.007614, over 14792.00 frames. ], tot_loss[loss=0.07878, simple_loss=0.09724, pruned_loss=0.01733, audio_tagging_loss=0.01283, over 2173683.33 frames. ], batch size: 54, lr: 3.87e-03, grad_scale: 16.0
2023-11-21 04:56:29,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1364333.3333333333, ans=0.09899494936611666
2023-11-21 04:56:36,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1364333.3333333333, ans=0.1
2023-11-21 04:56:55,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1364466.6666666667, ans=0.1
2023-11-21 04:57:19,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1364600.0, ans=0.125
2023-11-21 04:57:22,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.31 vs. limit=15.0
2023-11-21 04:57:22,717 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.99 vs. limit=15.0
2023-11-21 04:57:24,234 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.51 vs. limit=22.5
2023-11-21 04:57:31,698 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204700
2023-11-21 04:57:34,637 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 300, loss[loss=0.04558, simple_loss=0.05666, pruned_loss=0.006798, audio_tagging_loss=0.01046, over 14422.00 frames. ], tot_loss[loss=0.07806, simple_loss=0.09746, pruned_loss=0.01741, audio_tagging_loss=0.01193, over 2366088.26 frames. ], batch size: 58, lr: 3.87e-03, grad_scale: 16.0
2023-11-21 04:57:42,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1364666.6666666667, ans=0.125
2023-11-21 04:57:46,744 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.596e+01 8.107e+01 8.931e+01 9.351e+01 1.204e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 04:58:10,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1364866.6666666667, ans=0.125
2023-11-21 04:58:25,968 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.28 vs. limit=15.0
2023-11-21 04:58:28,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0
2023-11-21 04:58:35,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204750
2023-11-21 04:58:37,520 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 350, loss[loss=0.08417, simple_loss=0.1059, pruned_loss=0.02029, audio_tagging_loss=0.01092, over 15930.00 frames. ], tot_loss[loss=0.07768, simple_loss=0.09786, pruned_loss=0.01751, audio_tagging_loss=0.01124, over 2529347.95 frames. ], batch size: 58, lr: 3.87e-03, grad_scale: 16.0
2023-11-21 04:58:44,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1365000.0, ans=0.125
2023-11-21 04:59:06,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1365133.3333333333, ans=0.0
2023-11-21 04:59:13,755 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.35 vs. limit=15.0
2023-11-21 04:59:24,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1365200.0, ans=0.125
2023-11-21 04:59:30,441 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.66 vs. limit=15.0
2023-11-21 04:59:39,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204800
2023-11-21 04:59:42,262 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 400, loss[loss=0.06855, simple_loss=0.07852, pruned_loss=0.01586, audio_tagging_loss=0.01342, over 14553.00 frames. ], tot_loss[loss=0.07662, simple_loss=0.09659, pruned_loss=0.01727, audio_tagging_loss=0.01106, over 2651536.73 frames. ], batch size: 57, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 04:59:44,338 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.13 vs. limit=12.0
2023-11-21 04:59:55,650 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.688e+01 8.042e+01 8.695e+01 9.547e+01 1.167e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 05:00:17,898 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.24 vs. limit=15.0
2023-11-21 05:00:29,806 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.86 vs. limit=15.0
2023-11-21 05:00:40,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1365600.0, ans=0.125
2023-11-21 05:00:41,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1365600.0, ans=0.0
2023-11-21 05:00:45,037 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204850
2023-11-21 05:00:47,475 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 450, loss[loss=0.07836, simple_loss=0.1039, pruned_loss=0.01439, audio_tagging_loss=0.01203, over 14728.00 frames. ], tot_loss[loss=0.0771, simple_loss=0.09785, pruned_loss=0.0175, audio_tagging_loss=0.01068, over 2739619.79 frames. ], batch size: 54, lr: 3.87e-03, grad_scale: 32.0
2023-11-21 05:01:07,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1365733.3333333333, ans=0.125
2023-11-21 05:01:21,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1365800.0, ans=0.125
2023-11-21 05:01:25,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1365866.6666666667, ans=0.0
2023-11-21 05:01:49,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1365933.3333333333, ans=0.125
2023-11-21 05:01:50,493 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204900
2023-11-21 05:01:52,762 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 500, loss[loss=0.08228, simple_loss=0.109, pruned_loss=0.02157, audio_tagging_loss=0.006227, over 15706.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09772, pruned_loss=0.0176, audio_tagging_loss=0.01058, over 2806569.99 frames. ], batch size: 59, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:01:56,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1366000.0, ans=0.125
2023-11-21 05:01:59,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1366000.0, ans=0.1
2023-11-21 05:02:03,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1366000.0, ans=0.0
2023-11-21 05:02:05,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1366066.6666666667, ans=0.04949747468305833
2023-11-21 05:02:07,173 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.144e+01 8.900e+01 9.756e+01 1.828e+02, threshold=1.780e+02, percent-clipped=1.0
2023-11-21 05:02:28,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1366133.3333333333, ans=0.1
2023-11-21 05:02:55,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 204950
2023-11-21 05:02:57,717 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 550, loss[loss=0.0993, simple_loss=0.122, pruned_loss=0.0266, audio_tagging_loss=0.01172, over 15180.00 frames. ], tot_loss[loss=0.07699, simple_loss=0.09813, pruned_loss=0.0174, audio_tagging_loss=0.01052, over 2867550.30 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:02:59,929 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.41 vs. limit=15.0
2023-11-21 05:03:04,443 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=11.17 vs. limit=12.0
2023-11-21 05:03:17,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1366400.0, ans=0.125
2023-11-21 05:03:29,320 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.33 vs. limit=10.0
2023-11-21 05:03:48,445 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.18 vs. limit=6.0
2023-11-21 05:03:59,792 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205000
2023-11-21 05:04:02,516 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 600, loss[loss=0.1131, simple_loss=0.1513, pruned_loss=0.02998, audio_tagging_loss=0.007468, over 15828.00 frames. ], tot_loss[loss=0.0769, simple_loss=0.09826, pruned_loss=0.01749, audio_tagging_loss=0.01028, over 2909149.35 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:04:16,852 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.655e+01 8.071e+01 8.678e+01 9.459e+01 1.222e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 05:04:18,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1366733.3333333333, ans=0.125
2023-11-21 05:04:29,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1366800.0, ans=0.1
2023-11-21 05:04:31,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1366800.0, ans=0.125
2023-11-21 05:04:45,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1366866.6666666667, ans=0.2
2023-11-21 05:05:05,045 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205050
2023-11-21 05:05:07,409 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 650, loss[loss=0.07785, simple_loss=0.09326, pruned_loss=0.0217, audio_tagging_loss=0.009514, over 15144.00 frames. ], tot_loss[loss=0.077, simple_loss=0.09859, pruned_loss=0.01762, audio_tagging_loss=0.01009, over 2940521.27 frames. ], batch size: 59, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:05:10,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1367000.0, ans=0.1
2023-11-21 05:05:14,611 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.65 vs. limit=22.5
2023-11-21 05:05:17,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1367000.0, ans=0.2
2023-11-21 05:06:05,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.80 vs. limit=15.0
2023-11-21 05:06:10,178 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205100
2023-11-21 05:06:12,517 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 700, loss[loss=0.07584, simple_loss=0.1052, pruned_loss=0.01416, audio_tagging_loss=0.009098, over 16373.00 frames. ], tot_loss[loss=0.0765, simple_loss=0.098, pruned_loss=0.01749, audio_tagging_loss=0.01001, over 2964796.90 frames. ], batch size: 58, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:06:19,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1367333.3333333333, ans=0.0
2023-11-21 05:06:23,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1367333.3333333333, ans=0.125
2023-11-21 05:06:24,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1367400.0, ans=0.2
2023-11-21 05:06:27,564 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.155e+01 8.827e+01 9.700e+01 1.144e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-21 05:06:49,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1367466.6666666667, ans=0.2
2023-11-21 05:07:16,688 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205150
2023-11-21 05:07:19,619 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 750, loss[loss=0.07411, simple_loss=0.09542, pruned_loss=0.01616, audio_tagging_loss=0.01024, over 14048.00 frames. ], tot_loss[loss=0.07734, simple_loss=0.0993, pruned_loss=0.01784, audio_tagging_loss=0.00985, over 2987375.07 frames. ], batch size: 54, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:07:38,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1367733.3333333333, ans=0.125
2023-11-21 05:07:50,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1367800.0, ans=0.1
2023-11-21 05:07:51,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1367800.0, ans=0.0
2023-11-21 05:08:12,913 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.20 vs. limit=6.0
2023-11-21 05:08:22,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205200
2023-11-21 05:08:25,719 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 800, loss[loss=0.08265, simple_loss=0.1129, pruned_loss=0.01886, audio_tagging_loss=0.007338, over 15233.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09879, pruned_loss=0.0176, audio_tagging_loss=0.009816, over 3007716.24 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:08:26,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1368000.0, ans=0.125
2023-11-21 05:08:35,445 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.62 vs. limit=10.0
2023-11-21 05:08:39,580 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.688e+01 8.168e+01 8.753e+01 9.823e+01 1.363e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 05:08:48,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1368066.6666666667, ans=0.2
2023-11-21 05:09:08,969 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.38 vs. limit=15.0
2023-11-21 05:09:28,114 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205250
2023-11-21 05:09:30,509 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 850, loss[loss=0.05071, simple_loss=0.06401, pruned_loss=0.00732, audio_tagging_loss=0.01138, over 14061.00 frames. ], tot_loss[loss=0.07708, simple_loss=0.09912, pruned_loss=0.01768, audio_tagging_loss=0.009841, over 3015935.90 frames. ], batch size: 53, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:10:13,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1368533.3333333333, ans=0.125
2023-11-21 05:10:14,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1368533.3333333333, ans=0.125
2023-11-21 05:10:20,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1368600.0, ans=0.0
2023-11-21 05:10:22,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1368600.0, ans=0.1
2023-11-21 05:10:32,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205300
2023-11-21 05:10:32,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1368600.0, ans=0.125
2023-11-21 05:10:35,161 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 900, loss[loss=0.06128, simple_loss=0.07987, pruned_loss=0.0126, audio_tagging_loss=0.008746, over 15702.00 frames. ], tot_loss[loss=0.07705, simple_loss=0.09875, pruned_loss=0.0177, audio_tagging_loss=0.009971, over 3025474.46 frames. ], batch size: 60, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:10:35,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1368666.6666666667, ans=0.125
2023-11-21 05:10:39,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1368666.6666666667, ans=0.125
2023-11-21 05:10:39,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.89 vs. limit=22.5
2023-11-21 05:10:48,991 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.47 vs. limit=12.0
2023-11-21 05:10:50,735 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 8.117e+01 8.694e+01 9.614e+01 1.237e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 05:10:52,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1368733.3333333333, ans=0.0
2023-11-21 05:11:15,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1368866.6666666667, ans=0.05
2023-11-21 05:11:25,720 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.61 vs. limit=6.0
2023-11-21 05:11:33,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1368933.3333333333, ans=0.2
2023-11-21 05:11:38,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1368933.3333333333, ans=0.07
2023-11-21 05:11:39,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205350
2023-11-21 05:11:40,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1369000.0, ans=0.125
2023-11-21 05:11:41,873 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 950, loss[loss=0.07024, simple_loss=0.08608, pruned_loss=0.01907, audio_tagging_loss=0.008128, over 14385.00 frames. ], tot_loss[loss=0.07706, simple_loss=0.09893, pruned_loss=0.0177, audio_tagging_loss=0.009901, over 3032679.90 frames. ], batch size: 55, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:11:48,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1369000.0, ans=0.0
2023-11-21 05:11:55,008 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.75 vs. limit=15.0
2023-11-21 05:12:11,345 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.14 vs. limit=10.0
2023-11-21 05:12:26,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1369200.0, ans=0.125
2023-11-21 05:12:28,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1369200.0, ans=0.2
2023-11-21 05:12:43,738 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205400
2023-11-21 05:12:46,440 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1000, loss[loss=0.08668, simple_loss=0.1111, pruned_loss=0.02073, audio_tagging_loss=0.01041, over 15188.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.09833, pruned_loss=0.01753, audio_tagging_loss=0.009826, over 3035758.10 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:12:48,360 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.63 vs. limit=12.0
2023-11-21 05:12:58,307 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.41 vs. limit=15.0
2023-11-21 05:13:01,198 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.289e+01 9.122e+01 9.845e+01 1.226e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-21 05:13:14,412 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:13:17,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1369466.6666666667, ans=0.125
2023-11-21 05:13:18,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1369466.6666666667, ans=0.125
2023-11-21 05:13:45,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1369600.0, ans=0.04949747468305833
2023-11-21 05:13:47,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.19 vs. limit=15.0
2023-11-21 05:13:48,401 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205450
2023-11-21 05:13:50,696 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1050, loss[loss=0.06191, simple_loss=0.07544, pruned_loss=0.01314, audio_tagging_loss=0.01105, over 15051.00 frames. ], tot_loss[loss=0.07701, simple_loss=0.09921, pruned_loss=0.01772, audio_tagging_loss=0.009688, over 3034094.05 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:13:51,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1369666.6666666667, ans=0.125
2023-11-21 05:14:09,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1369733.3333333333, ans=0.125
2023-11-21 05:14:55,394 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205500
2023-11-21 05:14:57,819 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1100, loss[loss=0.06763, simple_loss=0.08653, pruned_loss=0.01518, audio_tagging_loss=0.009186, over 15218.00 frames. ], tot_loss[loss=0.07658, simple_loss=0.09856, pruned_loss=0.01767, audio_tagging_loss=0.009634, over 3032090.06 frames. ], batch size: 55, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:15:01,465 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:15:04,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1370000.0, ans=0.125
2023-11-21 05:15:04,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1370000.0, ans=0.0
2023-11-21 05:15:11,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1370066.6666666667, ans=0.0
2023-11-21 05:15:12,509 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.708e+01 8.447e+01 9.123e+01 9.824e+01 1.304e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-21 05:15:21,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1370133.3333333333, ans=0.0
2023-11-21 05:15:22,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.25 vs. limit=12.0
2023-11-21 05:15:41,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1370200.0, ans=0.1
2023-11-21 05:15:46,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.21 vs. limit=15.0
2023-11-21 05:15:50,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1370266.6666666667, ans=0.125
2023-11-21 05:15:51,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1370266.6666666667, ans=0.1
2023-11-21 05:15:59,779 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205550
2023-11-21 05:16:02,106 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1150, loss[loss=0.08275, simple_loss=0.106, pruned_loss=0.01965, audio_tagging_loss=0.0101, over 15507.00 frames. ], tot_loss[loss=0.07625, simple_loss=0.09815, pruned_loss=0.01766, audio_tagging_loss=0.00951, over 3031894.15 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 16.0
2023-11-21 05:16:06,539 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.51 vs. limit=15.0
2023-11-21 05:16:21,381 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.36 vs. limit=12.0
2023-11-21 05:16:24,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1370400.0, ans=0.125
2023-11-21 05:16:28,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1370466.6666666667, ans=0.0
2023-11-21 05:16:35,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1370466.6666666667, ans=0.125
2023-11-21 05:16:40,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1370533.3333333333, ans=0.125
2023-11-21 05:16:44,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1370533.3333333333, ans=0.2
2023-11-21 05:16:50,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.11 vs. limit=15.0
2023-11-21 05:17:04,207 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205600
2023-11-21 05:17:07,024 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1200, loss[loss=0.06632, simple_loss=0.07057, pruned_loss=0.01763, audio_tagging_loss=0.01341, over 16123.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09748, pruned_loss=0.01768, audio_tagging_loss=0.009678, over 3037857.37 frames. ], batch size: 61, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:17:23,722 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.058e+01 8.835e+01 9.727e+01 1.276e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 05:17:29,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1370733.3333333333, ans=0.0
2023-11-21 05:17:36,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1370800.0, ans=0.1
2023-11-21 05:17:53,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1370866.6666666667, ans=0.0
2023-11-21 05:17:53,532 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.65 vs. limit=10.0
2023-11-21 05:17:55,837 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:18:04,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=1370933.3333333333, ans=10.0
2023-11-21 05:18:09,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1370933.3333333333, ans=0.0
2023-11-21 05:18:09,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1370933.3333333333, ans=0.0
2023-11-21 05:18:09,641 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.85 vs. limit=12.0
2023-11-21 05:18:10,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205650
2023-11-21 05:18:12,819 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1250, loss[loss=0.08918, simple_loss=0.1178, pruned_loss=0.02081, audio_tagging_loss=0.009483, over 16000.00 frames. ], tot_loss[loss=0.07549, simple_loss=0.09666, pruned_loss=0.01749, audio_tagging_loss=0.009674, over 3041316.62 frames. ], batch size: 61, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:18:25,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1371066.6666666667, ans=0.125
2023-11-21 05:18:28,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.41 vs. limit=5.0
2023-11-21 05:18:38,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1371133.3333333333, ans=0.125
2023-11-21 05:18:46,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1371133.3333333333, ans=0.125
2023-11-21 05:18:52,903 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.59 vs. limit=10.0
2023-11-21 05:18:55,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1371200.0, ans=0.125
2023-11-21 05:19:07,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1371266.6666666667, ans=0.0
2023-11-21 05:19:16,738 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205700
2023-11-21 05:19:19,139 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1300, loss[loss=0.08333, simple_loss=0.1092, pruned_loss=0.02229, audio_tagging_loss=0.006426, over 14449.00 frames. ], tot_loss[loss=0.07551, simple_loss=0.09683, pruned_loss=0.01749, audio_tagging_loss=0.0096, over 3039566.50 frames. ], batch size: 55, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:19:33,953 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.156e+01 8.683e+01 9.710e+01 1.341e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 05:19:35,880 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.68 vs. limit=22.5
2023-11-21 05:19:57,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.23 vs. limit=15.0
2023-11-21 05:20:06,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1371533.3333333333, ans=0.05
2023-11-21 05:20:19,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1371600.0, ans=0.0
2023-11-21 05:20:21,731 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205750
2023-11-21 05:20:24,026 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1350, loss[loss=0.09468, simple_loss=0.1278, pruned_loss=0.02306, audio_tagging_loss=0.007708, over 14633.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.0967, pruned_loss=0.01743, audio_tagging_loss=0.009573, over 3042722.78 frames. ], batch size: 53, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:20:24,789 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.91 vs. limit=15.0
2023-11-21 05:20:25,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1371666.6666666667, ans=0.2
2023-11-21 05:20:29,660 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=14.19 vs. limit=15.0
2023-11-21 05:20:38,745 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.66 vs. limit=5.0
2023-11-21 05:20:55,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1371800.0, ans=0.5
2023-11-21 05:21:11,034 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:21:26,999 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205800
2023-11-21 05:21:29,767 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1400, loss[loss=0.07068, simple_loss=0.09241, pruned_loss=0.01785, audio_tagging_loss=0.006627, over 15333.00 frames. ], tot_loss[loss=0.07518, simple_loss=0.09674, pruned_loss=0.01725, audio_tagging_loss=0.009552, over 3044463.61 frames. ], batch size: 58, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:21:40,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.23 vs. limit=10.0
2023-11-21 05:21:45,819 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.942e+01 8.095e+01 8.796e+01 9.675e+01 1.234e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 05:21:56,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1372133.3333333333, ans=0.1
2023-11-21 05:21:57,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1372133.3333333333, ans=0.2
2023-11-21 05:22:32,399 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205850
2023-11-21 05:22:32,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1372266.6666666667, ans=0.2
2023-11-21 05:22:32,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1372266.6666666667, ans=0.125
2023-11-21 05:22:35,555 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1450, loss[loss=0.0874, simple_loss=0.1112, pruned_loss=0.02142, audio_tagging_loss=0.01039, over 15799.00 frames. ], tot_loss[loss=0.07571, simple_loss=0.09738, pruned_loss=0.01743, audio_tagging_loss=0.009595, over 3043793.94 frames. ], batch size: 57, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:22:43,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.68 vs. limit=12.0
2023-11-21 05:22:44,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1372333.3333333333, ans=0.1
2023-11-21 05:22:49,561 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:22:57,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1372400.0, ans=0.0
2023-11-21 05:23:06,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1372466.6666666667, ans=0.0
2023-11-21 05:23:06,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1372466.6666666667, ans=0.0
2023-11-21 05:23:13,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1372533.3333333333, ans=0.0
2023-11-21 05:23:28,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1372600.0, ans=0.0
2023-11-21 05:23:36,985 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.03 vs. limit=12.0
2023-11-21 05:23:37,672 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205900
2023-11-21 05:23:40,180 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1500, loss[loss=0.05712, simple_loss=0.06708, pruned_loss=0.01326, audio_tagging_loss=0.01032, over 13802.00 frames. ], tot_loss[loss=0.07523, simple_loss=0.09673, pruned_loss=0.01715, audio_tagging_loss=0.009711, over 3039097.69 frames. ], batch size: 56, lr: 3.86e-03, grad_scale: 32.0
2023-11-21 05:23:45,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1372666.6666666667, ans=0.2
2023-11-21 05:23:46,625 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:23:46,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1372666.6666666667, ans=0.125
2023-11-21 05:23:55,895 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.737e+01 8.193e+01 8.878e+01 9.448e+01 1.243e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 05:23:59,463 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.13 vs. limit=15.0
2023-11-21 05:24:09,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1372800.0, ans=0.0
2023-11-21 05:24:13,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1372800.0, ans=0.09899494936611666
2023-11-21 05:24:16,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1372800.0, ans=0.125
2023-11-21 05:24:43,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 205950
2023-11-21 05:24:45,770 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1550, loss[loss=0.06564, simple_loss=0.07571, pruned_loss=0.01404, audio_tagging_loss=0.01375, over 15072.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09601, pruned_loss=0.0172, audio_tagging_loss=0.009856, over 3043689.65 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:25:02,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1373066.6666666667, ans=0.0
2023-11-21 05:25:18,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1373133.3333333333, ans=0.0
2023-11-21 05:25:22,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1373133.3333333333, ans=0.1
2023-11-21 05:25:22,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1373133.3333333333, ans=0.0
2023-11-21 05:25:23,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1373200.0, ans=0.0
2023-11-21 05:25:23,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1373200.0, ans=0.125
2023-11-21 05:25:33,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1373200.0, ans=0.125
2023-11-21 05:25:40,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1373266.6666666667, ans=0.125
2023-11-21 05:25:48,385 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206000
2023-11-21 05:25:51,195 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1600, loss[loss=0.06431, simple_loss=0.07376, pruned_loss=0.01542, audio_tagging_loss=0.01201, over 14768.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09673, pruned_loss=0.01735, audio_tagging_loss=0.009919, over 3043195.49 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:26:05,782 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.65 vs. limit=15.0
2023-11-21 05:26:07,371 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.284e+01 8.965e+01 9.783e+01 1.317e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 05:26:07,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1373400.0, ans=0.0
2023-11-21 05:26:15,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1373400.0, ans=0.125
2023-11-21 05:26:17,627 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:26:22,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1373466.6666666667, ans=0.0
2023-11-21 05:26:35,016 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.83 vs. limit=15.0
2023-11-21 05:26:39,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1373533.3333333333, ans=0.125
2023-11-21 05:26:44,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1373600.0, ans=0.025
2023-11-21 05:26:55,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206050
2023-11-21 05:26:56,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1373666.6666666667, ans=0.0
2023-11-21 05:26:57,597 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1650, loss[loss=0.05924, simple_loss=0.07531, pruned_loss=0.01054, audio_tagging_loss=0.01104, over 16586.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.0963, pruned_loss=0.01728, audio_tagging_loss=0.009989, over 3044021.71 frames. ], batch size: 63, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:27:01,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1373666.6666666667, ans=0.0
2023-11-21 05:27:05,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1373666.6666666667, ans=0.2
2023-11-21 05:27:24,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1373800.0, ans=0.0
2023-11-21 05:28:00,637 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206100
2023-11-21 05:28:03,620 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1700, loss[loss=0.09382, simple_loss=0.1179, pruned_loss=0.02744, audio_tagging_loss=0.00743, over 15583.00 frames. ], tot_loss[loss=0.0753, simple_loss=0.09645, pruned_loss=0.01719, audio_tagging_loss=0.009888, over 3047560.59 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:28:08,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.69 vs. limit=15.0
2023-11-21 05:28:18,888 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.627e+01 8.074e+01 8.560e+01 9.664e+01 1.224e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 05:28:25,514 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:28:26,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1374066.6666666667, ans=0.125
2023-11-21 05:28:44,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1374200.0, ans=0.125
2023-11-21 05:28:46,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1374200.0, ans=0.95
2023-11-21 05:28:47,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1374200.0, ans=0.1
2023-11-21 05:28:57,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1374266.6666666667, ans=0.125
2023-11-21 05:29:06,296 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206150
2023-11-21 05:29:06,947 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.08 vs. limit=12.0
2023-11-21 05:29:08,592 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1750, loss[loss=0.08137, simple_loss=0.1058, pruned_loss=0.02013, audio_tagging_loss=0.008341, over 14293.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09664, pruned_loss=0.01716, audio_tagging_loss=0.009761, over 3043704.97 frames. ], batch size: 54, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:29:38,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1374466.6666666667, ans=0.035
2023-11-21 05:29:52,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1374533.3333333333, ans=0.1
2023-11-21 05:29:54,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1374533.3333333333, ans=0.0
2023-11-21 05:29:58,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1374533.3333333333, ans=0.125
2023-11-21 05:30:10,799 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206200
2023-11-21 05:30:13,643 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1800, loss[loss=0.07974, simple_loss=0.1115, pruned_loss=0.01672, audio_tagging_loss=0.007258, over 15595.00 frames. ], tot_loss[loss=0.07527, simple_loss=0.09677, pruned_loss=0.01718, audio_tagging_loss=0.009704, over 3050041.19 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:30:29,704 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.350e+01 8.891e+01 9.779e+01 1.328e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 05:30:47,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1374800.0, ans=0.125
2023-11-21 05:30:47,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1374800.0, ans=0.0
2023-11-21 05:30:59,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1374866.6666666667, ans=0.0
2023-11-21 05:31:06,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1374933.3333333333, ans=0.1
2023-11-21 05:31:10,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1374933.3333333333, ans=0.125
2023-11-21 05:31:15,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1374933.3333333333, ans=0.0
2023-11-21 05:31:16,160 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206250
2023-11-21 05:31:16,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1374933.3333333333, ans=0.125
2023-11-21 05:31:18,493 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1850, loss[loss=0.06257, simple_loss=0.08057, pruned_loss=0.014, audio_tagging_loss=0.008285, over 14469.00 frames. ], tot_loss[loss=0.07541, simple_loss=0.09673, pruned_loss=0.01734, audio_tagging_loss=0.009701, over 3047091.66 frames. ], batch size: 53, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:31:25,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1375000.0, ans=0.1
2023-11-21 05:32:02,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1375200.0, ans=0.125
2023-11-21 05:32:06,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.81 vs. limit=15.0
2023-11-21 05:32:16,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1375266.6666666667, ans=0.125
2023-11-21 05:32:22,546 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206300
2023-11-21 05:32:24,923 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1900, loss[loss=0.08361, simple_loss=0.1094, pruned_loss=0.01958, audio_tagging_loss=0.009352, over 14904.00 frames. ], tot_loss[loss=0.07549, simple_loss=0.09688, pruned_loss=0.01743, audio_tagging_loss=0.009611, over 3048683.72 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:32:25,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1375333.3333333333, ans=0.2
2023-11-21 05:32:27,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1375333.3333333333, ans=0.125
2023-11-21 05:32:30,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.85 vs. limit=15.0
2023-11-21 05:32:33,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.48 vs. limit=15.0
2023-11-21 05:32:40,742 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.312e+01 7.949e+01 8.733e+01 9.558e+01 1.328e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 05:33:15,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1375533.3333333333, ans=0.0
2023-11-21 05:33:18,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1375600.0, ans=0.125
2023-11-21 05:33:23,947 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.95 vs. limit=10.0
2023-11-21 05:33:27,162 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206350
2023-11-21 05:33:29,477 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 1950, loss[loss=0.08958, simple_loss=0.1184, pruned_loss=0.02299, audio_tagging_loss=0.00738, over 15509.00 frames. ], tot_loss[loss=0.0753, simple_loss=0.09676, pruned_loss=0.01732, audio_tagging_loss=0.009599, over 3041677.70 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:33:42,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1375733.3333333333, ans=0.125
2023-11-21 05:33:56,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1375800.0, ans=0.125
2023-11-21 05:34:16,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1375866.6666666667, ans=0.0
2023-11-21 05:34:30,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206400
2023-11-21 05:34:34,507 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2000, loss[loss=0.1067, simple_loss=0.1395, pruned_loss=0.03099, audio_tagging_loss=0.006004, over 14911.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09692, pruned_loss=0.01745, audio_tagging_loss=0.009661, over 3040900.57 frames. ], batch size: 53, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:34:40,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1376000.0, ans=0.2
2023-11-21 05:34:47,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1376066.6666666667, ans=0.125
2023-11-21 05:34:50,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1376066.6666666667, ans=0.125
2023-11-21 05:34:53,482 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 8.010e+01 8.880e+01 9.739e+01 1.562e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 05:35:08,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1376133.3333333333, ans=0.125
2023-11-21 05:35:28,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1376266.6666666667, ans=0.0
2023-11-21 05:35:37,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206450
2023-11-21 05:35:40,821 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2050, loss[loss=0.08665, simple_loss=0.1103, pruned_loss=0.02375, audio_tagging_loss=0.007756, over 15572.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09716, pruned_loss=0.01758, audio_tagging_loss=0.009618, over 3032595.16 frames. ], batch size: 60, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:35:42,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=1376333.3333333333, ans=0.05
2023-11-21 05:35:45,498 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.10 vs. limit=6.0
2023-11-21 05:35:47,326 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.857e-02
2023-11-21 05:35:52,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1376400.0, ans=0.0
2023-11-21 05:35:57,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1376400.0, ans=0.2
2023-11-21 05:36:04,643 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.59 vs. limit=22.5
2023-11-21 05:36:28,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1376533.3333333333, ans=0.0
2023-11-21 05:36:42,075 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206500
2023-11-21 05:36:44,383 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2100, loss[loss=0.05989, simple_loss=0.07622, pruned_loss=0.01238, audio_tagging_loss=0.009396, over 14963.00 frames. ], tot_loss[loss=0.07644, simple_loss=0.09837, pruned_loss=0.01769, audio_tagging_loss=0.009568, over 3033019.90 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:36:52,374 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.22 vs. limit=15.0
2023-11-21 05:36:59,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1376733.3333333333, ans=0.125
2023-11-21 05:37:01,371 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.791e+01 8.008e+01 8.892e+01 9.733e+01 1.759e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 05:37:04,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1376733.3333333333, ans=0.07
2023-11-21 05:37:11,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1376800.0, ans=0.2
2023-11-21 05:37:13,166 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.06 vs. limit=22.5
2023-11-21 05:37:13,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1376800.0, ans=0.125
2023-11-21 05:37:44,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206550
2023-11-21 05:37:46,817 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2150, loss[loss=0.07313, simple_loss=0.1073, pruned_loss=0.0113, audio_tagging_loss=0.008202, over 15356.00 frames. ], tot_loss[loss=0.07596, simple_loss=0.09775, pruned_loss=0.01752, audio_tagging_loss=0.009567, over 3036304.64 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:37:56,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1377000.0, ans=0.0
2023-11-21 05:37:56,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1377000.0, ans=0.1
2023-11-21 05:37:56,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1377000.0, ans=0.0
2023-11-21 05:38:00,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1377066.6666666667, ans=0.125
2023-11-21 05:38:09,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1377066.6666666667, ans=0.0
2023-11-21 05:38:26,407 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:38:29,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1377200.0, ans=0.125
2023-11-21 05:38:32,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1377200.0, ans=0.1
2023-11-21 05:38:33,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff2.min_abs, batch_count=1377200.0, ans=0.1
2023-11-21 05:38:36,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1377266.6666666667, ans=0.125
2023-11-21 05:38:37,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1377266.6666666667, ans=0.125
2023-11-21 05:38:48,798 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206600
2023-11-21 05:38:50,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1377333.3333333333, ans=0.125
2023-11-21 05:38:51,608 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2200, loss[loss=0.09763, simple_loss=0.1304, pruned_loss=0.0245, audio_tagging_loss=0.007957, over 15094.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.09846, pruned_loss=0.01767, audio_tagging_loss=0.009623, over 3042997.63 frames. ], batch size: 54, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:38:59,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1377333.3333333333, ans=0.125
2023-11-21 05:39:06,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1377400.0, ans=0.125
2023-11-21 05:39:07,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1377400.0, ans=0.0
2023-11-21 05:39:09,749 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.508e+01 7.959e+01 8.463e+01 9.337e+01 1.181e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-21 05:39:13,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1377400.0, ans=0.125
2023-11-21 05:39:14,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1377400.0, ans=0.05
2023-11-21 05:39:16,666 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.18 vs. limit=22.5
2023-11-21 05:39:39,302 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:39:49,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1377600.0, ans=0.0
2023-11-21 05:39:54,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206650
2023-11-21 05:39:54,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1377600.0, ans=0.125
2023-11-21 05:39:57,054 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2250, loss[loss=0.0845, simple_loss=0.1113, pruned_loss=0.02091, audio_tagging_loss=0.007963, over 14806.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09828, pruned_loss=0.0176, audio_tagging_loss=0.009631, over 3046800.50 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:39:58,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1377666.6666666667, ans=0.125
2023-11-21 05:40:30,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1377800.0, ans=0.125
2023-11-21 05:40:35,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1377866.6666666667, ans=0.1
2023-11-21 05:40:50,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1377933.3333333333, ans=0.1
2023-11-21 05:40:59,131 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206700
2023-11-21 05:41:01,606 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2300, loss[loss=0.08957, simple_loss=0.1213, pruned_loss=0.01819, audio_tagging_loss=0.01072, over 15695.00 frames. ], tot_loss[loss=0.07679, simple_loss=0.09907, pruned_loss=0.01765, audio_tagging_loss=0.0096, over 3046251.59 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:41:14,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1378066.6666666667, ans=0.0
2023-11-21 05:41:20,802 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.169e+01 8.944e+01 9.700e+01 1.275e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-21 05:41:26,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1378066.6666666667, ans=0.125
2023-11-21 05:41:36,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1378133.3333333333, ans=0.1
2023-11-21 05:41:40,634 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.58 vs. limit=15.0
2023-11-21 05:41:42,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1378200.0, ans=0.0
2023-11-21 05:42:00,197 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:42:04,564 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206750
2023-11-21 05:42:07,017 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2350, loss[loss=0.05919, simple_loss=0.07411, pruned_loss=0.01269, audio_tagging_loss=0.009448, over 16314.00 frames. ], tot_loss[loss=0.07687, simple_loss=0.09905, pruned_loss=0.01762, audio_tagging_loss=0.009717, over 3047595.07 frames. ], batch size: 63, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:42:21,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1378400.0, ans=0.1
2023-11-21 05:42:42,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1378466.6666666667, ans=0.0
2023-11-21 05:42:43,238 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.47 vs. limit=6.0
2023-11-21 05:42:44,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.28 vs. limit=15.0
2023-11-21 05:42:59,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.81 vs. limit=22.5
2023-11-21 05:43:07,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1378600.0, ans=0.125
2023-11-21 05:43:08,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1378600.0, ans=0.2
2023-11-21 05:43:09,473 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206800
2023-11-21 05:43:12,381 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2400, loss[loss=0.06404, simple_loss=0.07777, pruned_loss=0.01516, audio_tagging_loss=0.009995, over 15027.00 frames. ], tot_loss[loss=0.07667, simple_loss=0.09869, pruned_loss=0.01751, audio_tagging_loss=0.009821, over 3047582.01 frames. ], batch size: 56, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:43:29,495 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.194e+01 8.214e+01 8.717e+01 9.423e+01 1.179e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 05:43:31,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.63 vs. limit=15.0
2023-11-21 05:43:33,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1378733.3333333333, ans=0.07
2023-11-21 05:43:34,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1378733.3333333333, ans=0.1
2023-11-21 05:43:35,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1378800.0, ans=0.0
2023-11-21 05:43:41,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1378800.0, ans=0.125
2023-11-21 05:44:06,629 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.63 vs. limit=15.0
2023-11-21 05:44:13,304 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206850
2023-11-21 05:44:15,712 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2450, loss[loss=0.06403, simple_loss=0.07175, pruned_loss=0.01423, audio_tagging_loss=0.01393, over 16107.00 frames. ], tot_loss[loss=0.07545, simple_loss=0.09651, pruned_loss=0.01717, audio_tagging_loss=0.01003, over 3043291.80 frames. ], batch size: 61, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:44:30,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1379066.6666666667, ans=0.125
2023-11-21 05:44:38,415 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.44 vs. limit=15.0
2023-11-21 05:44:42,451 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:45:16,883 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206900
2023-11-21 05:45:19,797 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2500, loss[loss=0.07427, simple_loss=0.1048, pruned_loss=0.01478, audio_tagging_loss=0.0071, over 14101.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09671, pruned_loss=0.0172, audio_tagging_loss=0.01008, over 3039409.44 frames. ], batch size: 55, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:45:29,648 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.41 vs. limit=15.0
2023-11-21 05:45:32,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1379400.0, ans=0.2
2023-11-21 05:45:38,423 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 8.201e+01 8.897e+01 9.730e+01 1.405e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 05:45:42,624 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.04 vs. limit=15.0
2023-11-21 05:45:48,888 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=11.12 vs. limit=12.0
2023-11-21 05:46:09,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1379533.3333333333, ans=0.2
2023-11-21 05:46:13,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1379600.0, ans=0.0
2023-11-21 05:46:13,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1379600.0, ans=0.125
2023-11-21 05:46:17,114 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.65 vs. limit=15.0
2023-11-21 05:46:22,209 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 206950
2023-11-21 05:46:25,214 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2550, loss[loss=0.07539, simple_loss=0.09702, pruned_loss=0.0165, audio_tagging_loss=0.01038, over 15024.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.09626, pruned_loss=0.01696, audio_tagging_loss=0.009998, over 3038947.77 frames. ], batch size: 54, lr: 3.85e-03, grad_scale: 32.0
2023-11-21 05:46:27,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1379666.6666666667, ans=0.0
2023-11-21 05:46:31,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1379666.6666666667, ans=0.0
2023-11-21 05:46:50,219 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.56 vs. limit=15.0
2023-11-21 05:46:55,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1379800.0, ans=0.125
2023-11-21 05:46:55,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1379800.0, ans=0.0
2023-11-21 05:46:57,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1379800.0, ans=0.2
2023-11-21 05:46:59,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1379800.0, ans=0.125
2023-11-21 05:47:21,310 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.01 vs. limit=15.0
2023-11-21 05:47:25,756 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207000
2023-11-21 05:47:28,466 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2600, loss[loss=0.09574, simple_loss=0.1326, pruned_loss=0.02152, audio_tagging_loss=0.007945, over 16264.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.09644, pruned_loss=0.01687, audio_tagging_loss=0.009844, over 3042001.83 frames. ], batch size: 57, lr: 3.85e-03, grad_scale: 16.0
2023-11-21 05:47:30,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1380000.0, ans=0.125
2023-11-21 05:47:45,661 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:47:47,824 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.166e+01 8.159e+01 8.772e+01 9.350e+01 1.184e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 05:48:08,220 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.25 vs. limit=15.0
2023-11-21 05:48:29,795 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207050
2023-11-21 05:48:32,787 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2650, loss[loss=0.06926, simple_loss=0.08717, pruned_loss=0.0163, audio_tagging_loss=0.009369, over 15843.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.0955, pruned_loss=0.01672, audio_tagging_loss=0.009774, over 3039955.19 frames. ], batch size: 60, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:48:36,032 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 05:49:10,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1380533.3333333333, ans=0.2
2023-11-21 05:49:11,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1380533.3333333333, ans=0.0
2023-11-21 05:49:27,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1380600.0, ans=0.125
2023-11-21 05:49:27,756 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.24 vs. limit=10.0
2023-11-21 05:49:34,970 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207100
2023-11-21 05:49:36,807 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.80 vs. limit=15.0
2023-11-21 05:49:37,448 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2700, loss[loss=0.06093, simple_loss=0.0713, pruned_loss=0.01433, audio_tagging_loss=0.01095, over 13977.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09549, pruned_loss=0.01681, audio_tagging_loss=0.009765, over 3045685.87 frames. ], batch size: 54, lr: 3.84e-03, grad_scale: 8.0
2023-11-21 05:49:53,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.11 vs. limit=15.0
2023-11-21 05:49:57,495 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 7.943e+01 8.665e+01 9.544e+01 1.191e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 05:50:00,607 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.03 vs. limit=22.5
2023-11-21 05:50:01,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1380800.0, ans=0.0
2023-11-21 05:50:30,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.69 vs. limit=15.0
2023-11-21 05:50:40,207 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207150
2023-11-21 05:50:41,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1381000.0, ans=0.125
2023-11-21 05:50:42,634 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2750, loss[loss=0.06915, simple_loss=0.1003, pruned_loss=0.01221, audio_tagging_loss=0.006805, over 15389.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09603, pruned_loss=0.01695, audio_tagging_loss=0.009683, over 3046681.90 frames. ], batch size: 54, lr: 3.84e-03, grad_scale: 8.0
2023-11-21 05:50:46,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1381000.0, ans=0.125
2023-11-21 05:51:29,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1381200.0, ans=0.125
2023-11-21 05:51:40,432 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:51:45,570 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207200
2023-11-21 05:51:46,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.07 vs. limit=12.0
2023-11-21 05:51:48,296 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2800, loss[loss=0.06889, simple_loss=0.08846, pruned_loss=0.01845, audio_tagging_loss=0.006204, over 14583.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09746, pruned_loss=0.01735, audio_tagging_loss=0.009546, over 3050355.65 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:51:54,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1381333.3333333333, ans=0.2
2023-11-21 05:52:09,758 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.379e+01 7.920e+01 8.497e+01 9.162e+01 1.236e+02, threshold=1.699e+02, percent-clipped=0.0
2023-11-21 05:52:11,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1381400.0, ans=0.125
2023-11-21 05:52:11,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1381400.0, ans=0.125
2023-11-21 05:52:21,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1381466.6666666667, ans=0.125
2023-11-21 05:52:24,684 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.32 vs. limit=22.5
2023-11-21 05:52:28,576 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.96 vs. limit=22.5
2023-11-21 05:52:33,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1381533.3333333333, ans=0.125
2023-11-21 05:52:48,333 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.38 vs. limit=15.0
2023-11-21 05:52:50,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1381600.0, ans=0.125
2023-11-21 05:52:51,569 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207250
2023-11-21 05:52:54,691 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2850, loss[loss=0.08806, simple_loss=0.1099, pruned_loss=0.02186, audio_tagging_loss=0.01128, over 15325.00 frames. ], tot_loss[loss=0.07567, simple_loss=0.09776, pruned_loss=0.01739, audio_tagging_loss=0.0094, over 3050633.13 frames. ], batch size: 57, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:53:11,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1381733.3333333333, ans=0.09899494936611666
2023-11-21 05:53:25,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1381800.0, ans=0.125
2023-11-21 05:53:33,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten.whitening_limit, batch_count=1381866.6666666667, ans=15.0
2023-11-21 05:53:57,034 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207300
2023-11-21 05:53:59,447 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2900, loss[loss=0.09973, simple_loss=0.122, pruned_loss=0.02784, audio_tagging_loss=0.01088, over 15439.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09637, pruned_loss=0.01711, audio_tagging_loss=0.009431, over 3049254.72 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:54:02,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1382000.0, ans=0.125
2023-11-21 05:54:06,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1382000.0, ans=0.1
2023-11-21 05:54:12,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1382066.6666666667, ans=0.125
2023-11-21 05:54:20,736 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.538e+01 7.887e+01 8.991e+01 9.698e+01 1.464e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-21 05:54:24,602 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.23 vs. limit=12.0
2023-11-21 05:54:30,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1382133.3333333333, ans=0.125
2023-11-21 05:54:48,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1382200.0, ans=0.125
2023-11-21 05:55:02,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207350
2023-11-21 05:55:03,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1382333.3333333333, ans=0.1
2023-11-21 05:55:04,466 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 2950, loss[loss=0.08568, simple_loss=0.1107, pruned_loss=0.01917, audio_tagging_loss=0.01118, over 15047.00 frames. ], tot_loss[loss=0.075, simple_loss=0.09686, pruned_loss=0.01717, audio_tagging_loss=0.009401, over 3046757.80 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:55:09,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1382333.3333333333, ans=0.125
2023-11-21 05:55:41,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1382466.6666666667, ans=0.125
2023-11-21 05:55:42,781 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.14 vs. limit=22.5
2023-11-21 05:56:02,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1382600.0, ans=0.0
2023-11-21 05:56:06,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1382600.0, ans=0.0
2023-11-21 05:56:07,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207400
2023-11-21 05:56:10,288 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3000, loss[loss=0.08137, simple_loss=0.1006, pruned_loss=0.01784, audio_tagging_loss=0.01322, over 14511.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.09773, pruned_loss=0.01736, audio_tagging_loss=0.009507, over 3048576.47 frames. ], batch size: 53, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:56:10,291 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 05:56:36,627 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7350, 5.8050, 5.8718, 5.8803], device='cuda:0')
2023-11-21 05:56:50,264 INFO [train_asr.py:1253] (0/4) Epoch 18, validation: loss=0.06024, simple_loss=0.05252, pruned_loss=0.00529, audio_tagging_loss=0.02869, over 4681554.00 frames. 
2023-11-21 05:56:50,265 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 05:57:11,728 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.230e+01 8.309e+01 8.951e+01 9.724e+01 1.389e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-21 05:57:52,808 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207450
2023-11-21 05:57:55,779 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3050, loss[loss=0.06268, simple_loss=0.07914, pruned_loss=0.01176, audio_tagging_loss=0.01135, over 15310.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09832, pruned_loss=0.01745, audio_tagging_loss=0.009493, over 3050017.94 frames. ], batch size: 57, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:58:32,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1383133.3333333333, ans=0.0
2023-11-21 05:58:34,418 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 05:58:43,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1383200.0, ans=0.125
2023-11-21 05:58:56,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1383266.6666666667, ans=0.2
2023-11-21 05:58:58,594 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207500
2023-11-21 05:59:01,625 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3100, loss[loss=0.07493, simple_loss=0.08704, pruned_loss=0.01968, audio_tagging_loss=0.01174, over 14485.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09833, pruned_loss=0.01734, audio_tagging_loss=0.00966, over 3047145.53 frames. ], batch size: 55, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 05:59:13,265 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.54 vs. limit=10.0
2023-11-21 05:59:21,230 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.138e+01 8.768e+01 9.517e+01 1.436e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 05:59:35,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1383466.6666666667, ans=0.0
2023-11-21 06:00:03,925 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207550
2023-11-21 06:00:06,332 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3150, loss[loss=0.08035, simple_loss=0.1073, pruned_loss=0.01789, audio_tagging_loss=0.008807, over 16823.00 frames. ], tot_loss[loss=0.07606, simple_loss=0.09789, pruned_loss=0.01734, audio_tagging_loss=0.009777, over 3047588.64 frames. ], batch size: 62, lr: 3.84e-03, grad_scale: 16.0
2023-11-21 06:00:10,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1383666.6666666667, ans=0.2
2023-11-21 06:00:15,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1383666.6666666667, ans=0.05
2023-11-21 06:00:36,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1383800.0, ans=0.1
2023-11-21 06:01:00,946 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.75 vs. limit=22.5
2023-11-21 06:01:07,818 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207600
2023-11-21 06:01:08,317 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.35 vs. limit=15.0
2023-11-21 06:01:10,786 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3200, loss[loss=0.06771, simple_loss=0.08994, pruned_loss=0.01233, audio_tagging_loss=0.0104, over 14948.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.09815, pruned_loss=0.01743, audio_tagging_loss=0.009915, over 3044899.80 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:01:18,887 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.66 vs. limit=15.0
2023-11-21 06:01:22,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1384000.0, ans=0.125
2023-11-21 06:01:32,251 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.077e+01 8.904e+01 9.561e+01 1.400e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 06:01:32,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1384066.6666666667, ans=0.0
2023-11-21 06:01:44,020 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.76 vs. limit=15.0
2023-11-21 06:01:47,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1384133.3333333333, ans=0.0
2023-11-21 06:01:53,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1384200.0, ans=0.0
2023-11-21 06:01:56,131 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:02:01,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1384266.6666666667, ans=0.0
2023-11-21 06:02:13,611 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207650
2023-11-21 06:02:15,257 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.89 vs. limit=15.0
2023-11-21 06:02:15,885 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3250, loss[loss=0.07514, simple_loss=0.09946, pruned_loss=0.01546, audio_tagging_loss=0.009948, over 14948.00 frames. ], tot_loss[loss=0.07593, simple_loss=0.09739, pruned_loss=0.01724, audio_tagging_loss=0.009986, over 3043677.00 frames. ], batch size: 57, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:02:18,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1384333.3333333333, ans=0.125
2023-11-21 06:02:22,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1384333.3333333333, ans=0.0
2023-11-21 06:02:27,186 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=17.17 vs. limit=22.5
2023-11-21 06:02:28,671 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.22 vs. limit=5.0
2023-11-21 06:02:33,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1384400.0, ans=0.0
2023-11-21 06:02:49,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1384466.6666666667, ans=0.1
2023-11-21 06:02:50,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1384466.6666666667, ans=0.125
2023-11-21 06:03:08,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1384600.0, ans=0.125
2023-11-21 06:03:12,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1384600.0, ans=0.0
2023-11-21 06:03:13,611 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.39 vs. limit=15.0
2023-11-21 06:03:17,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207700
2023-11-21 06:03:20,328 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3300, loss[loss=0.08481, simple_loss=0.1198, pruned_loss=0.01743, audio_tagging_loss=0.007493, over 16385.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.09691, pruned_loss=0.01718, audio_tagging_loss=0.01002, over 3044096.70 frames. ], batch size: 59, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:03:39,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1384733.3333333333, ans=0.0
2023-11-21 06:03:40,719 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.701e+01 8.200e+01 8.829e+01 9.665e+01 1.279e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 06:03:48,225 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.28 vs. limit=15.0
2023-11-21 06:03:52,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1384800.0, ans=0.125
2023-11-21 06:03:53,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1384800.0, ans=0.0
2023-11-21 06:04:05,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1384866.6666666667, ans=0.2
2023-11-21 06:04:21,600 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207750
2023-11-21 06:04:24,070 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3350, loss[loss=0.0739, simple_loss=0.09294, pruned_loss=0.01736, audio_tagging_loss=0.01008, over 14860.00 frames. ], tot_loss[loss=0.07499, simple_loss=0.09612, pruned_loss=0.017, audio_tagging_loss=0.009928, over 3036917.40 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:04:56,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1385133.3333333333, ans=0.1
2023-11-21 06:05:05,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1385200.0, ans=0.125
2023-11-21 06:05:22,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1385266.6666666667, ans=0.125
2023-11-21 06:05:27,902 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207800
2023-11-21 06:05:30,665 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3400, loss[loss=0.09357, simple_loss=0.1288, pruned_loss=0.02235, audio_tagging_loss=0.00684, over 16385.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09664, pruned_loss=0.0172, audio_tagging_loss=0.009812, over 3043846.64 frames. ], batch size: 58, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:05:40,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1385333.3333333333, ans=0.1
2023-11-21 06:05:50,874 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.555e+01 8.344e+01 9.058e+01 9.925e+01 1.172e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-21 06:05:56,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1385466.6666666667, ans=0.125
2023-11-21 06:05:56,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1385466.6666666667, ans=0.2
2023-11-21 06:06:12,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1385533.3333333333, ans=0.125
2023-11-21 06:06:13,468 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:06:17,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1385533.3333333333, ans=0.1
2023-11-21 06:06:30,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1385600.0, ans=0.125
2023-11-21 06:06:31,083 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.02 vs. limit=22.5
2023-11-21 06:06:33,007 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207850
2023-11-21 06:06:34,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1385666.6666666667, ans=0.0
2023-11-21 06:06:35,418 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3450, loss[loss=0.06949, simple_loss=0.0934, pruned_loss=0.01316, audio_tagging_loss=0.009625, over 16155.00 frames. ], tot_loss[loss=0.07566, simple_loss=0.09702, pruned_loss=0.01742, audio_tagging_loss=0.009724, over 3041449.87 frames. ], batch size: 60, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:06:44,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1385666.6666666667, ans=0.125
2023-11-21 06:06:54,899 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.22 vs. limit=15.0
2023-11-21 06:07:01,685 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:07:12,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1385800.0, ans=0.125
2023-11-21 06:07:26,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1385933.3333333333, ans=0.125
2023-11-21 06:07:37,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207900
2023-11-21 06:07:39,801 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3500, loss[loss=0.08793, simple_loss=0.1161, pruned_loss=0.02383, audio_tagging_loss=0.006053, over 15273.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09737, pruned_loss=0.01738, audio_tagging_loss=0.009661, over 3045176.76 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:07:42,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1386000.0, ans=0.125
2023-11-21 06:08:00,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1386066.6666666667, ans=0.1
2023-11-21 06:08:02,169 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.378e+01 8.049e+01 8.836e+01 9.881e+01 1.284e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 06:08:04,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1386066.6666666667, ans=0.125
2023-11-21 06:08:12,737 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.63 vs. limit=22.5
2023-11-21 06:08:14,487 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:08:20,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1386200.0, ans=0.125
2023-11-21 06:08:27,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=1386200.0, ans=0.2
2023-11-21 06:08:43,493 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 207950
2023-11-21 06:08:45,813 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3550, loss[loss=0.06394, simple_loss=0.07192, pruned_loss=0.01906, audio_tagging_loss=0.008916, over 15932.00 frames. ], tot_loss[loss=0.07539, simple_loss=0.09682, pruned_loss=0.0173, audio_tagging_loss=0.009675, over 3042256.51 frames. ], batch size: 62, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:09:47,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1386600.0, ans=0.2
2023-11-21 06:09:49,107 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208000
2023-11-21 06:09:50,705 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-208000.pt
2023-11-21 06:09:54,966 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3600, loss[loss=0.06138, simple_loss=0.07391, pruned_loss=0.01773, audio_tagging_loss=0.006693, over 14624.00 frames. ], tot_loss[loss=0.07567, simple_loss=0.0975, pruned_loss=0.01742, audio_tagging_loss=0.009506, over 3046486.05 frames. ], batch size: 57, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:10:06,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1386733.3333333333, ans=0.125
2023-11-21 06:10:14,772 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.348e+01 7.918e+01 8.568e+01 9.474e+01 1.185e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 06:10:32,511 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:10:56,122 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208050
2023-11-21 06:10:58,432 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3650, loss[loss=0.08166, simple_loss=0.1146, pruned_loss=0.01635, audio_tagging_loss=0.008, over 15253.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09722, pruned_loss=0.01737, audio_tagging_loss=0.00948, over 3045681.38 frames. ], batch size: 56, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:11:01,123 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:11:10,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1387066.6666666667, ans=0.125
2023-11-21 06:11:17,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1387066.6666666667, ans=0.2
2023-11-21 06:11:27,478 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.23 vs. limit=22.5
2023-11-21 06:11:28,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1387133.3333333333, ans=0.0
2023-11-21 06:11:33,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1387133.3333333333, ans=0.5
2023-11-21 06:11:42,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.39 vs. limit=15.0
2023-11-21 06:12:01,524 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208100
2023-11-21 06:12:01,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1387266.6666666667, ans=0.0
2023-11-21 06:12:03,989 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3700, loss[loss=0.1005, simple_loss=0.131, pruned_loss=0.02883, audio_tagging_loss=0.006221, over 16910.00 frames. ], tot_loss[loss=0.07585, simple_loss=0.09757, pruned_loss=0.01754, audio_tagging_loss=0.00953, over 3048788.22 frames. ], batch size: 58, lr: 3.84e-03, grad_scale: 32.0
2023-11-21 06:12:15,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1387333.3333333333, ans=0.1
2023-11-21 06:12:19,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1387400.0, ans=0.125
2023-11-21 06:12:25,065 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.253e+01 8.797e+01 9.684e+01 1.136e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 06:12:34,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1387466.6666666667, ans=0.04949747468305833
2023-11-21 06:12:46,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1387533.3333333333, ans=0.0
2023-11-21 06:12:50,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1387533.3333333333, ans=0.125
2023-11-21 06:13:07,352 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208150
2023-11-21 06:13:09,732 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3750, loss[loss=0.0643, simple_loss=0.08289, pruned_loss=0.01337, audio_tagging_loss=0.009486, over 16426.00 frames. ], tot_loss[loss=0.07561, simple_loss=0.09737, pruned_loss=0.0174, audio_tagging_loss=0.009529, over 3046466.90 frames. ], batch size: 61, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:13:29,113 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.84 vs. limit=15.0
2023-11-21 06:13:31,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1387733.3333333333, ans=0.125
2023-11-21 06:13:38,349 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.63 vs. limit=6.0
2023-11-21 06:13:55,207 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:13:58,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1387866.6666666667, ans=0.125
2023-11-21 06:14:04,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1387933.3333333333, ans=0.125
2023-11-21 06:14:08,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1387933.3333333333, ans=0.125
2023-11-21 06:14:10,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1387933.3333333333, ans=0.1
2023-11-21 06:14:11,903 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208200
2023-11-21 06:14:14,597 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3800, loss[loss=0.07216, simple_loss=0.08995, pruned_loss=0.01765, audio_tagging_loss=0.009541, over 14815.00 frames. ], tot_loss[loss=0.07577, simple_loss=0.09741, pruned_loss=0.01743, audio_tagging_loss=0.009636, over 3047983.84 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:14:29,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1388066.6666666667, ans=0.1
2023-11-21 06:14:35,464 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.099e+01 8.755e+01 9.581e+01 1.226e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 06:14:41,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1388133.3333333333, ans=0.125
2023-11-21 06:14:43,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1388133.3333333333, ans=22.5
2023-11-21 06:14:46,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1388133.3333333333, ans=0.125
2023-11-21 06:14:49,670 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.68 vs. limit=12.0
2023-11-21 06:15:00,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1388200.0, ans=0.0
2023-11-21 06:15:04,575 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.87 vs. limit=6.0
2023-11-21 06:15:09,490 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.14 vs. limit=22.5
2023-11-21 06:15:16,767 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208250
2023-11-21 06:15:16,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1388266.6666666667, ans=0.125
2023-11-21 06:15:19,743 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3850, loss[loss=0.08357, simple_loss=0.0968, pruned_loss=0.02222, audio_tagging_loss=0.01296, over 15021.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09706, pruned_loss=0.01734, audio_tagging_loss=0.00968, over 3050445.40 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:15:23,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1388333.3333333333, ans=0.125
2023-11-21 06:16:22,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208300
2023-11-21 06:16:24,669 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.47 vs. limit=15.0
2023-11-21 06:16:24,946 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3900, loss[loss=0.09817, simple_loss=0.127, pruned_loss=0.02547, audio_tagging_loss=0.009215, over 15695.00 frames. ], tot_loss[loss=0.07569, simple_loss=0.09729, pruned_loss=0.01734, audio_tagging_loss=0.009709, over 3039774.04 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:16:26,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1388666.6666666667, ans=0.125
2023-11-21 06:16:46,304 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.632e+01 8.288e+01 8.904e+01 9.716e+01 1.197e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 06:17:14,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1388866.6666666667, ans=0.04949747468305833
2023-11-21 06:17:27,510 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208350
2023-11-21 06:17:29,824 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 3950, loss[loss=0.08583, simple_loss=0.1079, pruned_loss=0.01973, audio_tagging_loss=0.01217, over 15269.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09686, pruned_loss=0.01739, audio_tagging_loss=0.009754, over 3043697.61 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:17:32,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1389000.0, ans=0.125
2023-11-21 06:17:51,521 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.11 vs. limit=22.5
2023-11-21 06:17:52,747 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.04 vs. limit=22.5
2023-11-21 06:18:13,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1389200.0, ans=0.125
2023-11-21 06:18:22,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1389266.6666666667, ans=0.1
2023-11-21 06:18:26,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.45 vs. limit=15.0
2023-11-21 06:18:31,475 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208400
2023-11-21 06:18:34,776 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4000, loss[loss=0.1129, simple_loss=0.1461, pruned_loss=0.02915, audio_tagging_loss=0.01066, over 15232.00 frames. ], tot_loss[loss=0.07671, simple_loss=0.09836, pruned_loss=0.01769, audio_tagging_loss=0.009837, over 3042158.40 frames. ], batch size: 53, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:18:52,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1389400.0, ans=0.0
2023-11-21 06:18:56,797 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.265e+01 8.108e+01 8.819e+01 9.792e+01 1.460e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 06:19:15,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1389533.3333333333, ans=0.125
2023-11-21 06:19:37,683 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208450
2023-11-21 06:19:40,000 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4050, loss[loss=0.06965, simple_loss=0.08675, pruned_loss=0.01422, audio_tagging_loss=0.01206, over 14474.00 frames. ], tot_loss[loss=0.07707, simple_loss=0.09893, pruned_loss=0.01779, audio_tagging_loss=0.009815, over 3040559.01 frames. ], batch size: 54, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:19:43,775 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:19:48,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1389666.6666666667, ans=0.125
2023-11-21 06:19:59,573 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.50 vs. limit=10.0
2023-11-21 06:20:02,689 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:20:03,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1389800.0, ans=0.125
2023-11-21 06:20:10,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1389800.0, ans=0.125
2023-11-21 06:20:29,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1389866.6666666667, ans=0.125
2023-11-21 06:20:33,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1389933.3333333333, ans=0.125
2023-11-21 06:20:34,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1389933.3333333333, ans=0.04949747468305833
2023-11-21 06:20:35,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1389933.3333333333, ans=0.2
2023-11-21 06:20:41,756 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208500
2023-11-21 06:20:44,112 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4100, loss[loss=0.07017, simple_loss=0.08444, pruned_loss=0.01736, audio_tagging_loss=0.01059, over 14130.00 frames. ], tot_loss[loss=0.07711, simple_loss=0.09889, pruned_loss=0.01779, audio_tagging_loss=0.009872, over 3039732.55 frames. ], batch size: 53, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:20:52,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1390000.0, ans=0.125
2023-11-21 06:21:06,980 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.52 vs. limit=6.0
2023-11-21 06:21:08,095 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.901e+01 8.154e+01 8.779e+01 9.471e+01 2.037e+02, threshold=1.756e+02, percent-clipped=1.0
2023-11-21 06:21:18,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=1390133.3333333333, ans=10.0
2023-11-21 06:21:29,872 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:21:46,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208550
2023-11-21 06:21:48,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1390333.3333333333, ans=0.2
2023-11-21 06:21:49,135 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4150, loss[loss=0.07431, simple_loss=0.1018, pruned_loss=0.01698, audio_tagging_loss=0.006413, over 14965.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.1, pruned_loss=0.01794, audio_tagging_loss=0.009644, over 3048700.67 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:22:34,614 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.95 vs. limit=15.0
2023-11-21 06:22:35,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1390533.3333333333, ans=0.125
2023-11-21 06:22:37,660 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:22:42,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1390600.0, ans=0.0
2023-11-21 06:22:47,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1390600.0, ans=0.125
2023-11-21 06:22:52,589 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208600
2023-11-21 06:22:55,642 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4200, loss[loss=0.07393, simple_loss=0.09507, pruned_loss=0.01603, audio_tagging_loss=0.01037, over 14701.00 frames. ], tot_loss[loss=0.07753, simple_loss=0.1001, pruned_loss=0.01795, audio_tagging_loss=0.00951, over 3048385.15 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:22:57,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1390666.6666666667, ans=0.125
2023-11-21 06:22:57,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.79 vs. limit=22.5
2023-11-21 06:22:58,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1390666.6666666667, ans=0.125
2023-11-21 06:23:17,912 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.548e+01 7.993e+01 8.905e+01 1.014e+02 1.723e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 06:23:21,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1390800.0, ans=0.0
2023-11-21 06:23:36,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1390866.6666666667, ans=0.0
2023-11-21 06:23:47,649 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:23:57,506 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208650
2023-11-21 06:23:59,945 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4250, loss[loss=0.0706, simple_loss=0.08839, pruned_loss=0.01683, audio_tagging_loss=0.00958, over 15480.00 frames. ], tot_loss[loss=0.07685, simple_loss=0.09917, pruned_loss=0.01771, audio_tagging_loss=0.009555, over 3046064.71 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:24:06,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1391000.0, ans=0.125
2023-11-21 06:24:24,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1391066.6666666667, ans=0.125
2023-11-21 06:24:32,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.76 vs. limit=15.0
2023-11-21 06:24:52,917 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.18 vs. limit=15.0
2023-11-21 06:25:02,591 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208700
2023-11-21 06:25:02,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1391266.6666666667, ans=0.125
2023-11-21 06:25:05,635 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4300, loss[loss=0.07895, simple_loss=0.09997, pruned_loss=0.02035, audio_tagging_loss=0.008615, over 14628.00 frames. ], tot_loss[loss=0.07662, simple_loss=0.09903, pruned_loss=0.01756, audio_tagging_loss=0.009541, over 3049149.67 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:25:16,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1391333.3333333333, ans=0.2
2023-11-21 06:25:30,098 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.945e+01 8.185e+01 9.001e+01 9.859e+01 1.198e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 06:25:50,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1391533.3333333333, ans=0.0
2023-11-21 06:25:50,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1391533.3333333333, ans=0.125
2023-11-21 06:26:09,719 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208750
2023-11-21 06:26:12,090 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4350, loss[loss=0.07187, simple_loss=0.09307, pruned_loss=0.01375, audio_tagging_loss=0.01159, over 16064.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09833, pruned_loss=0.01735, audio_tagging_loss=0.009618, over 3048596.98 frames. ], batch size: 59, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:26:35,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1391733.3333333333, ans=0.125
2023-11-21 06:27:11,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1391933.3333333333, ans=0.125
2023-11-21 06:27:13,104 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:27:14,173 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208800
2023-11-21 06:27:16,875 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4400, loss[loss=0.05597, simple_loss=0.0708, pruned_loss=0.01, audio_tagging_loss=0.01057, over 15379.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09756, pruned_loss=0.01722, audio_tagging_loss=0.009595, over 3050575.18 frames. ], batch size: 57, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:27:39,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1392066.6666666667, ans=0.1
2023-11-21 06:27:40,619 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.458e+01 8.121e+01 8.675e+01 9.482e+01 1.551e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 06:28:06,147 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.06 vs. limit=15.0
2023-11-21 06:28:08,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1392266.6666666667, ans=0.5
2023-11-21 06:28:18,968 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208850
2023-11-21 06:28:21,397 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4450, loss[loss=0.06051, simple_loss=0.0706, pruned_loss=0.0144, audio_tagging_loss=0.01081, over 16382.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09698, pruned_loss=0.01719, audio_tagging_loss=0.009652, over 3054443.71 frames. ], batch size: 61, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:28:22,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.47 vs. limit=15.0
2023-11-21 06:28:23,463 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.52 vs. limit=22.5
2023-11-21 06:29:01,072 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.97 vs. limit=10.0
2023-11-21 06:29:21,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1392600.0, ans=0.0
2023-11-21 06:29:23,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208900
2023-11-21 06:29:26,782 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4500, loss[loss=0.0837, simple_loss=0.1132, pruned_loss=0.01741, audio_tagging_loss=0.009665, over 14802.00 frames. ], tot_loss[loss=0.07587, simple_loss=0.09782, pruned_loss=0.01736, audio_tagging_loss=0.009591, over 3057689.99 frames. ], batch size: 54, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:29:47,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1392733.3333333333, ans=0.0
2023-11-21 06:29:49,970 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.149e+01 8.678e+01 9.442e+01 1.327e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 06:29:55,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1392800.0, ans=0.125
2023-11-21 06:29:56,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1392800.0, ans=0.2
2023-11-21 06:30:02,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1392800.0, ans=0.1
2023-11-21 06:30:16,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1392866.6666666667, ans=0.015
2023-11-21 06:30:20,706 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.56 vs. limit=15.0
2023-11-21 06:30:29,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 208950
2023-11-21 06:30:32,273 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4550, loss[loss=0.0917, simple_loss=0.1241, pruned_loss=0.022, audio_tagging_loss=0.007673, over 16517.00 frames. ], tot_loss[loss=0.07599, simple_loss=0.09818, pruned_loss=0.01735, audio_tagging_loss=0.009547, over 3054510.55 frames. ], batch size: 63, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:31:17,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1393200.0, ans=0.2
2023-11-21 06:31:20,143 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.29 vs. limit=15.0
2023-11-21 06:31:21,095 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.72 vs. limit=15.0
2023-11-21 06:31:23,160 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:31:34,190 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209000
2023-11-21 06:31:37,011 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4600, loss[loss=0.06828, simple_loss=0.08068, pruned_loss=0.01821, audio_tagging_loss=0.009731, over 14731.00 frames. ], tot_loss[loss=0.07505, simple_loss=0.09677, pruned_loss=0.01704, audio_tagging_loss=0.009628, over 3050190.51 frames. ], batch size: 56, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:31:39,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1393333.3333333333, ans=0.125
2023-11-21 06:31:41,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1393333.3333333333, ans=0.125
2023-11-21 06:31:43,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1393333.3333333333, ans=0.1
2023-11-21 06:31:49,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1393400.0, ans=0.125
2023-11-21 06:32:01,351 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.053e+01 8.667e+01 9.436e+01 2.145e+02, threshold=1.733e+02, percent-clipped=1.0
2023-11-21 06:32:26,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1393533.3333333333, ans=0.125
2023-11-21 06:32:38,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1393600.0, ans=0.0
2023-11-21 06:32:39,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209050
2023-11-21 06:32:41,481 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4650, loss[loss=0.09551, simple_loss=0.1229, pruned_loss=0.02381, audio_tagging_loss=0.01025, over 14612.00 frames. ], tot_loss[loss=0.07545, simple_loss=0.09714, pruned_loss=0.01718, audio_tagging_loss=0.009703, over 3046249.56 frames. ], batch size: 55, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:32:42,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1393666.6666666667, ans=0.1
2023-11-21 06:32:49,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1393666.6666666667, ans=0.125
2023-11-21 06:32:53,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1393666.6666666667, ans=0.125
2023-11-21 06:33:21,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.88 vs. limit=6.0
2023-11-21 06:33:26,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=1393866.6666666667, ans=15.0
2023-11-21 06:33:44,829 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209100
2023-11-21 06:33:47,226 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4700, loss[loss=0.0601, simple_loss=0.07635, pruned_loss=0.01159, audio_tagging_loss=0.01034, over 15552.00 frames. ], tot_loss[loss=0.07468, simple_loss=0.0958, pruned_loss=0.0169, audio_tagging_loss=0.00988, over 3047890.65 frames. ], batch size: 58, lr: 3.83e-03, grad_scale: 32.0
2023-11-21 06:33:51,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1394000.0, ans=0.2
2023-11-21 06:34:10,864 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.569e+01 8.149e+01 8.884e+01 9.885e+01 2.074e+02, threshold=1.777e+02, percent-clipped=1.0
2023-11-21 06:34:24,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1394200.0, ans=0.125
2023-11-21 06:34:44,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1394266.6666666667, ans=0.125
2023-11-21 06:34:48,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209150
2023-11-21 06:34:51,276 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4750, loss[loss=0.05964, simple_loss=0.07216, pruned_loss=0.01353, audio_tagging_loss=0.01003, over 15569.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09682, pruned_loss=0.01736, audio_tagging_loss=0.009859, over 3055275.99 frames. ], batch size: 61, lr: 3.83e-03, grad_scale: 16.0
2023-11-21 06:34:58,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1394333.3333333333, ans=0.125
2023-11-21 06:35:21,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1394466.6666666667, ans=0.125
2023-11-21 06:35:29,290 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.61 vs. limit=15.0
2023-11-21 06:35:40,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1394533.3333333333, ans=0.125
2023-11-21 06:35:44,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1394600.0, ans=0.125
2023-11-21 06:35:54,367 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209200
2023-11-21 06:35:56,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1394666.6666666667, ans=0.5
2023-11-21 06:35:57,081 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4800, loss[loss=0.06988, simple_loss=0.09516, pruned_loss=0.01261, audio_tagging_loss=0.009692, over 15324.00 frames. ], tot_loss[loss=0.07508, simple_loss=0.09585, pruned_loss=0.01716, audio_tagging_loss=0.01, over 3047722.97 frames. ], batch size: 56, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:36:01,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1394666.6666666667, ans=0.2
2023-11-21 06:36:01,106 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:36:15,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1394733.3333333333, ans=0.1
2023-11-21 06:36:24,239 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.086e+01 8.238e+01 8.873e+01 9.665e+01 1.246e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 06:36:43,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1394866.6666666667, ans=0.1
2023-11-21 06:36:46,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1394866.6666666667, ans=0.0
2023-11-21 06:36:55,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1394933.3333333333, ans=0.125
2023-11-21 06:37:01,810 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209250
2023-11-21 06:37:04,140 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4850, loss[loss=0.06404, simple_loss=0.08101, pruned_loss=0.01073, audio_tagging_loss=0.01281, over 15278.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09591, pruned_loss=0.01714, audio_tagging_loss=0.01015, over 3044362.03 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:37:14,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.25 vs. limit=15.0
2023-11-21 06:37:20,455 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:37:29,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1395133.3333333333, ans=0.04949747468305833
2023-11-21 06:37:34,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1395133.3333333333, ans=0.1
2023-11-21 06:37:34,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1395133.3333333333, ans=0.125
2023-11-21 06:37:36,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1395133.3333333333, ans=0.125
2023-11-21 06:37:54,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1395200.0, ans=0.04949747468305833
2023-11-21 06:38:05,809 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.82 vs. limit=15.0
2023-11-21 06:38:06,308 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209300
2023-11-21 06:38:06,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1395266.6666666667, ans=0.0
2023-11-21 06:38:08,792 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4900, loss[loss=0.07311, simple_loss=0.09277, pruned_loss=0.01669, audio_tagging_loss=0.01004, over 15439.00 frames. ], tot_loss[loss=0.0765, simple_loss=0.09807, pruned_loss=0.01751, audio_tagging_loss=0.009951, over 3050289.77 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:38:34,568 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.171e+01 8.744e+01 9.651e+01 1.565e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 06:38:39,730 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.55 vs. limit=10.0
2023-11-21 06:38:48,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1395533.3333333333, ans=0.125
2023-11-21 06:39:10,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209350
2023-11-21 06:39:13,570 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 4950, loss[loss=0.07104, simple_loss=0.08849, pruned_loss=0.01935, audio_tagging_loss=0.007446, over 15452.00 frames. ], tot_loss[loss=0.07594, simple_loss=0.09761, pruned_loss=0.01743, audio_tagging_loss=0.009703, over 3037068.31 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:39:27,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1395733.3333333333, ans=0.125
2023-11-21 06:39:45,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1395800.0, ans=0.125
2023-11-21 06:39:55,988 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.68 vs. limit=15.0
2023-11-21 06:40:17,539 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209400
2023-11-21 06:40:20,268 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5000, loss[loss=0.08107, simple_loss=0.1059, pruned_loss=0.01836, audio_tagging_loss=0.009757, over 14505.00 frames. ], tot_loss[loss=0.07545, simple_loss=0.09712, pruned_loss=0.01732, audio_tagging_loss=0.009567, over 3040273.45 frames. ], batch size: 54, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:40:38,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1396066.6666666667, ans=0.125
2023-11-21 06:40:47,079 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.108e+01 8.729e+01 9.428e+01 1.080e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 06:40:49,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1396133.3333333333, ans=0.125
2023-11-21 06:40:52,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1396133.3333333333, ans=0.125
2023-11-21 06:40:52,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1396133.3333333333, ans=0.2
2023-11-21 06:40:59,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1396200.0, ans=0.0
2023-11-21 06:41:23,014 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209450
2023-11-21 06:41:25,420 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5050, loss[loss=0.06571, simple_loss=0.08006, pruned_loss=0.01643, audio_tagging_loss=0.009246, over 16295.00 frames. ], tot_loss[loss=0.07525, simple_loss=0.09687, pruned_loss=0.01733, audio_tagging_loss=0.009484, over 3041765.30 frames. ], batch size: 62, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:41:29,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1396333.3333333333, ans=0.0
2023-11-21 06:41:52,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1396466.6666666667, ans=0.125
2023-11-21 06:42:10,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1396533.3333333333, ans=0.125
2023-11-21 06:42:27,060 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209500
2023-11-21 06:42:29,388 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5100, loss[loss=0.1013, simple_loss=0.1378, pruned_loss=0.02571, audio_tagging_loss=0.006666, over 15532.00 frames. ], tot_loss[loss=0.07547, simple_loss=0.09723, pruned_loss=0.0173, audio_tagging_loss=0.009561, over 3045079.31 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:42:29,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1396666.6666666667, ans=0.015
2023-11-21 06:42:57,149 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.138e+01 8.643e+01 9.208e+01 2.160e+02, threshold=1.729e+02, percent-clipped=1.0
2023-11-21 06:43:05,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1396800.0, ans=0.125
2023-11-21 06:43:15,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1396866.6666666667, ans=0.125
2023-11-21 06:43:17,110 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.79 vs. limit=15.0
2023-11-21 06:43:19,140 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:43:32,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209550
2023-11-21 06:43:35,659 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5150, loss[loss=0.1061, simple_loss=0.1451, pruned_loss=0.02564, audio_tagging_loss=0.007891, over 15377.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.09779, pruned_loss=0.01746, audio_tagging_loss=0.009543, over 3044452.15 frames. ], batch size: 56, lr: 3.82e-03, grad_scale: 8.0
2023-11-21 06:43:42,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1397000.0, ans=0.125
2023-11-21 06:43:46,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1397000.0, ans=0.2
2023-11-21 06:43:48,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1397066.6666666667, ans=0.125
2023-11-21 06:44:09,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1397133.3333333333, ans=0.2
2023-11-21 06:44:37,984 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209600
2023-11-21 06:44:40,765 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5200, loss[loss=0.07556, simple_loss=0.1037, pruned_loss=0.01505, audio_tagging_loss=0.00868, over 15183.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09784, pruned_loss=0.01741, audio_tagging_loss=0.009447, over 3044311.49 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:45:07,614 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.125e+01 8.190e+01 8.762e+01 9.315e+01 1.275e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 06:45:19,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1397533.3333333333, ans=0.125
2023-11-21 06:45:36,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1397600.0, ans=0.2
2023-11-21 06:45:38,586 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.86 vs. limit=22.5
2023-11-21 06:45:39,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1397600.0, ans=0.125
2023-11-21 06:45:42,972 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209650
2023-11-21 06:45:45,302 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5250, loss[loss=0.0978, simple_loss=0.13, pruned_loss=0.02594, audio_tagging_loss=0.006868, over 15372.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09743, pruned_loss=0.01742, audio_tagging_loss=0.00956, over 3046186.56 frames. ], batch size: 60, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:45:51,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1397666.6666666667, ans=0.1
2023-11-21 06:45:54,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1397666.6666666667, ans=0.0
2023-11-21 06:45:58,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1397733.3333333333, ans=0.0
2023-11-21 06:46:11,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=1397800.0, ans=10.0
2023-11-21 06:46:26,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1397866.6666666667, ans=0.125
2023-11-21 06:46:47,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209700
2023-11-21 06:46:50,778 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5300, loss[loss=0.05024, simple_loss=0.06924, pruned_loss=0.00721, audio_tagging_loss=0.008409, over 15238.00 frames. ], tot_loss[loss=0.07528, simple_loss=0.0969, pruned_loss=0.0173, audio_tagging_loss=0.009531, over 3046246.65 frames. ], batch size: 56, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:46:54,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1398000.0, ans=0.07
2023-11-21 06:46:55,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1398000.0, ans=0.0
2023-11-21 06:47:12,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1398066.6666666667, ans=0.0
2023-11-21 06:47:17,460 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.345e+01 8.152e+01 8.842e+01 9.695e+01 1.214e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 06:47:52,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209750
2023-11-21 06:47:55,001 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5350, loss[loss=0.06401, simple_loss=0.07943, pruned_loss=0.0134, audio_tagging_loss=0.01089, over 14880.00 frames. ], tot_loss[loss=0.07576, simple_loss=0.09763, pruned_loss=0.01746, audio_tagging_loss=0.009477, over 3044126.00 frames. ], batch size: 57, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:48:01,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1398333.3333333333, ans=0.125
2023-11-21 06:48:10,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1398400.0, ans=0.125
2023-11-21 06:48:13,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1398400.0, ans=0.125
2023-11-21 06:48:13,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1398400.0, ans=0.125
2023-11-21 06:48:35,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1398533.3333333333, ans=0.09899494936611666
2023-11-21 06:48:38,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1398533.3333333333, ans=0.0
2023-11-21 06:48:46,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1398600.0, ans=0.1
2023-11-21 06:48:49,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1398600.0, ans=0.0
2023-11-21 06:48:57,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209800
2023-11-21 06:49:00,610 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5400, loss[loss=0.09236, simple_loss=0.1325, pruned_loss=0.01936, audio_tagging_loss=0.006736, over 16452.00 frames. ], tot_loss[loss=0.07617, simple_loss=0.09827, pruned_loss=0.01759, audio_tagging_loss=0.009445, over 3048723.32 frames. ], batch size: 61, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:49:02,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1398666.6666666667, ans=0.1
2023-11-21 06:49:07,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=1398666.6666666667, ans=0.5
2023-11-21 06:49:28,200 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.049e+01 8.809e+01 9.277e+01 1.117e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 06:49:30,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1398800.0, ans=0.125
2023-11-21 06:50:02,644 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209850
2023-11-21 06:50:04,910 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5450, loss[loss=0.07377, simple_loss=0.08406, pruned_loss=0.0198, audio_tagging_loss=0.01194, over 14796.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09839, pruned_loss=0.01772, audio_tagging_loss=0.00948, over 3051138.12 frames. ], batch size: 55, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:50:15,415 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.24 vs. limit=10.0
2023-11-21 06:50:52,026 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:51:07,104 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.030e-01
2023-11-21 06:51:08,076 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209900
2023-11-21 06:51:10,488 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5500, loss[loss=0.07952, simple_loss=0.1001, pruned_loss=0.01794, audio_tagging_loss=0.01153, over 15423.00 frames. ], tot_loss[loss=0.07664, simple_loss=0.0989, pruned_loss=0.01767, audio_tagging_loss=0.009519, over 3056332.34 frames. ], batch size: 56, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:51:19,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1399333.3333333333, ans=0.0
2023-11-21 06:51:34,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1399466.6666666667, ans=0.0
2023-11-21 06:51:37,451 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.332e+01 7.977e+01 8.798e+01 9.625e+01 1.160e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 06:51:45,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1399466.6666666667, ans=0.0
2023-11-21 06:51:45,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1399466.6666666667, ans=0.2
2023-11-21 06:52:01,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1399600.0, ans=0.125
2023-11-21 06:52:01,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1399600.0, ans=0.0
2023-11-21 06:52:10,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1399600.0, ans=0.0
2023-11-21 06:52:11,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 209950
2023-11-21 06:52:13,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1399666.6666666667, ans=0.125
2023-11-21 06:52:14,297 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5550, loss[loss=0.08027, simple_loss=0.108, pruned_loss=0.01904, audio_tagging_loss=0.00725, over 15610.00 frames. ], tot_loss[loss=0.07653, simple_loss=0.09887, pruned_loss=0.01746, audio_tagging_loss=0.009629, over 3057449.78 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 16.0
2023-11-21 06:52:14,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1399666.6666666667, ans=0.2
2023-11-21 06:52:18,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1399666.6666666667, ans=0.125
2023-11-21 06:52:21,563 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.18 vs. limit=10.0
2023-11-21 06:52:30,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1399733.3333333333, ans=0.125
2023-11-21 06:52:45,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1399800.0, ans=0.125
2023-11-21 06:53:06,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1399933.3333333333, ans=0.0
2023-11-21 06:53:17,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210000
2023-11-21 06:53:19,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=1400000.0, ans=0.05
2023-11-21 06:53:20,219 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5600, loss[loss=0.09049, simple_loss=0.1265, pruned_loss=0.0188, audio_tagging_loss=0.008454, over 15108.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.09802, pruned_loss=0.01708, audio_tagging_loss=0.009804, over 3056575.24 frames. ], batch size: 56, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:53:47,686 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.103e+01 7.991e+01 8.686e+01 9.530e+01 1.132e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 06:54:07,404 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 06:54:12,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1400266.6666666667, ans=0.125
2023-11-21 06:54:23,411 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210050
2023-11-21 06:54:25,747 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5650, loss[loss=0.05636, simple_loss=0.06446, pruned_loss=0.01039, audio_tagging_loss=0.01374, over 16370.00 frames. ], tot_loss[loss=0.07588, simple_loss=0.09753, pruned_loss=0.01724, audio_tagging_loss=0.009884, over 3052984.16 frames. ], batch size: 63, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:54:32,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1400333.3333333333, ans=0.125
2023-11-21 06:54:54,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1400466.6666666667, ans=0.125
2023-11-21 06:55:03,285 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.77 vs. limit=15.0
2023-11-21 06:55:04,576 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.11 vs. limit=15.0
2023-11-21 06:55:26,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210100
2023-11-21 06:55:28,793 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5700, loss[loss=0.08564, simple_loss=0.1097, pruned_loss=0.02229, audio_tagging_loss=0.008519, over 15332.00 frames. ], tot_loss[loss=0.07634, simple_loss=0.09792, pruned_loss=0.01749, audio_tagging_loss=0.00989, over 3052870.76 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:55:37,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1400666.6666666667, ans=0.2
2023-11-21 06:55:38,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1400666.6666666667, ans=0.125
2023-11-21 06:55:57,066 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.246e+01 8.120e+01 8.868e+01 9.513e+01 1.273e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 06:56:12,425 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.80 vs. limit=15.0
2023-11-21 06:56:15,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=1400866.6666666667, ans=0.05
2023-11-21 06:56:21,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1400933.3333333333, ans=0.1
2023-11-21 06:56:29,986 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.48 vs. limit=15.0
2023-11-21 06:56:30,823 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210150
2023-11-21 06:56:33,153 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.66 vs. limit=6.0
2023-11-21 06:56:33,736 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5750, loss[loss=0.06258, simple_loss=0.07807, pruned_loss=0.01481, audio_tagging_loss=0.008727, over 14593.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09772, pruned_loss=0.0174, audio_tagging_loss=0.009768, over 3045940.23 frames. ], batch size: 58, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:56:41,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1401000.0, ans=0.0
2023-11-21 06:56:51,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1401066.6666666667, ans=0.1
2023-11-21 06:57:12,876 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.79 vs. limit=15.0
2023-11-21 06:57:22,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=1401200.0, ans=0.02
2023-11-21 06:57:36,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210200
2023-11-21 06:57:39,677 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5800, loss[loss=0.09271, simple_loss=0.1175, pruned_loss=0.02306, audio_tagging_loss=0.01091, over 15808.00 frames. ], tot_loss[loss=0.0766, simple_loss=0.09841, pruned_loss=0.01777, audio_tagging_loss=0.00962, over 3045223.64 frames. ], batch size: 59, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:57:39,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1401333.3333333333, ans=0.125
2023-11-21 06:57:52,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1401400.0, ans=0.125
2023-11-21 06:57:54,785 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 06:58:05,606 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.011e+01 8.333e+01 8.998e+01 9.692e+01 1.329e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 06:58:08,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1401466.6666666667, ans=0.0
2023-11-21 06:58:09,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1401466.6666666667, ans=0.125
2023-11-21 06:58:15,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1401466.6666666667, ans=0.0
2023-11-21 06:58:17,350 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.29 vs. limit=15.0
2023-11-21 06:58:24,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.56 vs. limit=22.5
2023-11-21 06:58:31,353 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.11 vs. limit=15.0
2023-11-21 06:58:33,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1401600.0, ans=0.0
2023-11-21 06:58:41,991 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210250
2023-11-21 06:58:43,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1401666.6666666667, ans=0.07
2023-11-21 06:58:44,339 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5850, loss[loss=0.09506, simple_loss=0.1253, pruned_loss=0.02459, audio_tagging_loss=0.007822, over 14576.00 frames. ], tot_loss[loss=0.07704, simple_loss=0.09917, pruned_loss=0.01795, audio_tagging_loss=0.009506, over 3043838.97 frames. ], batch size: 53, lr: 3.82e-03, grad_scale: 32.0
2023-11-21 06:58:54,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1401666.6666666667, ans=0.125
2023-11-21 06:58:56,341 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.70 vs. limit=15.0
2023-11-21 06:59:03,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1401733.3333333333, ans=0.0
2023-11-21 06:59:07,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1401733.3333333333, ans=0.07
2023-11-21 06:59:14,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1401800.0, ans=0.125
2023-11-21 06:59:17,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1401800.0, ans=0.0
2023-11-21 06:59:18,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1401800.0, ans=0.0
2023-11-21 06:59:27,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1401866.6666666667, ans=0.125
2023-11-21 06:59:45,573 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210300
2023-11-21 06:59:47,886 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5900, loss[loss=0.08148, simple_loss=0.1046, pruned_loss=0.01639, audio_tagging_loss=0.0128, over 15335.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09848, pruned_loss=0.01755, audio_tagging_loss=0.009582, over 3049582.34 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:00:13,140 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.35 vs. limit=15.0
2023-11-21 07:00:17,475 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.132e+01 8.306e+01 8.999e+01 9.965e+01 1.195e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 07:00:21,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1402133.3333333333, ans=0.0
2023-11-21 07:00:33,234 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.69 vs. limit=15.0
2023-11-21 07:00:50,788 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.46 vs. limit=22.5
2023-11-21 07:00:51,564 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210350
2023-11-21 07:00:54,587 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 5950, loss[loss=0.07871, simple_loss=0.09629, pruned_loss=0.01987, audio_tagging_loss=0.0107, over 15424.00 frames. ], tot_loss[loss=0.07629, simple_loss=0.09841, pruned_loss=0.01755, audio_tagging_loss=0.009537, over 3054945.97 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:00:56,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1402333.3333333333, ans=0.0
2023-11-21 07:00:57,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.13 vs. limit=22.5
2023-11-21 07:01:05,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1402400.0, ans=0.125
2023-11-21 07:01:15,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1402400.0, ans=0.125
2023-11-21 07:01:43,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1402533.3333333333, ans=0.1
2023-11-21 07:01:55,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210400
2023-11-21 07:01:58,243 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6000, loss[loss=0.06648, simple_loss=0.08214, pruned_loss=0.01466, audio_tagging_loss=0.01074, over 13882.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.0964, pruned_loss=0.01715, audio_tagging_loss=0.009616, over 3047096.25 frames. ], batch size: 53, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:01:58,246 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 07:02:26,616 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4627, 3.7409, 2.5483, 3.6460], device='cuda:0')
2023-11-21 07:02:40,603 INFO [train_asr.py:1253] (0/4) Epoch 18, validation: loss=0.0604, simple_loss=0.05257, pruned_loss=0.00537, audio_tagging_loss=0.02874, over 4681554.00 frames. 
2023-11-21 07:02:40,604 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 07:03:09,956 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 7.991e+01 8.851e+01 9.572e+01 1.607e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 07:03:21,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1402866.6666666667, ans=0.125
2023-11-21 07:03:27,306 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 07:03:31,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1402933.3333333333, ans=0.0
2023-11-21 07:03:33,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1402933.3333333333, ans=0.125
2023-11-21 07:03:44,054 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210450
2023-11-21 07:03:47,037 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6050, loss[loss=0.07129, simple_loss=0.1001, pruned_loss=0.01156, audio_tagging_loss=0.0097, over 13909.00 frames. ], tot_loss[loss=0.07477, simple_loss=0.0962, pruned_loss=0.01706, audio_tagging_loss=0.009603, over 3048269.08 frames. ], batch size: 52, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:04:13,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1403133.3333333333, ans=0.125
2023-11-21 07:04:15,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1403133.3333333333, ans=0.0
2023-11-21 07:04:16,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1403133.3333333333, ans=0.0
2023-11-21 07:04:22,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1403133.3333333333, ans=0.0
2023-11-21 07:04:27,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1403200.0, ans=0.125
2023-11-21 07:04:40,637 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.47 vs. limit=15.0
2023-11-21 07:04:48,589 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210500
2023-11-21 07:04:50,999 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6100, loss[loss=0.07953, simple_loss=0.1068, pruned_loss=0.01689, audio_tagging_loss=0.009213, over 15227.00 frames. ], tot_loss[loss=0.0748, simple_loss=0.09653, pruned_loss=0.01699, audio_tagging_loss=0.009541, over 3049027.26 frames. ], batch size: 55, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:04:53,761 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:04:55,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1403333.3333333333, ans=0.125
2023-11-21 07:05:06,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1403400.0, ans=0.2
2023-11-21 07:05:20,538 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.632e+01 8.223e+01 8.598e+01 9.128e+01 4.094e+02, threshold=1.720e+02, percent-clipped=1.0
2023-11-21 07:05:31,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1403533.3333333333, ans=0.125
2023-11-21 07:05:35,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1403533.3333333333, ans=0.0
2023-11-21 07:05:35,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.81 vs. limit=6.0
2023-11-21 07:05:38,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1403533.3333333333, ans=0.2
2023-11-21 07:05:53,444 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210550
2023-11-21 07:05:53,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1403600.0, ans=0.2
2023-11-21 07:05:55,830 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6150, loss[loss=0.07873, simple_loss=0.1033, pruned_loss=0.01874, audio_tagging_loss=0.008318, over 14659.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09599, pruned_loss=0.01688, audio_tagging_loss=0.009622, over 3044409.62 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:06:04,682 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:06:25,231 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.33 vs. limit=15.0
2023-11-21 07:06:28,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1403800.0, ans=0.125
2023-11-21 07:06:34,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1403866.6666666667, ans=0.125
2023-11-21 07:06:39,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1403866.6666666667, ans=0.0
2023-11-21 07:06:41,218 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:06:51,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1403933.3333333333, ans=0.0
2023-11-21 07:06:57,688 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.39 vs. limit=15.0
2023-11-21 07:07:00,658 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210600
2023-11-21 07:07:02,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1404000.0, ans=0.2
2023-11-21 07:07:03,403 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6200, loss[loss=0.05512, simple_loss=0.06641, pruned_loss=0.01196, audio_tagging_loss=0.009953, over 15443.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09596, pruned_loss=0.01674, audio_tagging_loss=0.00969, over 3047982.82 frames. ], batch size: 60, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:07:26,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1404066.6666666667, ans=0.0
2023-11-21 07:07:31,492 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.071e+01 8.570e+01 9.490e+01 1.156e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 07:07:42,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1404200.0, ans=0.0
2023-11-21 07:08:06,186 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210650
2023-11-21 07:08:08,593 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6250, loss[loss=0.0649, simple_loss=0.08075, pruned_loss=0.01488, audio_tagging_loss=0.009644, over 14714.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09627, pruned_loss=0.01679, audio_tagging_loss=0.009834, over 3053045.26 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:08:12,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1404333.3333333333, ans=0.125
2023-11-21 07:08:24,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1404400.0, ans=0.125
2023-11-21 07:08:30,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1404400.0, ans=0.125
2023-11-21 07:08:30,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1404400.0, ans=0.0
2023-11-21 07:08:40,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1404466.6666666667, ans=0.125
2023-11-21 07:08:42,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1404466.6666666667, ans=0.0
2023-11-21 07:08:57,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1404533.3333333333, ans=0.07
2023-11-21 07:09:10,159 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210700
2023-11-21 07:09:12,479 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6300, loss[loss=0.07366, simple_loss=0.1089, pruned_loss=0.01397, audio_tagging_loss=0.005259, over 15190.00 frames. ], tot_loss[loss=0.07492, simple_loss=0.09637, pruned_loss=0.0169, audio_tagging_loss=0.009845, over 3049429.09 frames. ], batch size: 55, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:09:19,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1404666.6666666667, ans=0.125
2023-11-21 07:09:21,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1404666.6666666667, ans=0.125
2023-11-21 07:09:41,477 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.208e+01 8.088e+01 8.800e+01 9.638e+01 1.138e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 07:09:53,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1404866.6666666667, ans=0.125
2023-11-21 07:10:04,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1404933.3333333333, ans=0.0
2023-11-21 07:10:10,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1404933.3333333333, ans=0.1
2023-11-21 07:10:10,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1404933.3333333333, ans=0.2
2023-11-21 07:10:15,417 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210750
2023-11-21 07:10:18,532 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6350, loss[loss=0.08231, simple_loss=0.1021, pruned_loss=0.01822, audio_tagging_loss=0.01302, over 13948.00 frames. ], tot_loss[loss=0.075, simple_loss=0.09619, pruned_loss=0.01698, audio_tagging_loss=0.009931, over 3053087.93 frames. ], batch size: 52, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:10:41,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.31 vs. limit=6.0
2023-11-21 07:11:05,864 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.57 vs. limit=15.0
2023-11-21 07:11:20,627 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210800
2023-11-21 07:11:23,320 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6400, loss[loss=0.08979, simple_loss=0.1078, pruned_loss=0.02385, audio_tagging_loss=0.01203, over 15813.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.09572, pruned_loss=0.01695, audio_tagging_loss=0.01003, over 3050141.37 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:11:31,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1405333.3333333333, ans=22.5
2023-11-21 07:11:34,266 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0
2023-11-21 07:11:55,151 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.546e+01 8.103e+01 8.679e+01 9.633e+01 1.173e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 07:11:55,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1405466.6666666667, ans=0.2
2023-11-21 07:12:04,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1405533.3333333333, ans=0.125
2023-11-21 07:12:25,958 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210850
2023-11-21 07:12:28,232 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6450, loss[loss=0.08903, simple_loss=0.1137, pruned_loss=0.02113, audio_tagging_loss=0.01105, over 15489.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09496, pruned_loss=0.01666, audio_tagging_loss=0.01012, over 3046948.59 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:12:34,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1405666.6666666667, ans=0.0
2023-11-21 07:12:46,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1405733.3333333333, ans=0.125
2023-11-21 07:12:50,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1405733.3333333333, ans=0.125
2023-11-21 07:13:32,053 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210900
2023-11-21 07:13:34,485 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6500, loss[loss=0.07978, simple_loss=0.1042, pruned_loss=0.0198, audio_tagging_loss=0.007903, over 15340.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.09543, pruned_loss=0.01692, audio_tagging_loss=0.009942, over 3047967.80 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:14:05,521 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.017e+01 8.785e+01 9.599e+01 1.242e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 07:14:31,934 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.73 vs. limit=10.0
2023-11-21 07:14:37,179 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 210950
2023-11-21 07:14:39,588 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6550, loss[loss=0.04759, simple_loss=0.05655, pruned_loss=0.007916, audio_tagging_loss=0.0114, over 16319.00 frames. ], tot_loss[loss=0.0746, simple_loss=0.09562, pruned_loss=0.01694, audio_tagging_loss=0.009856, over 3052660.64 frames. ], batch size: 66, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:14:45,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1406333.3333333333, ans=0.1
2023-11-21 07:14:52,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1406400.0, ans=0.0
2023-11-21 07:14:56,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1406400.0, ans=0.2
2023-11-21 07:15:20,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1406533.3333333333, ans=0.2
2023-11-21 07:15:21,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.90 vs. limit=15.0
2023-11-21 07:15:35,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1406600.0, ans=0.0
2023-11-21 07:15:41,935 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211000
2023-11-21 07:15:44,653 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6600, loss[loss=0.09586, simple_loss=0.1148, pruned_loss=0.02802, audio_tagging_loss=0.01046, over 15704.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09717, pruned_loss=0.01732, audio_tagging_loss=0.009643, over 3053403.57 frames. ], batch size: 57, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:15:49,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1406666.6666666667, ans=0.125
2023-11-21 07:15:52,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1406666.6666666667, ans=0.125
2023-11-21 07:16:03,865 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.92 vs. limit=10.0
2023-11-21 07:16:16,219 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.708e+01 8.176e+01 8.712e+01 9.397e+01 1.234e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 07:16:23,239 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.96 vs. limit=15.0
2023-11-21 07:16:35,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1406933.3333333333, ans=0.1
2023-11-21 07:16:47,264 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211050
2023-11-21 07:16:48,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1407000.0, ans=0.0
2023-11-21 07:16:50,703 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6650, loss[loss=0.06657, simple_loss=0.0858, pruned_loss=0.01441, audio_tagging_loss=0.009261, over 14578.00 frames. ], tot_loss[loss=0.07523, simple_loss=0.09698, pruned_loss=0.01715, audio_tagging_loss=0.009597, over 3057508.95 frames. ], batch size: 54, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:17:25,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1407133.3333333333, ans=0.0
2023-11-21 07:17:37,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1407200.0, ans=0.0
2023-11-21 07:17:38,929 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:17:53,696 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211100
2023-11-21 07:17:56,139 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6700, loss[loss=0.08131, simple_loss=0.1037, pruned_loss=0.01984, audio_tagging_loss=0.00963, over 16368.00 frames. ], tot_loss[loss=0.07495, simple_loss=0.09676, pruned_loss=0.01703, audio_tagging_loss=0.009546, over 3052948.39 frames. ], batch size: 61, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:18:06,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1407333.3333333333, ans=0.125
2023-11-21 07:18:17,253 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.54 vs. limit=15.0
2023-11-21 07:18:18,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1407400.0, ans=0.2
2023-11-21 07:18:22,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1407466.6666666667, ans=0.125
2023-11-21 07:18:26,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1407466.6666666667, ans=0.1
2023-11-21 07:18:27,197 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.113e+01 8.605e+01 9.333e+01 1.253e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 07:18:38,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1407533.3333333333, ans=0.0
2023-11-21 07:18:39,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1407533.3333333333, ans=0.1
2023-11-21 07:18:58,341 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211150
2023-11-21 07:19:00,708 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6750, loss[loss=0.06214, simple_loss=0.08524, pruned_loss=0.009803, audio_tagging_loss=0.009711, over 14828.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.09677, pruned_loss=0.01704, audio_tagging_loss=0.009657, over 3049693.25 frames. ], batch size: 56, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:19:12,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1407733.3333333333, ans=0.1
2023-11-21 07:19:42,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1407866.6666666667, ans=0.2
2023-11-21 07:19:46,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.93 vs. limit=22.5
2023-11-21 07:19:53,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.60 vs. limit=15.0
2023-11-21 07:19:59,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1407933.3333333333, ans=0.1
2023-11-21 07:20:03,476 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211200
2023-11-21 07:20:06,278 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6800, loss[loss=0.06777, simple_loss=0.08545, pruned_loss=0.017, audio_tagging_loss=0.008047, over 14082.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.09666, pruned_loss=0.01697, audio_tagging_loss=0.009579, over 3044101.64 frames. ], batch size: 55, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:20:07,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1408000.0, ans=0.2
2023-11-21 07:20:15,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1408000.0, ans=0.125
2023-11-21 07:20:25,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1408066.6666666667, ans=0.0
2023-11-21 07:20:25,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1408066.6666666667, ans=0.0
2023-11-21 07:20:37,398 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.900e+01 8.078e+01 8.657e+01 9.285e+01 1.100e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 07:20:40,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1408133.3333333333, ans=0.125
2023-11-21 07:21:03,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1408266.6666666667, ans=0.1
2023-11-21 07:21:06,174 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=4.17 vs. limit=15.0
2023-11-21 07:21:06,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1408266.6666666667, ans=0.125
2023-11-21 07:21:09,528 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211250
2023-11-21 07:21:10,863 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:21:10,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1408333.3333333333, ans=0.1
2023-11-21 07:21:11,853 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6850, loss[loss=0.07571, simple_loss=0.1016, pruned_loss=0.01479, audio_tagging_loss=0.01011, over 15580.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09669, pruned_loss=0.01699, audio_tagging_loss=0.009514, over 3045824.77 frames. ], batch size: 60, lr: 3.81e-03, grad_scale: 32.0
2023-11-21 07:21:17,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1408333.3333333333, ans=0.125
2023-11-21 07:21:31,219 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.08 vs. limit=15.0
2023-11-21 07:21:32,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1408400.0, ans=0.0
2023-11-21 07:21:51,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1408533.3333333333, ans=0.125
2023-11-21 07:21:53,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1408533.3333333333, ans=0.1
2023-11-21 07:21:55,130 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.63 vs. limit=15.0
2023-11-21 07:22:04,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1408600.0, ans=0.125
2023-11-21 07:22:04,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1408600.0, ans=0.125
2023-11-21 07:22:14,406 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211300
2023-11-21 07:22:16,915 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6900, loss[loss=0.07724, simple_loss=0.09743, pruned_loss=0.01901, audio_tagging_loss=0.009506, over 15943.00 frames. ], tot_loss[loss=0.07528, simple_loss=0.09744, pruned_loss=0.01703, audio_tagging_loss=0.00953, over 3051999.24 frames. ], batch size: 58, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:22:26,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1408666.6666666667, ans=0.2
2023-11-21 07:22:47,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1408800.0, ans=0.2
2023-11-21 07:22:50,580 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.236e+01 8.770e+01 9.429e+01 1.247e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 07:22:58,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1408866.6666666667, ans=0.125
2023-11-21 07:23:06,768 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 07:23:08,545 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.91 vs. limit=15.0
2023-11-21 07:23:18,951 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211350
2023-11-21 07:23:21,804 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 6950, loss[loss=0.07571, simple_loss=0.09975, pruned_loss=0.01742, audio_tagging_loss=0.008413, over 16170.00 frames. ], tot_loss[loss=0.07443, simple_loss=0.0962, pruned_loss=0.01672, audio_tagging_loss=0.009606, over 3047721.87 frames. ], batch size: 60, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:23:30,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1409000.0, ans=0.1
2023-11-21 07:23:33,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1409000.0, ans=0.125
2023-11-21 07:24:25,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211400
2023-11-21 07:24:27,848 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7000, loss[loss=0.06142, simple_loss=0.07477, pruned_loss=0.01238, audio_tagging_loss=0.01166, over 15310.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.0973, pruned_loss=0.01678, audio_tagging_loss=0.009671, over 3055536.10 frames. ], batch size: 59, lr: 3.81e-03, grad_scale: 16.0
2023-11-21 07:24:44,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1409400.0, ans=0.1
2023-11-21 07:24:44,762 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.34 vs. limit=15.0
2023-11-21 07:24:49,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1409400.0, ans=0.0
2023-11-21 07:24:59,166 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 7.925e+01 8.553e+01 9.217e+01 1.576e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 07:25:16,170 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.51 vs. limit=15.0
2023-11-21 07:25:19,429 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:25:30,548 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211450
2023-11-21 07:25:32,874 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7050, loss[loss=0.09366, simple_loss=0.1241, pruned_loss=0.02208, audio_tagging_loss=0.009545, over 15577.00 frames. ], tot_loss[loss=0.07453, simple_loss=0.09618, pruned_loss=0.0167, audio_tagging_loss=0.009744, over 3046170.19 frames. ], batch size: 58, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:25:33,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1409666.6666666667, ans=0.125
2023-11-21 07:25:36,084 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.01 vs. limit=15.0
2023-11-21 07:25:38,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1409666.6666666667, ans=0.125
2023-11-21 07:25:38,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1409666.6666666667, ans=0.025
2023-11-21 07:26:02,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1409800.0, ans=0.125
2023-11-21 07:26:11,963 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.78 vs. limit=6.0
2023-11-21 07:26:12,938 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.15 vs. limit=15.0
2023-11-21 07:26:34,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211500
2023-11-21 07:26:36,964 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7100, loss[loss=0.07922, simple_loss=0.09684, pruned_loss=0.01821, audio_tagging_loss=0.01258, over 15672.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09666, pruned_loss=0.01683, audio_tagging_loss=0.009816, over 3053877.31 frames. ], batch size: 60, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:26:56,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1410066.6666666667, ans=0.5
2023-11-21 07:27:10,190 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.406e+01 8.252e+01 8.974e+01 1.013e+02 1.228e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-21 07:27:13,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1410133.3333333333, ans=0.2
2023-11-21 07:27:19,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1410200.0, ans=0.2
2023-11-21 07:27:19,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.71 vs. limit=12.0
2023-11-21 07:27:41,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211550
2023-11-21 07:27:43,609 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7150, loss[loss=0.08196, simple_loss=0.1082, pruned_loss=0.01922, audio_tagging_loss=0.008634, over 14765.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09602, pruned_loss=0.01668, audio_tagging_loss=0.009854, over 3057469.25 frames. ], batch size: 55, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:27:51,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1410333.3333333333, ans=0.0
2023-11-21 07:28:06,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1410400.0, ans=0.125
2023-11-21 07:28:13,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1410466.6666666667, ans=0.125
2023-11-21 07:28:17,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1410466.6666666667, ans=0.0
2023-11-21 07:28:38,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1410600.0, ans=0.125
2023-11-21 07:28:45,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211600
2023-11-21 07:28:48,051 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7200, loss[loss=0.04568, simple_loss=0.05559, pruned_loss=0.006384, audio_tagging_loss=0.0115, over 15735.00 frames. ], tot_loss[loss=0.07378, simple_loss=0.09494, pruned_loss=0.01637, audio_tagging_loss=0.009932, over 3049761.24 frames. ], batch size: 60, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:28:49,919 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.67 vs. limit=15.0
2023-11-21 07:28:58,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1410666.6666666667, ans=0.125
2023-11-21 07:29:07,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1410733.3333333333, ans=0.125
2023-11-21 07:29:20,722 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 7.997e+01 8.820e+01 9.312e+01 1.213e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 07:29:33,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1410866.6666666667, ans=0.125
2023-11-21 07:29:50,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211650
2023-11-21 07:29:52,671 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7250, loss[loss=0.06134, simple_loss=0.05721, pruned_loss=0.01609, audio_tagging_loss=0.01665, over 15767.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09399, pruned_loss=0.01626, audio_tagging_loss=0.01007, over 3050330.83 frames. ], batch size: 60, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:30:03,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1411000.0, ans=0.125
2023-11-21 07:30:04,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1411066.6666666667, ans=0.0
2023-11-21 07:30:34,542 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.96 vs. limit=22.5
2023-11-21 07:30:40,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1411200.0, ans=0.04949747468305833
2023-11-21 07:30:55,375 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211700
2023-11-21 07:30:58,970 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7300, loss[loss=0.06429, simple_loss=0.07847, pruned_loss=0.01698, audio_tagging_loss=0.008077, over 15372.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.09404, pruned_loss=0.01634, audio_tagging_loss=0.009977, over 3051197.40 frames. ], batch size: 57, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:31:03,178 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.72 vs. limit=22.5
2023-11-21 07:31:13,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1411400.0, ans=0.125
2023-11-21 07:31:21,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1411400.0, ans=0.1
2023-11-21 07:31:30,107 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.648e+01 8.326e+01 8.844e+01 9.526e+01 1.717e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 07:31:31,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_na.min_abs, batch_count=1411466.6666666667, ans=0.02
2023-11-21 07:32:00,790 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.15 vs. limit=15.0
2023-11-21 07:32:01,348 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211750
2023-11-21 07:32:03,699 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7350, loss[loss=0.05438, simple_loss=0.06322, pruned_loss=0.009849, audio_tagging_loss=0.01292, over 15667.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09434, pruned_loss=0.01643, audio_tagging_loss=0.00987, over 3054698.52 frames. ], batch size: 59, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:32:26,889 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.85 vs. limit=15.0
2023-11-21 07:33:04,772 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211800
2023-11-21 07:33:07,579 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7400, loss[loss=0.06661, simple_loss=0.08967, pruned_loss=0.01332, audio_tagging_loss=0.008451, over 14994.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09506, pruned_loss=0.01661, audio_tagging_loss=0.009665, over 3052227.86 frames. ], batch size: 57, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:33:31,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1412066.6666666667, ans=15.0
2023-11-21 07:33:40,823 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 7.896e+01 8.684e+01 9.314e+01 1.199e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 07:34:09,725 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211850
2023-11-21 07:34:12,036 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7450, loss[loss=0.06691, simple_loss=0.08604, pruned_loss=0.01236, audio_tagging_loss=0.01153, over 15336.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.0957, pruned_loss=0.01673, audio_tagging_loss=0.009611, over 3048551.33 frames. ], batch size: 61, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:34:24,654 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.14 vs. limit=12.0
2023-11-21 07:34:39,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1412466.6666666667, ans=0.125
2023-11-21 07:34:40,357 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:34:44,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1412466.6666666667, ans=0.125
2023-11-21 07:34:59,478 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.26 vs. limit=15.0
2023-11-21 07:35:05,536 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.23 vs. limit=10.0
2023-11-21 07:35:13,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211900
2023-11-21 07:35:15,887 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7500, loss[loss=0.08603, simple_loss=0.1221, pruned_loss=0.01893, audio_tagging_loss=0.006037, over 16717.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09613, pruned_loss=0.01683, audio_tagging_loss=0.009573, over 3046812.12 frames. ], batch size: 59, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:35:16,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1412666.6666666667, ans=0.2
2023-11-21 07:35:30,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1412733.3333333333, ans=0.2
2023-11-21 07:35:36,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1412733.3333333333, ans=0.0
2023-11-21 07:35:47,746 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.101e+01 8.064e+01 8.532e+01 9.428e+01 1.202e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 07:35:49,728 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.04 vs. limit=15.0
2023-11-21 07:35:55,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1412866.6666666667, ans=0.125
2023-11-21 07:36:13,636 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.47 vs. limit=10.0
2023-11-21 07:36:14,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1412933.3333333333, ans=0.0
2023-11-21 07:36:17,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 211950
2023-11-21 07:36:18,439 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.42 vs. limit=12.0
2023-11-21 07:36:20,303 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7550, loss[loss=0.04183, simple_loss=0.04627, pruned_loss=0.005557, audio_tagging_loss=0.01314, over 15059.00 frames. ], tot_loss[loss=0.07443, simple_loss=0.09552, pruned_loss=0.01694, audio_tagging_loss=0.009729, over 3039944.23 frames. ], batch size: 59, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:36:30,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1413000.0, ans=0.0
2023-11-21 07:36:34,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1413066.6666666667, ans=0.125
2023-11-21 07:36:38,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1413066.6666666667, ans=10.0
2023-11-21 07:36:41,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1413066.6666666667, ans=0.2
2023-11-21 07:36:49,531 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.72 vs. limit=10.0
2023-11-21 07:36:52,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1413133.3333333333, ans=0.125
2023-11-21 07:37:02,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1413200.0, ans=0.0
2023-11-21 07:37:03,231 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:37:12,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1413266.6666666667, ans=0.2
2023-11-21 07:37:22,450 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212000
2023-11-21 07:37:23,964 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-212000.pt
2023-11-21 07:37:28,037 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7600, loss[loss=0.07715, simple_loss=0.09395, pruned_loss=0.01747, audio_tagging_loss=0.01271, over 15121.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.09523, pruned_loss=0.01698, audio_tagging_loss=0.009802, over 3040939.88 frames. ], batch size: 56, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:37:35,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1413333.3333333333, ans=0.0
2023-11-21 07:37:48,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1413400.0, ans=0.015
2023-11-21 07:38:00,162 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.299e+01 8.757e+01 9.275e+01 1.161e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 07:38:13,748 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.26 vs. limit=10.0
2023-11-21 07:38:17,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1413533.3333333333, ans=0.0
2023-11-21 07:38:30,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212050
2023-11-21 07:38:32,598 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7650, loss[loss=0.08992, simple_loss=0.1183, pruned_loss=0.02273, audio_tagging_loss=0.008019, over 15024.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.09646, pruned_loss=0.01707, audio_tagging_loss=0.009659, over 3041339.76 frames. ], batch size: 53, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:38:34,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1413666.6666666667, ans=0.125
2023-11-21 07:38:46,404 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.71 vs. limit=12.0
2023-11-21 07:38:56,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1413800.0, ans=0.125
2023-11-21 07:39:04,863 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:39:11,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1413866.6666666667, ans=15.0
2023-11-21 07:39:34,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212100
2023-11-21 07:39:37,905 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7700, loss[loss=0.066, simple_loss=0.08039, pruned_loss=0.01445, audio_tagging_loss=0.01135, over 15579.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.0973, pruned_loss=0.01718, audio_tagging_loss=0.009659, over 3038532.47 frames. ], batch size: 60, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:39:49,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1414066.6666666667, ans=0.125
2023-11-21 07:39:59,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.75 vs. limit=15.0
2023-11-21 07:40:02,208 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:40:03,843 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.61 vs. limit=12.0
2023-11-21 07:40:07,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1414133.3333333333, ans=0.0
2023-11-21 07:40:10,453 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.263e+01 8.127e+01 8.808e+01 9.594e+01 1.135e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 07:40:33,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1414266.6666666667, ans=0.0
2023-11-21 07:40:34,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1414266.6666666667, ans=0.07
2023-11-21 07:40:40,142 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212150
2023-11-21 07:40:42,656 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7750, loss[loss=0.07201, simple_loss=0.08645, pruned_loss=0.01791, audio_tagging_loss=0.01088, over 14944.00 frames. ], tot_loss[loss=0.07556, simple_loss=0.09726, pruned_loss=0.01732, audio_tagging_loss=0.009611, over 3043494.52 frames. ], batch size: 56, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:40:45,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1414333.3333333333, ans=0.0
2023-11-21 07:40:51,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1414333.3333333333, ans=0.125
2023-11-21 07:40:52,174 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.97 vs. limit=6.0
2023-11-21 07:41:30,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1414533.3333333333, ans=0.0
2023-11-21 07:41:44,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1414600.0, ans=0.1
2023-11-21 07:41:46,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212200
2023-11-21 07:41:49,592 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7800, loss[loss=0.08239, simple_loss=0.1074, pruned_loss=0.01928, audio_tagging_loss=0.009389, over 15011.00 frames. ], tot_loss[loss=0.07609, simple_loss=0.09805, pruned_loss=0.01748, audio_tagging_loss=0.009582, over 3043139.45 frames. ], batch size: 56, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:41:57,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1414666.6666666667, ans=0.0
2023-11-21 07:42:21,722 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.253e+01 8.940e+01 9.769e+01 1.668e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-21 07:42:26,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1414866.6666666667, ans=0.0
2023-11-21 07:42:27,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1414866.6666666667, ans=0.125
2023-11-21 07:42:33,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.26 vs. limit=22.5
2023-11-21 07:42:44,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.whiten.whitening_limit, batch_count=1414933.3333333333, ans=12.0
2023-11-21 07:42:51,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212250
2023-11-21 07:42:52,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1415000.0, ans=0.0
2023-11-21 07:42:53,718 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7850, loss[loss=0.07967, simple_loss=0.1083, pruned_loss=0.01654, audio_tagging_loss=0.008955, over 15350.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.09733, pruned_loss=0.01726, audio_tagging_loss=0.009658, over 3044526.32 frames. ], batch size: 56, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:43:00,192 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 07:43:09,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1415066.6666666667, ans=0.125
2023-11-21 07:43:13,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1415066.6666666667, ans=0.125
2023-11-21 07:43:16,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1415066.6666666667, ans=0.2
2023-11-21 07:43:22,920 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.52 vs. limit=15.0
2023-11-21 07:43:37,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1415200.0, ans=0.0
2023-11-21 07:43:40,284 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.46 vs. limit=22.5
2023-11-21 07:43:56,302 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212300
2023-11-21 07:43:59,293 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7900, loss[loss=0.08786, simple_loss=0.1126, pruned_loss=0.01962, audio_tagging_loss=0.01193, over 15221.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09724, pruned_loss=0.01725, audio_tagging_loss=0.009829, over 3050767.68 frames. ], batch size: 55, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:43:59,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1415333.3333333333, ans=0.125
2023-11-21 07:44:00,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1415333.3333333333, ans=0.125
2023-11-21 07:44:23,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1415466.6666666667, ans=0.125
2023-11-21 07:44:31,963 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.631e+01 8.051e+01 8.734e+01 9.440e+01 1.312e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 07:44:33,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1415466.6666666667, ans=0.125
2023-11-21 07:44:49,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1415600.0, ans=0.125
2023-11-21 07:45:01,040 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212350
2023-11-21 07:45:03,393 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 7950, loss[loss=0.05659, simple_loss=0.07389, pruned_loss=0.007229, audio_tagging_loss=0.01242, over 14302.00 frames. ], tot_loss[loss=0.07573, simple_loss=0.0973, pruned_loss=0.01721, audio_tagging_loss=0.009866, over 3045422.35 frames. ], batch size: 53, lr: 3.80e-03, grad_scale: 16.0
2023-11-21 07:45:13,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1415666.6666666667, ans=0.125
2023-11-21 07:45:17,048 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 07:45:17,767 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.34 vs. limit=22.5
2023-11-21 07:45:18,003 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.22 vs. limit=6.0
2023-11-21 07:45:52,137 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=12.0
2023-11-21 07:45:59,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1415933.3333333333, ans=0.125
2023-11-21 07:46:04,040 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212400
2023-11-21 07:46:06,782 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8000, loss[loss=0.07143, simple_loss=0.1034, pruned_loss=0.01361, audio_tagging_loss=0.006135, over 15677.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09709, pruned_loss=0.01714, audio_tagging_loss=0.009869, over 3048253.27 frames. ], batch size: 56, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:46:08,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1416000.0, ans=0.2
2023-11-21 07:46:40,022 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 7.990e+01 8.488e+01 9.256e+01 1.198e+02, threshold=1.698e+02, percent-clipped=0.0
2023-11-21 07:47:07,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212450
2023-11-21 07:47:09,804 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8050, loss[loss=0.08234, simple_loss=0.1061, pruned_loss=0.02146, audio_tagging_loss=0.007862, over 15237.00 frames. ], tot_loss[loss=0.07577, simple_loss=0.09724, pruned_loss=0.01718, audio_tagging_loss=0.009969, over 3045266.12 frames. ], batch size: 57, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:47:10,739 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.09 vs. limit=15.0
2023-11-21 07:47:29,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1416400.0, ans=0.2
2023-11-21 07:48:05,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1416600.0, ans=0.125
2023-11-21 07:48:14,143 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212500
2023-11-21 07:48:16,507 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8100, loss[loss=0.07143, simple_loss=0.09403, pruned_loss=0.01429, audio_tagging_loss=0.01012, over 14303.00 frames. ], tot_loss[loss=0.07638, simple_loss=0.09817, pruned_loss=0.01744, audio_tagging_loss=0.009857, over 3045779.72 frames. ], batch size: 54, lr: 3.80e-03, grad_scale: 32.0
2023-11-21 07:48:19,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1416666.6666666667, ans=0.0
2023-11-21 07:48:24,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.54 vs. limit=22.5
2023-11-21 07:48:29,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=1416733.3333333333, ans=0.05
2023-11-21 07:48:35,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1416733.3333333333, ans=0.125
2023-11-21 07:48:48,750 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 8.164e+01 8.831e+01 9.512e+01 1.309e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 07:49:00,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1416866.6666666667, ans=0.5
2023-11-21 07:49:10,599 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.74 vs. limit=15.0
2023-11-21 07:49:17,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1416933.3333333333, ans=0.025
2023-11-21 07:49:18,464 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212550
2023-11-21 07:49:20,840 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8150, loss[loss=0.1067, simple_loss=0.1334, pruned_loss=0.03299, audio_tagging_loss=0.007059, over 15345.00 frames. ], tot_loss[loss=0.07709, simple_loss=0.09953, pruned_loss=0.01772, audio_tagging_loss=0.00961, over 3044397.95 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:49:22,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1417000.0, ans=0.125
2023-11-21 07:49:27,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1417000.0, ans=0.125
2023-11-21 07:49:38,826 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.54 vs. limit=22.5
2023-11-21 07:50:07,616 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.75 vs. limit=8.0
2023-11-21 07:50:15,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1417266.6666666667, ans=0.125
2023-11-21 07:50:19,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1417266.6666666667, ans=0.125
2023-11-21 07:50:21,544 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212600
2023-11-21 07:50:24,235 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8200, loss[loss=0.09123, simple_loss=0.1164, pruned_loss=0.02147, audio_tagging_loss=0.01154, over 14969.00 frames. ], tot_loss[loss=0.07688, simple_loss=0.09904, pruned_loss=0.01772, audio_tagging_loss=0.009637, over 3043317.78 frames. ], batch size: 55, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:50:24,294 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 07:50:29,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.88 vs. limit=15.0
2023-11-21 07:50:34,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1417333.3333333333, ans=0.125
2023-11-21 07:50:47,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.97 vs. limit=22.5
2023-11-21 07:50:58,022 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.391e+01 9.092e+01 9.919e+01 1.241e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-21 07:51:13,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1417533.3333333333, ans=0.125
2023-11-21 07:51:13,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1417533.3333333333, ans=0.1
2023-11-21 07:51:20,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1417600.0, ans=0.125
2023-11-21 07:51:21,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1417600.0, ans=0.2
2023-11-21 07:51:26,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212650
2023-11-21 07:51:29,792 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8250, loss[loss=0.08232, simple_loss=0.09644, pruned_loss=0.02283, audio_tagging_loss=0.01127, over 16986.00 frames. ], tot_loss[loss=0.07651, simple_loss=0.09846, pruned_loss=0.0177, audio_tagging_loss=0.009583, over 3050506.84 frames. ], batch size: 63, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:51:33,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1417666.6666666667, ans=0.1
2023-11-21 07:52:10,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1417866.6666666667, ans=0.125
2023-11-21 07:52:23,208 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.56 vs. limit=15.0
2023-11-21 07:52:32,648 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212700
2023-11-21 07:52:35,042 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8300, loss[loss=0.05933, simple_loss=0.07436, pruned_loss=0.01305, audio_tagging_loss=0.009105, over 16520.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.09849, pruned_loss=0.01761, audio_tagging_loss=0.009444, over 3066221.23 frames. ], batch size: 64, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:52:41,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1418000.0, ans=0.125
2023-11-21 07:53:10,534 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.350e+01 9.034e+01 9.751e+01 1.328e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 07:53:23,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1418200.0, ans=0.0
2023-11-21 07:53:30,736 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.07 vs. limit=10.0
2023-11-21 07:53:35,273 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.78 vs. limit=15.0
2023-11-21 07:53:37,290 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212750
2023-11-21 07:53:37,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1418266.6666666667, ans=0.0
2023-11-21 07:53:39,701 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8350, loss[loss=0.05782, simple_loss=0.07412, pruned_loss=0.01011, audio_tagging_loss=0.01066, over 14441.00 frames. ], tot_loss[loss=0.07531, simple_loss=0.09718, pruned_loss=0.01723, audio_tagging_loss=0.009492, over 3061449.42 frames. ], batch size: 55, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 07:54:02,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1418400.0, ans=0.125
2023-11-21 07:54:06,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1418466.6666666667, ans=0.0
2023-11-21 07:54:11,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1418466.6666666667, ans=0.0
2023-11-21 07:54:17,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1418466.6666666667, ans=0.5
2023-11-21 07:54:24,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1418533.3333333333, ans=0.04949747468305833
2023-11-21 07:54:29,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1418533.3333333333, ans=0.1
2023-11-21 07:54:30,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.69 vs. limit=10.0
2023-11-21 07:54:40,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1418600.0, ans=0.0
2023-11-21 07:54:42,458 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212800
2023-11-21 07:54:45,217 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8400, loss[loss=0.06314, simple_loss=0.08275, pruned_loss=0.0125, audio_tagging_loss=0.009264, over 15735.00 frames. ], tot_loss[loss=0.0755, simple_loss=0.09764, pruned_loss=0.01721, audio_tagging_loss=0.009464, over 3067035.10 frames. ], batch size: 59, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:54:45,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1418666.6666666667, ans=0.2
2023-11-21 07:54:47,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1418666.6666666667, ans=0.0
2023-11-21 07:54:48,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1418666.6666666667, ans=0.1
2023-11-21 07:54:52,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1418666.6666666667, ans=0.125
2023-11-21 07:55:11,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1418800.0, ans=0.125
2023-11-21 07:55:21,067 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.837e+01 7.810e+01 8.454e+01 9.398e+01 1.177e+02, threshold=1.691e+02, percent-clipped=0.0
2023-11-21 07:55:27,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1418866.6666666667, ans=0.2
2023-11-21 07:55:48,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1418933.3333333333, ans=0.0
2023-11-21 07:55:49,250 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212850
2023-11-21 07:55:51,654 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8450, loss[loss=0.06071, simple_loss=0.07944, pruned_loss=0.01215, audio_tagging_loss=0.008838, over 13756.00 frames. ], tot_loss[loss=0.07547, simple_loss=0.09756, pruned_loss=0.01723, audio_tagging_loss=0.009458, over 3062962.82 frames. ], batch size: 54, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:56:23,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1419133.3333333333, ans=0.125
2023-11-21 07:56:40,637 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.81 vs. limit=15.0
2023-11-21 07:56:49,021 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.58 vs. limit=6.0
2023-11-21 07:56:53,380 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212900
2023-11-21 07:56:55,737 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8500, loss[loss=0.07594, simple_loss=0.09271, pruned_loss=0.01876, audio_tagging_loss=0.01082, over 15418.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09813, pruned_loss=0.01739, audio_tagging_loss=0.009373, over 3058148.57 frames. ], batch size: 58, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:57:11,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1419400.0, ans=0.125
2023-11-21 07:57:14,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1419400.0, ans=0.125
2023-11-21 07:57:26,083 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.15 vs. limit=15.0
2023-11-21 07:57:32,202 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.883e+01 8.044e+01 8.694e+01 9.368e+01 1.136e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 07:57:42,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff2.min_abs, batch_count=1419533.3333333333, ans=0.1
2023-11-21 07:57:46,392 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.059e-02
2023-11-21 07:57:48,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1419600.0, ans=0.1
2023-11-21 07:57:58,855 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 212950
2023-11-21 07:58:01,255 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8550, loss[loss=0.09176, simple_loss=0.1212, pruned_loss=0.02407, audio_tagging_loss=0.007079, over 16668.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09773, pruned_loss=0.01716, audio_tagging_loss=0.009454, over 3070313.03 frames. ], batch size: 60, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:58:14,232 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.38 vs. limit=10.0
2023-11-21 07:59:02,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1419933.3333333333, ans=0.125
2023-11-21 07:59:03,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213000
2023-11-21 07:59:05,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1420000.0, ans=0.2
2023-11-21 07:59:06,333 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8600, loss[loss=0.05552, simple_loss=0.068, pruned_loss=0.01304, audio_tagging_loss=0.008488, over 14585.00 frames. ], tot_loss[loss=0.07558, simple_loss=0.0977, pruned_loss=0.01719, audio_tagging_loss=0.009537, over 3075250.39 frames. ], batch size: 55, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 07:59:07,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1420000.0, ans=0.125
2023-11-21 07:59:28,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1420066.6666666667, ans=0.5
2023-11-21 07:59:36,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.61 vs. limit=15.0
2023-11-21 07:59:40,775 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.787e+01 8.296e+01 8.926e+01 9.744e+01 1.392e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-21 07:59:42,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1420133.3333333333, ans=0.125
2023-11-21 07:59:48,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1420200.0, ans=0.125
2023-11-21 07:59:55,807 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.77 vs. limit=15.0
2023-11-21 08:00:08,177 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213050
2023-11-21 08:00:08,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1420266.6666666667, ans=0.1
2023-11-21 08:00:10,604 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8650, loss[loss=0.08568, simple_loss=0.1127, pruned_loss=0.02223, audio_tagging_loss=0.007098, over 14976.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09821, pruned_loss=0.0174, audio_tagging_loss=0.009632, over 3068167.47 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:00:13,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1420333.3333333333, ans=0.1
2023-11-21 08:00:39,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1420466.6666666667, ans=0.2
2023-11-21 08:00:54,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1420533.3333333333, ans=0.0
2023-11-21 08:01:12,644 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213100
2023-11-21 08:01:15,627 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8700, loss[loss=0.07895, simple_loss=0.1095, pruned_loss=0.01592, audio_tagging_loss=0.008294, over 15894.00 frames. ], tot_loss[loss=0.07601, simple_loss=0.09807, pruned_loss=0.01736, audio_tagging_loss=0.009613, over 3063782.63 frames. ], batch size: 61, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:01:16,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1420666.6666666667, ans=0.1
2023-11-21 08:01:52,036 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.710e+01 8.075e+01 9.054e+01 1.029e+02 1.299e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-21 08:01:55,288 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.61 vs. limit=15.0
2023-11-21 08:02:04,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1420866.6666666667, ans=0.0
2023-11-21 08:02:07,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1420933.3333333333, ans=0.2
2023-11-21 08:02:16,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1420933.3333333333, ans=0.0
2023-11-21 08:02:18,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213150
2023-11-21 08:02:21,299 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8750, loss[loss=0.07351, simple_loss=0.09951, pruned_loss=0.01651, audio_tagging_loss=0.007248, over 14919.00 frames. ], tot_loss[loss=0.07642, simple_loss=0.09837, pruned_loss=0.0175, audio_tagging_loss=0.009742, over 3054432.38 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:03:23,188 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213200
2023-11-21 08:03:26,464 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8800, loss[loss=0.1012, simple_loss=0.1155, pruned_loss=0.03105, audio_tagging_loss=0.01243, over 15935.00 frames. ], tot_loss[loss=0.07792, simple_loss=0.1003, pruned_loss=0.01796, audio_tagging_loss=0.009792, over 3059317.57 frames. ], batch size: 60, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 08:04:03,665 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.706e+01 8.113e+01 8.751e+01 9.814e+01 1.127e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 08:04:27,053 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=16.12 vs. limit=15.0
2023-11-21 08:04:27,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213250
2023-11-21 08:04:29,978 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8850, loss[loss=0.07482, simple_loss=0.09993, pruned_loss=0.01534, audio_tagging_loss=0.009516, over 15627.00 frames. ], tot_loss[loss=0.07742, simple_loss=0.09984, pruned_loss=0.01776, audio_tagging_loss=0.009739, over 3057669.40 frames. ], batch size: 59, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:04:43,084 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:05:07,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1421800.0, ans=0.0
2023-11-21 08:05:08,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1421866.6666666667, ans=0.04949747468305833
2023-11-21 08:05:22,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1421933.3333333333, ans=0.0
2023-11-21 08:05:33,396 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213300
2023-11-21 08:05:35,827 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8900, loss[loss=0.08237, simple_loss=0.1046, pruned_loss=0.02034, audio_tagging_loss=0.009743, over 14953.00 frames. ], tot_loss[loss=0.07692, simple_loss=0.09936, pruned_loss=0.01769, audio_tagging_loss=0.009543, over 3056058.15 frames. ], batch size: 56, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:05:39,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1422000.0, ans=0.2
2023-11-21 08:05:49,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1422066.6666666667, ans=0.1
2023-11-21 08:06:12,892 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 8.059e+01 8.546e+01 9.604e+01 1.318e+02, threshold=1.709e+02, percent-clipped=0.0
2023-11-21 08:06:22,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1422200.0, ans=10.0
2023-11-21 08:06:37,948 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213350
2023-11-21 08:06:40,314 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 8950, loss[loss=0.08863, simple_loss=0.1211, pruned_loss=0.02104, audio_tagging_loss=0.007043, over 15788.00 frames. ], tot_loss[loss=0.07728, simple_loss=0.1003, pruned_loss=0.01778, audio_tagging_loss=0.009376, over 3057124.66 frames. ], batch size: 58, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:06:47,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1422333.3333333333, ans=0.125
2023-11-21 08:07:37,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1422600.0, ans=0.2
2023-11-21 08:07:42,341 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213400
2023-11-21 08:07:44,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1422666.6666666667, ans=0.0
2023-11-21 08:07:45,033 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9000, loss[loss=0.05024, simple_loss=0.06062, pruned_loss=0.006726, audio_tagging_loss=0.01321, over 14014.00 frames. ], tot_loss[loss=0.07685, simple_loss=0.0998, pruned_loss=0.01762, audio_tagging_loss=0.009326, over 3059463.54 frames. ], batch size: 54, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:07:45,036 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 08:08:22,987 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([6.0076, 5.9057, 5.6967, 5.5876], device='cuda:0')
2023-11-21 08:08:25,296 INFO [train_asr.py:1253] (0/4) Epoch 18, validation: loss=0.06098, simple_loss=0.05248, pruned_loss=0.005341, audio_tagging_loss=0.02939, over 4681554.00 frames. 
2023-11-21 08:08:25,297 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 08:08:29,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1422666.6666666667, ans=0.1
2023-11-21 08:09:01,757 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.253e+01 9.144e+01 1.038e+02 1.327e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-21 08:09:08,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1422866.6666666667, ans=0.1
2023-11-21 08:09:10,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1422866.6666666667, ans=0.125
2023-11-21 08:09:18,817 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.09 vs. limit=15.0
2023-11-21 08:09:26,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213450
2023-11-21 08:09:29,090 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9050, loss[loss=0.05877, simple_loss=0.07101, pruned_loss=0.01353, audio_tagging_loss=0.009736, over 15812.00 frames. ], tot_loss[loss=0.07576, simple_loss=0.09839, pruned_loss=0.01719, audio_tagging_loss=0.009378, over 3063973.16 frames. ], batch size: 61, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:09:51,942 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.63 vs. limit=22.5
2023-11-21 08:09:58,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1423133.3333333333, ans=0.2
2023-11-21 08:10:25,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1423266.6666666667, ans=0.0
2023-11-21 08:10:30,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1423266.6666666667, ans=0.125
2023-11-21 08:10:31,342 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213500
2023-11-21 08:10:33,721 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9100, loss[loss=0.09174, simple_loss=0.1174, pruned_loss=0.02283, audio_tagging_loss=0.01023, over 16269.00 frames. ], tot_loss[loss=0.07589, simple_loss=0.09845, pruned_loss=0.01732, audio_tagging_loss=0.009342, over 3059544.55 frames. ], batch size: 60, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:11:11,210 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.196e+01 8.647e+01 9.263e+01 1.186e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-21 08:11:23,913 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.14 vs. limit=15.0
2023-11-21 08:11:36,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213550
2023-11-21 08:11:40,084 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9150, loss[loss=0.07335, simple_loss=0.09216, pruned_loss=0.01989, audio_tagging_loss=0.007384, over 15367.00 frames. ], tot_loss[loss=0.07527, simple_loss=0.09748, pruned_loss=0.01715, audio_tagging_loss=0.009382, over 3067248.96 frames. ], batch size: 58, lr: 3.79e-03, grad_scale: 16.0
2023-11-21 08:11:46,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1423666.6666666667, ans=0.125
2023-11-21 08:12:07,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1423800.0, ans=0.125
2023-11-21 08:12:17,291 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:12:22,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1423866.6666666667, ans=0.125
2023-11-21 08:12:34,238 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.53 vs. limit=15.0
2023-11-21 08:12:36,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1423933.3333333333, ans=0.125
2023-11-21 08:12:40,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1423933.3333333333, ans=0.0
2023-11-21 08:12:42,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213600
2023-11-21 08:12:45,250 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9200, loss[loss=0.07169, simple_loss=0.08895, pruned_loss=0.01791, audio_tagging_loss=0.009302, over 15323.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09758, pruned_loss=0.01714, audio_tagging_loss=0.009399, over 3067026.81 frames. ], batch size: 61, lr: 3.79e-03, grad_scale: 32.0
2023-11-21 08:12:46,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.32 vs. limit=12.0
2023-11-21 08:13:12,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1424133.3333333333, ans=0.0
2023-11-21 08:13:12,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1424133.3333333333, ans=0.125
2023-11-21 08:13:23,933 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 7.966e+01 8.700e+01 9.351e+01 1.190e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 08:13:34,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1424200.0, ans=0.125
2023-11-21 08:13:42,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1424266.6666666667, ans=0.125
2023-11-21 08:13:47,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213650
2023-11-21 08:13:49,529 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.01 vs. limit=15.0
2023-11-21 08:13:49,891 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9250, loss[loss=0.0814, simple_loss=0.1038, pruned_loss=0.02095, audio_tagging_loss=0.008558, over 14924.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.09661, pruned_loss=0.01706, audio_tagging_loss=0.009474, over 3059773.73 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:14:01,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1424333.3333333333, ans=0.0
2023-11-21 08:14:23,789 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.12 vs. limit=15.0
2023-11-21 08:14:24,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1424466.6666666667, ans=0.0
2023-11-21 08:14:30,899 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.94 vs. limit=12.0
2023-11-21 08:14:39,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1424533.3333333333, ans=0.125
2023-11-21 08:14:39,710 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.75 vs. limit=15.0
2023-11-21 08:14:48,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1424600.0, ans=0.1
2023-11-21 08:14:50,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1424600.0, ans=0.1
2023-11-21 08:14:52,595 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213700
2023-11-21 08:14:55,002 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9300, loss[loss=0.07379, simple_loss=0.09712, pruned_loss=0.01781, audio_tagging_loss=0.007417, over 15753.00 frames. ], tot_loss[loss=0.07422, simple_loss=0.09561, pruned_loss=0.01679, audio_tagging_loss=0.009617, over 3061507.90 frames. ], batch size: 61, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:15:02,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1424666.6666666667, ans=0.1
2023-11-21 08:15:24,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1424800.0, ans=0.0
2023-11-21 08:15:28,334 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.15 vs. limit=15.0
2023-11-21 08:15:29,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1424800.0, ans=0.0
2023-11-21 08:15:31,882 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.285e+01 7.980e+01 8.564e+01 9.426e+01 1.371e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 08:15:37,089 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.37 vs. limit=15.0
2023-11-21 08:15:46,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1424933.3333333333, ans=0.1
2023-11-21 08:15:50,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1424933.3333333333, ans=0.125
2023-11-21 08:15:57,661 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213750
2023-11-21 08:15:59,942 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9350, loss[loss=0.07476, simple_loss=0.08947, pruned_loss=0.0206, audio_tagging_loss=0.009423, over 14422.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09639, pruned_loss=0.01693, audio_tagging_loss=0.009543, over 3060409.17 frames. ], batch size: 55, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:16:07,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1425000.0, ans=10.0
2023-11-21 08:16:16,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1425066.6666666667, ans=0.0
2023-11-21 08:16:30,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1425133.3333333333, ans=0.0
2023-11-21 08:16:34,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1425133.3333333333, ans=0.05
2023-11-21 08:16:40,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1425200.0, ans=0.125
2023-11-21 08:16:56,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1425266.6666666667, ans=0.0
2023-11-21 08:17:02,396 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213800
2023-11-21 08:17:05,133 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9400, loss[loss=0.08135, simple_loss=0.1081, pruned_loss=0.01687, audio_tagging_loss=0.01041, over 15098.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09662, pruned_loss=0.01703, audio_tagging_loss=0.009715, over 3060381.75 frames. ], batch size: 56, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:17:16,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1425333.3333333333, ans=0.0
2023-11-21 08:17:32,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.00 vs. limit=15.0
2023-11-21 08:17:37,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1425466.6666666667, ans=0.2
2023-11-21 08:17:43,419 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 8.186e+01 8.902e+01 9.704e+01 1.344e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 08:18:04,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1425600.0, ans=0.04949747468305833
2023-11-21 08:18:08,495 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:18:08,555 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213850
2023-11-21 08:18:10,880 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9450, loss[loss=0.08195, simple_loss=0.1187, pruned_loss=0.0151, audio_tagging_loss=0.007506, over 15871.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09716, pruned_loss=0.01704, audio_tagging_loss=0.009822, over 3054932.54 frames. ], batch size: 58, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:18:21,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.82 vs. limit=12.0
2023-11-21 08:18:59,446 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.66 vs. limit=15.0
2023-11-21 08:19:02,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1425933.3333333333, ans=0.0
2023-11-21 08:19:07,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1425933.3333333333, ans=0.2
2023-11-21 08:19:08,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1425933.3333333333, ans=0.125
2023-11-21 08:19:08,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1425933.3333333333, ans=0.1
2023-11-21 08:19:13,859 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213900
2023-11-21 08:19:16,298 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9500, loss[loss=0.0689, simple_loss=0.07766, pruned_loss=0.01806, audio_tagging_loss=0.01201, over 15138.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09835, pruned_loss=0.01739, audio_tagging_loss=0.00984, over 3056380.59 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:19:25,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1426000.0, ans=0.0
2023-11-21 08:19:27,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1426066.6666666667, ans=0.125
2023-11-21 08:19:32,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1426066.6666666667, ans=0.0
2023-11-21 08:19:53,767 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.280e+01 9.020e+01 9.623e+01 1.395e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-21 08:20:12,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1426266.6666666667, ans=0.125
2023-11-21 08:20:17,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 213950
2023-11-21 08:20:20,270 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9550, loss[loss=0.06492, simple_loss=0.07759, pruned_loss=0.01537, audio_tagging_loss=0.01076, over 15228.00 frames. ], tot_loss[loss=0.07655, simple_loss=0.09854, pruned_loss=0.0174, audio_tagging_loss=0.009877, over 3057592.64 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:20:20,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1426333.3333333333, ans=0.0
2023-11-21 08:20:25,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.56 vs. limit=15.0
2023-11-21 08:20:30,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1426333.3333333333, ans=0.2
2023-11-21 08:20:33,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1426400.0, ans=0.125
2023-11-21 08:20:39,022 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.34 vs. limit=15.0
2023-11-21 08:20:40,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1426400.0, ans=0.125
2023-11-21 08:20:42,409 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.03 vs. limit=15.0
2023-11-21 08:21:00,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1426533.3333333333, ans=0.125
2023-11-21 08:21:12,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1426600.0, ans=0.09899494936611666
2023-11-21 08:21:13,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1426600.0, ans=0.0
2023-11-21 08:21:14,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1426600.0, ans=0.0
2023-11-21 08:21:22,582 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214000
2023-11-21 08:21:25,347 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9600, loss[loss=0.09157, simple_loss=0.1143, pruned_loss=0.02334, audio_tagging_loss=0.01108, over 15849.00 frames. ], tot_loss[loss=0.07632, simple_loss=0.09819, pruned_loss=0.01727, audio_tagging_loss=0.00996, over 3057684.98 frames. ], batch size: 63, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:21:52,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1426800.0, ans=0.0
2023-11-21 08:21:52,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1426800.0, ans=0.1
2023-11-21 08:21:59,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1426800.0, ans=0.2
2023-11-21 08:22:02,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1426800.0, ans=0.125
2023-11-21 08:22:02,942 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.086e+01 8.937e+01 9.718e+01 1.374e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-21 08:22:08,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1426866.6666666667, ans=0.0
2023-11-21 08:22:11,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1426866.6666666667, ans=0.1
2023-11-21 08:22:12,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1426866.6666666667, ans=0.125
2023-11-21 08:22:13,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1426866.6666666667, ans=0.0
2023-11-21 08:22:29,482 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214050
2023-11-21 08:22:31,885 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9650, loss[loss=0.06727, simple_loss=0.09304, pruned_loss=0.0114, audio_tagging_loss=0.009351, over 15686.00 frames. ], tot_loss[loss=0.07628, simple_loss=0.09809, pruned_loss=0.01723, audio_tagging_loss=0.01, over 3058687.26 frames. ], batch size: 61, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:23:01,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1427133.3333333333, ans=0.125
2023-11-21 08:23:12,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1427200.0, ans=0.125
2023-11-21 08:23:34,006 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214100
2023-11-21 08:23:36,415 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9700, loss[loss=0.08741, simple_loss=0.1111, pruned_loss=0.02411, audio_tagging_loss=0.007755, over 15207.00 frames. ], tot_loss[loss=0.0763, simple_loss=0.09831, pruned_loss=0.01736, audio_tagging_loss=0.009786, over 3061226.74 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:23:44,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1427333.3333333333, ans=0.125
2023-11-21 08:24:00,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1427400.0, ans=0.125
2023-11-21 08:24:09,159 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.91 vs. limit=15.0
2023-11-21 08:24:16,180 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.525e+01 8.096e+01 8.550e+01 9.320e+01 1.224e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-21 08:24:18,333 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.00 vs. limit=15.0
2023-11-21 08:24:35,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1427600.0, ans=0.0
2023-11-21 08:24:38,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214150
2023-11-21 08:24:41,401 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9750, loss[loss=0.07863, simple_loss=0.1027, pruned_loss=0.01744, audio_tagging_loss=0.009835, over 14649.00 frames. ], tot_loss[loss=0.07582, simple_loss=0.09777, pruned_loss=0.01718, audio_tagging_loss=0.009756, over 3049663.99 frames. ], batch size: 53, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:24:44,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1427666.6666666667, ans=0.125
2023-11-21 08:25:06,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1427733.3333333333, ans=0.125
2023-11-21 08:25:15,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1427800.0, ans=0.125
2023-11-21 08:25:19,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1427866.6666666667, ans=0.125
2023-11-21 08:25:21,492 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.40 vs. limit=15.0
2023-11-21 08:25:34,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1427933.3333333333, ans=0.1
2023-11-21 08:25:39,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1427933.3333333333, ans=0.0
2023-11-21 08:25:42,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1427933.3333333333, ans=0.0
2023-11-21 08:25:44,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.77 vs. limit=15.0
2023-11-21 08:25:45,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214200
2023-11-21 08:25:47,816 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9800, loss[loss=0.06534, simple_loss=0.08016, pruned_loss=0.01071, audio_tagging_loss=0.01456, over 14832.00 frames. ], tot_loss[loss=0.07586, simple_loss=0.09811, pruned_loss=0.01717, audio_tagging_loss=0.009632, over 3049596.58 frames. ], batch size: 56, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:25:55,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.55 vs. limit=12.0
2023-11-21 08:26:12,542 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.37 vs. limit=22.5
2023-11-21 08:26:13,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1428133.3333333333, ans=0.05
2023-11-21 08:26:25,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1428200.0, ans=0.05
2023-11-21 08:26:26,809 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.933e+01 8.204e+01 8.669e+01 9.345e+01 1.280e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 08:26:44,290 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:26:50,507 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214250
2023-11-21 08:26:50,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1428266.6666666667, ans=0.1
2023-11-21 08:26:52,997 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9850, loss[loss=0.08087, simple_loss=0.1073, pruned_loss=0.01674, audio_tagging_loss=0.01051, over 15532.00 frames. ], tot_loss[loss=0.07589, simple_loss=0.09814, pruned_loss=0.01735, audio_tagging_loss=0.009471, over 3049406.40 frames. ], batch size: 58, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:27:13,779 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:27:31,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1428533.3333333333, ans=0.1
2023-11-21 08:27:39,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1428533.3333333333, ans=0.125
2023-11-21 08:27:43,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.66 vs. limit=15.0
2023-11-21 08:27:55,029 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214300
2023-11-21 08:27:57,967 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9900, loss[loss=0.1056, simple_loss=0.1335, pruned_loss=0.02976, audio_tagging_loss=0.009077, over 14496.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09695, pruned_loss=0.01707, audio_tagging_loss=0.009552, over 3039426.16 frames. ], batch size: 55, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:28:14,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1428733.3333333333, ans=0.0
2023-11-21 08:28:18,267 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:28:37,136 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.117e+01 8.820e+01 9.379e+01 1.222e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 08:28:43,001 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.16 vs. limit=15.0
2023-11-21 08:28:43,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1428866.6666666667, ans=0.125
2023-11-21 08:28:44,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1428866.6666666667, ans=0.0
2023-11-21 08:28:47,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1428866.6666666667, ans=0.125
2023-11-21 08:29:01,202 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214350
2023-11-21 08:29:03,563 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 9950, loss[loss=0.0674, simple_loss=0.08498, pruned_loss=0.01326, audio_tagging_loss=0.01165, over 14354.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.09615, pruned_loss=0.0168, audio_tagging_loss=0.009651, over 3044096.52 frames. ], batch size: 54, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:29:13,092 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.12 vs. limit=15.0
2023-11-21 08:29:21,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1429066.6666666667, ans=10.0
2023-11-21 08:30:06,145 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214400
2023-11-21 08:30:08,810 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10000, loss[loss=0.08172, simple_loss=0.1043, pruned_loss=0.01885, audio_tagging_loss=0.01072, over 15324.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09658, pruned_loss=0.01686, audio_tagging_loss=0.009569, over 3043318.08 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 32.0
2023-11-21 08:30:19,589 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.63 vs. limit=15.0
2023-11-21 08:30:27,506 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.81 vs. limit=6.0
2023-11-21 08:30:29,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1429400.0, ans=0.1
2023-11-21 08:30:47,788 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.908e+01 8.046e+01 8.852e+01 9.481e+01 1.142e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 08:30:50,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1429533.3333333333, ans=0.125
2023-11-21 08:30:59,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1429600.0, ans=0.125
2023-11-21 08:31:10,687 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214450
2023-11-21 08:31:13,002 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10050, loss[loss=0.07777, simple_loss=0.1068, pruned_loss=0.01638, audio_tagging_loss=0.007985, over 14942.00 frames. ], tot_loss[loss=0.07489, simple_loss=0.09695, pruned_loss=0.01692, audio_tagging_loss=0.00949, over 3045431.00 frames. ], batch size: 55, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:31:16,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1429666.6666666667, ans=0.125
2023-11-21 08:31:18,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1429666.6666666667, ans=0.125
2023-11-21 08:31:20,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1429666.6666666667, ans=0.2
2023-11-21 08:31:43,944 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.85 vs. limit=10.0
2023-11-21 08:31:57,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1429866.6666666667, ans=0.0
2023-11-21 08:31:59,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1429866.6666666667, ans=6.0
2023-11-21 08:32:16,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214500
2023-11-21 08:32:19,104 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10100, loss[loss=0.07284, simple_loss=0.09865, pruned_loss=0.01472, audio_tagging_loss=0.008794, over 15349.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.09668, pruned_loss=0.01693, audio_tagging_loss=0.009513, over 3043454.00 frames. ], batch size: 55, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:32:31,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1430066.6666666667, ans=0.125
2023-11-21 08:32:39,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1430066.6666666667, ans=0.125
2023-11-21 08:32:39,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1430066.6666666667, ans=0.0
2023-11-21 08:32:58,371 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.203e+01 8.739e+01 9.472e+01 1.226e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 08:33:07,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1430200.0, ans=0.2
2023-11-21 08:33:09,601 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:33:17,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1430266.6666666667, ans=0.1
2023-11-21 08:33:18,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1430266.6666666667, ans=0.125
2023-11-21 08:33:20,675 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214550
2023-11-21 08:33:22,984 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10150, loss[loss=0.05988, simple_loss=0.08084, pruned_loss=0.01137, audio_tagging_loss=0.008099, over 15039.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09526, pruned_loss=0.01658, audio_tagging_loss=0.009612, over 3041072.22 frames. ], batch size: 58, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:33:51,622 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:33:51,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1430466.6666666667, ans=0.125
2023-11-21 08:34:02,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1430533.3333333333, ans=0.5
2023-11-21 08:34:02,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1430533.3333333333, ans=0.125
2023-11-21 08:34:06,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1430533.3333333333, ans=0.5
2023-11-21 08:34:12,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1430533.3333333333, ans=0.125
2023-11-21 08:34:25,576 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214600
2023-11-21 08:34:27,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1430666.6666666667, ans=22.5
2023-11-21 08:34:28,265 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10200, loss[loss=0.09498, simple_loss=0.1218, pruned_loss=0.02497, audio_tagging_loss=0.009139, over 15756.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09541, pruned_loss=0.01658, audio_tagging_loss=0.009664, over 3043373.49 frames. ], batch size: 58, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:34:37,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1430666.6666666667, ans=0.0
2023-11-21 08:34:40,545 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.78 vs. limit=15.0
2023-11-21 08:34:51,433 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:35:02,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1430800.0, ans=0.125
2023-11-21 08:35:08,560 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.283e+01 8.036e+01 8.614e+01 9.487e+01 2.302e+02, threshold=1.723e+02, percent-clipped=1.0
2023-11-21 08:35:10,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1430866.6666666667, ans=0.125
2023-11-21 08:35:10,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.94 vs. limit=15.0
2023-11-21 08:35:31,168 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214650
2023-11-21 08:35:33,544 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10250, loss[loss=0.107, simple_loss=0.1509, pruned_loss=0.02575, audio_tagging_loss=0.005833, over 15355.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.0957, pruned_loss=0.01674, audio_tagging_loss=0.00965, over 3044746.16 frames. ], batch size: 57, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:35:36,279 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.90 vs. limit=15.0
2023-11-21 08:35:51,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.94 vs. limit=22.5
2023-11-21 08:36:21,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1431200.0, ans=0.0
2023-11-21 08:36:33,256 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.30 vs. limit=15.0
2023-11-21 08:36:36,412 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214700
2023-11-21 08:36:38,824 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10300, loss[loss=0.07316, simple_loss=0.0965, pruned_loss=0.01653, audio_tagging_loss=0.008384, over 16333.00 frames. ], tot_loss[loss=0.07448, simple_loss=0.09566, pruned_loss=0.01684, audio_tagging_loss=0.00981, over 3046564.01 frames. ], batch size: 63, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:36:51,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1431400.0, ans=0.5
2023-11-21 08:37:07,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1431466.6666666667, ans=0.0
2023-11-21 08:37:09,212 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.20 vs. limit=15.0
2023-11-21 08:37:19,818 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.659e+01 8.199e+01 8.798e+01 9.773e+01 1.568e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 08:37:28,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1431533.3333333333, ans=0.125
2023-11-21 08:37:41,031 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214750
2023-11-21 08:37:43,452 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10350, loss[loss=0.07056, simple_loss=0.09269, pruned_loss=0.01478, audio_tagging_loss=0.009435, over 15148.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09511, pruned_loss=0.01655, audio_tagging_loss=0.009894, over 3050243.46 frames. ], batch size: 58, lr: 3.78e-03, grad_scale: 16.0
2023-11-21 08:37:50,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1431666.6666666667, ans=0.125
2023-11-21 08:37:54,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1431666.6666666667, ans=0.0
2023-11-21 08:37:57,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1431733.3333333333, ans=0.125
2023-11-21 08:38:09,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1431800.0, ans=0.0
2023-11-21 08:38:09,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1431800.0, ans=0.0
2023-11-21 08:38:28,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1431866.6666666667, ans=0.0
2023-11-21 08:38:46,841 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214800
2023-11-21 08:38:49,911 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10400, loss[loss=0.08294, simple_loss=0.1139, pruned_loss=0.01877, audio_tagging_loss=0.007237, over 15300.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.0958, pruned_loss=0.01657, audio_tagging_loss=0.009927, over 3058198.62 frames. ], batch size: 57, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:38:58,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1432000.0, ans=0.05
2023-11-21 08:39:08,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1432066.6666666667, ans=0.125
2023-11-21 08:39:15,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1432133.3333333333, ans=0.125
2023-11-21 08:39:22,202 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.72 vs. limit=12.0
2023-11-21 08:39:31,171 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.245e+01 8.104e+01 8.738e+01 9.653e+01 1.281e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 08:39:46,384 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:39:50,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1432266.6666666667, ans=0.125
2023-11-21 08:39:52,326 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214850
2023-11-21 08:39:54,705 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10450, loss[loss=0.05307, simple_loss=0.07022, pruned_loss=0.009144, audio_tagging_loss=0.008816, over 14714.00 frames. ], tot_loss[loss=0.07443, simple_loss=0.09592, pruned_loss=0.01658, audio_tagging_loss=0.009896, over 3056680.20 frames. ], batch size: 55, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:39:56,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1432333.3333333333, ans=0.07
2023-11-21 08:40:08,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1432400.0, ans=0.125
2023-11-21 08:40:09,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1432400.0, ans=0.1
2023-11-21 08:40:35,367 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.15 vs. limit=6.0
2023-11-21 08:40:35,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1432533.3333333333, ans=0.0
2023-11-21 08:40:36,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1432533.3333333333, ans=0.2
2023-11-21 08:40:47,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1432600.0, ans=0.125
2023-11-21 08:40:56,658 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214900
2023-11-21 08:40:59,142 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10500, loss[loss=0.07963, simple_loss=0.1007, pruned_loss=0.02219, audio_tagging_loss=0.007112, over 16592.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09595, pruned_loss=0.01656, audio_tagging_loss=0.009675, over 3054461.90 frames. ], batch size: 61, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:41:03,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1432666.6666666667, ans=0.0
2023-11-21 08:41:09,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1432666.6666666667, ans=0.1
2023-11-21 08:41:19,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1432733.3333333333, ans=0.125
2023-11-21 08:41:36,904 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.71 vs. limit=6.0
2023-11-21 08:41:40,092 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.754e+01 7.879e+01 8.434e+01 9.185e+01 1.259e+02, threshold=1.687e+02, percent-clipped=0.0
2023-11-21 08:41:44,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1432866.6666666667, ans=0.125
2023-11-21 08:42:01,012 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 214950
2023-11-21 08:42:03,541 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10550, loss[loss=0.08922, simple_loss=0.1188, pruned_loss=0.02164, audio_tagging_loss=0.008157, over 15537.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09575, pruned_loss=0.01655, audio_tagging_loss=0.009629, over 3056331.38 frames. ], batch size: 59, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:42:11,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1433000.0, ans=0.0
2023-11-21 08:42:12,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1433000.0, ans=0.125
2023-11-21 08:42:16,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1433066.6666666667, ans=0.125
2023-11-21 08:42:27,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1433066.6666666667, ans=0.0
2023-11-21 08:43:06,202 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215000
2023-11-21 08:43:08,948 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10600, loss[loss=0.07307, simple_loss=0.09143, pruned_loss=0.01854, audio_tagging_loss=0.008815, over 15528.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09619, pruned_loss=0.01665, audio_tagging_loss=0.009599, over 3050401.50 frames. ], batch size: 57, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:43:14,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1433333.3333333333, ans=0.0
2023-11-21 08:43:21,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1433400.0, ans=0.125
2023-11-21 08:43:51,460 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.274e+01 8.887e+01 9.806e+01 1.375e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 08:44:02,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1433600.0, ans=0.125
2023-11-21 08:44:04,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1433600.0, ans=0.1
2023-11-21 08:44:09,886 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215050
2023-11-21 08:44:10,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1433600.0, ans=0.04949747468305833
2023-11-21 08:44:12,277 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10650, loss[loss=0.07041, simple_loss=0.09024, pruned_loss=0.01568, audio_tagging_loss=0.009608, over 14936.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09619, pruned_loss=0.01681, audio_tagging_loss=0.009597, over 3046406.25 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 8.0
2023-11-21 08:44:20,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1433666.6666666667, ans=0.0
2023-11-21 08:44:25,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1433733.3333333333, ans=0.125
2023-11-21 08:44:27,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1433733.3333333333, ans=0.125
2023-11-21 08:44:54,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1433866.6666666667, ans=0.0
2023-11-21 08:45:01,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1433866.6666666667, ans=0.125
2023-11-21 08:45:14,991 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215100
2023-11-21 08:45:15,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1433933.3333333333, ans=0.2
2023-11-21 08:45:17,255 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10700, loss[loss=0.07441, simple_loss=0.09221, pruned_loss=0.0177, audio_tagging_loss=0.0106, over 15596.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.0966, pruned_loss=0.01685, audio_tagging_loss=0.009565, over 3043537.35 frames. ], batch size: 57, lr: 3.77e-03, grad_scale: 8.0
2023-11-21 08:45:27,495 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.05 vs. limit=12.0
2023-11-21 08:45:30,054 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.85 vs. limit=15.0
2023-11-21 08:46:00,182 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.074e+01 8.707e+01 9.700e+01 1.286e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-21 08:46:20,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1434266.6666666667, ans=0.04949747468305833
2023-11-21 08:46:21,366 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215150
2023-11-21 08:46:23,336 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.68 vs. limit=15.0
2023-11-21 08:46:23,789 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10750, loss[loss=0.07143, simple_loss=0.08447, pruned_loss=0.02044, audio_tagging_loss=0.008749, over 14857.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09639, pruned_loss=0.01685, audio_tagging_loss=0.009585, over 3049805.20 frames. ], batch size: 59, lr: 3.77e-03, grad_scale: 8.0
2023-11-21 08:46:41,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1434400.0, ans=0.2
2023-11-21 08:46:49,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1434466.6666666667, ans=0.0
2023-11-21 08:47:05,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1434533.3333333333, ans=15.0
2023-11-21 08:47:15,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1434600.0, ans=0.125
2023-11-21 08:47:25,598 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215200
2023-11-21 08:47:28,318 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10800, loss[loss=0.1021, simple_loss=0.1443, pruned_loss=0.02263, audio_tagging_loss=0.007374, over 16180.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09638, pruned_loss=0.01689, audio_tagging_loss=0.00947, over 3046056.07 frames. ], batch size: 57, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:47:38,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1434666.6666666667, ans=10.0
2023-11-21 08:47:42,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1434733.3333333333, ans=0.1
2023-11-21 08:48:11,102 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.512e+01 8.195e+01 8.985e+01 9.543e+01 1.112e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-21 08:48:12,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1434866.6666666667, ans=0.0
2023-11-21 08:48:17,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1434866.6666666667, ans=0.0
2023-11-21 08:48:22,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1434933.3333333333, ans=0.0
2023-11-21 08:48:29,595 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215250
2023-11-21 08:48:30,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.35 vs. limit=15.0
2023-11-21 08:48:32,556 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10850, loss[loss=0.06634, simple_loss=0.08876, pruned_loss=0.01219, audio_tagging_loss=0.009776, over 15324.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.09696, pruned_loss=0.01691, audio_tagging_loss=0.009445, over 3043524.48 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:48:33,351 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.77 vs. limit=10.0
2023-11-21 08:48:34,620 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=5.83 vs. limit=15.0
2023-11-21 08:48:40,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1435000.0, ans=0.0
2023-11-21 08:48:54,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1435066.6666666667, ans=0.125
2023-11-21 08:49:12,133 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.31 vs. limit=15.0
2023-11-21 08:49:14,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1435200.0, ans=0.0
2023-11-21 08:49:17,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.00 vs. limit=6.0
2023-11-21 08:49:19,173 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 08:49:33,045 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:49:35,527 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215300
2023-11-21 08:49:38,452 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10900, loss[loss=0.06499, simple_loss=0.07808, pruned_loss=0.01554, audio_tagging_loss=0.01041, over 14970.00 frames. ], tot_loss[loss=0.07527, simple_loss=0.09747, pruned_loss=0.017, audio_tagging_loss=0.009535, over 3043309.21 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:49:57,878 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.20 vs. limit=5.0
2023-11-21 08:50:05,266 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.03 vs. limit=22.5
2023-11-21 08:50:05,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1435466.6666666667, ans=0.0
2023-11-21 08:50:20,529 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.007e+01 7.936e+01 8.755e+01 9.807e+01 1.231e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 08:50:35,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1435600.0, ans=0.125
2023-11-21 08:50:40,622 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215350
2023-11-21 08:50:40,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1435600.0, ans=0.125
2023-11-21 08:50:42,938 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 10950, loss[loss=0.07263, simple_loss=0.09355, pruned_loss=0.01548, audio_tagging_loss=0.01037, over 15232.00 frames. ], tot_loss[loss=0.07522, simple_loss=0.09721, pruned_loss=0.01704, audio_tagging_loss=0.009575, over 3040163.28 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:50:56,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1435733.3333333333, ans=0.0
2023-11-21 08:51:15,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1435800.0, ans=0.125
2023-11-21 08:51:44,470 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215400
2023-11-21 08:51:47,125 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11000, loss[loss=0.0474, simple_loss=0.05144, pruned_loss=0.008615, audio_tagging_loss=0.01307, over 14750.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09654, pruned_loss=0.01694, audio_tagging_loss=0.009638, over 3039886.84 frames. ], batch size: 58, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:51:56,355 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 08:52:20,714 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.48 vs. limit=6.0
2023-11-21 08:52:29,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1436200.0, ans=0.0
2023-11-21 08:52:30,016 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 8.054e+01 8.831e+01 9.756e+01 2.281e+02, threshold=1.766e+02, percent-clipped=1.0
2023-11-21 08:52:50,276 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215450
2023-11-21 08:52:52,569 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11050, loss[loss=0.06285, simple_loss=0.08413, pruned_loss=0.01183, audio_tagging_loss=0.008959, over 14221.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.0968, pruned_loss=0.01708, audio_tagging_loss=0.009622, over 3041381.86 frames. ], batch size: 55, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:52:55,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1436333.3333333333, ans=0.2
2023-11-21 08:53:05,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1436400.0, ans=0.125
2023-11-21 08:53:10,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1436400.0, ans=0.2
2023-11-21 08:53:22,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1436466.6666666667, ans=0.125
2023-11-21 08:53:23,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1436466.6666666667, ans=0.125
2023-11-21 08:53:42,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1436600.0, ans=0.125
2023-11-21 08:53:50,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1436600.0, ans=0.0
2023-11-21 08:53:53,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=5.73 vs. limit=15.0
2023-11-21 08:53:54,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215500
2023-11-21 08:53:54,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1436600.0, ans=0.1
2023-11-21 08:53:56,883 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11100, loss[loss=0.0533, simple_loss=0.06546, pruned_loss=0.007681, audio_tagging_loss=0.01288, over 15692.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09699, pruned_loss=0.01718, audio_tagging_loss=0.009806, over 3046227.54 frames. ], batch size: 58, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:53:58,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1436666.6666666667, ans=0.0
2023-11-21 08:54:09,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1436733.3333333333, ans=0.125
2023-11-21 08:54:15,737 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.33 vs. limit=22.5
2023-11-21 08:54:27,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1436800.0, ans=0.0
2023-11-21 08:54:29,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1436800.0, ans=0.0
2023-11-21 08:54:39,002 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.330e+01 8.983e+01 9.923e+01 1.426e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-21 08:54:46,450 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.53 vs. limit=12.0
2023-11-21 08:54:58,164 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215550
2023-11-21 08:55:00,455 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11150, loss[loss=0.06911, simple_loss=0.08839, pruned_loss=0.01672, audio_tagging_loss=0.008196, over 14123.00 frames. ], tot_loss[loss=0.07532, simple_loss=0.09653, pruned_loss=0.01714, audio_tagging_loss=0.009905, over 3045678.87 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 16.0
2023-11-21 08:55:24,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1437066.6666666667, ans=0.125
2023-11-21 08:55:29,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1437133.3333333333, ans=0.1
2023-11-21 08:55:42,316 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.13 vs. limit=15.0
2023-11-21 08:55:46,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1437200.0, ans=0.125
2023-11-21 08:56:02,943 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215600
2023-11-21 08:56:06,631 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11200, loss[loss=0.1139, simple_loss=0.1451, pruned_loss=0.0343, audio_tagging_loss=0.007095, over 15663.00 frames. ], tot_loss[loss=0.07551, simple_loss=0.09678, pruned_loss=0.0172, audio_tagging_loss=0.009919, over 3041313.76 frames. ], batch size: 55, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:56:10,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1437333.3333333333, ans=0.2
2023-11-21 08:56:42,784 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.87 vs. limit=15.0
2023-11-21 08:56:48,513 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.237e+01 8.761e+01 9.554e+01 1.513e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 08:56:56,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1437533.3333333333, ans=0.0
2023-11-21 08:56:58,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1437600.0, ans=0.1
2023-11-21 08:56:59,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1437600.0, ans=0.0
2023-11-21 08:57:05,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1437600.0, ans=0.125
2023-11-21 08:57:08,364 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215650
2023-11-21 08:57:09,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1437666.6666666667, ans=0.0
2023-11-21 08:57:10,693 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11250, loss[loss=0.0753, simple_loss=0.1008, pruned_loss=0.01585, audio_tagging_loss=0.009071, over 15159.00 frames. ], tot_loss[loss=0.07459, simple_loss=0.09547, pruned_loss=0.01683, audio_tagging_loss=0.01002, over 3045363.80 frames. ], batch size: 56, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:57:19,457 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.16 vs. limit=15.0
2023-11-21 08:57:31,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1437733.3333333333, ans=0.04949747468305833
2023-11-21 08:57:57,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1437866.6666666667, ans=0.125
2023-11-21 08:58:04,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1437933.3333333333, ans=0.0
2023-11-21 08:58:12,867 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215700
2023-11-21 08:58:15,340 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11300, loss[loss=0.08846, simple_loss=0.121, pruned_loss=0.01859, audio_tagging_loss=0.009388, over 16320.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09578, pruned_loss=0.01691, audio_tagging_loss=0.009878, over 3038170.58 frames. ], batch size: 63, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:58:40,641 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.13 vs. limit=22.5
2023-11-21 08:58:41,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1438133.3333333333, ans=0.0
2023-11-21 08:58:43,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1438133.3333333333, ans=0.0
2023-11-21 08:58:48,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1438133.3333333333, ans=0.0
2023-11-21 08:58:51,801 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.81 vs. limit=15.0
2023-11-21 08:58:57,847 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.881e+01 8.122e+01 8.636e+01 9.244e+01 1.479e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-21 08:59:17,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215750
2023-11-21 08:59:20,438 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11350, loss[loss=0.08513, simple_loss=0.1141, pruned_loss=0.02137, audio_tagging_loss=0.006729, over 16308.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09652, pruned_loss=0.0172, audio_tagging_loss=0.009644, over 3031006.75 frames. ], batch size: 58, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 08:59:33,345 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.13 vs. limit=15.0
2023-11-21 08:59:34,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1438400.0, ans=0.0
2023-11-21 08:59:45,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1438466.6666666667, ans=0.0
2023-11-21 08:59:45,698 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.00 vs. limit=15.0
2023-11-21 08:59:48,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1438466.6666666667, ans=0.5
2023-11-21 08:59:48,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1438466.6666666667, ans=0.125
2023-11-21 09:00:16,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1438600.0, ans=0.125
2023-11-21 09:00:17,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1438600.0, ans=0.0
2023-11-21 09:00:22,135 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215800
2023-11-21 09:00:24,846 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11400, loss[loss=0.0745, simple_loss=0.09853, pruned_loss=0.01393, audio_tagging_loss=0.0113, over 15247.00 frames. ], tot_loss[loss=0.07549, simple_loss=0.0972, pruned_loss=0.01728, audio_tagging_loss=0.00961, over 3033244.31 frames. ], batch size: 61, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 09:00:26,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1438666.6666666667, ans=0.125
2023-11-21 09:00:28,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1438666.6666666667, ans=0.09899494936611666
2023-11-21 09:01:07,971 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.283e+01 9.027e+01 9.488e+01 1.297e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-21 09:01:26,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215850
2023-11-21 09:01:29,041 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11450, loss[loss=0.07025, simple_loss=0.08726, pruned_loss=0.01488, audio_tagging_loss=0.01174, over 15386.00 frames. ], tot_loss[loss=0.07612, simple_loss=0.09846, pruned_loss=0.01741, audio_tagging_loss=0.009477, over 3036302.58 frames. ], batch size: 60, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 09:01:48,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1439066.6666666667, ans=0.0
2023-11-21 09:01:57,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1439133.3333333333, ans=0.0
2023-11-21 09:02:19,220 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.06 vs. limit=15.0
2023-11-21 09:02:32,331 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215900
2023-11-21 09:02:34,715 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11500, loss[loss=0.08605, simple_loss=0.1117, pruned_loss=0.01881, audio_tagging_loss=0.01141, over 15981.00 frames. ], tot_loss[loss=0.07596, simple_loss=0.09831, pruned_loss=0.01726, audio_tagging_loss=0.009538, over 3043055.82 frames. ], batch size: 59, lr: 3.77e-03, grad_scale: 32.0
2023-11-21 09:02:55,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1439400.0, ans=0.125
2023-11-21 09:02:57,853 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.96 vs. limit=22.5
2023-11-21 09:03:17,265 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.959e+01 7.996e+01 8.548e+01 9.424e+01 1.295e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-21 09:03:17,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1439533.3333333333, ans=0.2
2023-11-21 09:03:37,754 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 215950
2023-11-21 09:03:40,094 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11550, loss[loss=0.07044, simple_loss=0.08739, pruned_loss=0.01647, audio_tagging_loss=0.01027, over 15214.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09767, pruned_loss=0.01709, audio_tagging_loss=0.009538, over 3050816.68 frames. ], batch size: 58, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:03:49,482 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.60 vs. limit=12.0
2023-11-21 09:03:57,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1439733.3333333333, ans=0.0
2023-11-21 09:04:09,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1439800.0, ans=0.125
2023-11-21 09:04:14,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1439800.0, ans=0.125
2023-11-21 09:04:18,425 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 09:04:22,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1439866.6666666667, ans=0.1
2023-11-21 09:04:26,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1439866.6666666667, ans=0.125
2023-11-21 09:04:40,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.77 vs. limit=22.5
2023-11-21 09:04:40,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1439933.3333333333, ans=0.125
2023-11-21 09:04:42,032 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216000
2023-11-21 09:04:43,598 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-216000.pt
2023-11-21 09:04:47,533 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11600, loss[loss=0.03039, simple_loss=0.02946, pruned_loss=0.005571, audio_tagging_loss=0.01009, over 14088.00 frames. ], tot_loss[loss=0.07518, simple_loss=0.09711, pruned_loss=0.01712, audio_tagging_loss=0.00951, over 3044908.00 frames. ], batch size: 56, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:05:00,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1440066.6666666667, ans=0.04949747468305833
2023-11-21 09:05:22,998 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.92 vs. limit=15.0
2023-11-21 09:05:29,708 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.026e+01 8.268e+01 8.853e+01 9.480e+01 1.180e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 09:05:46,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1440266.6666666667, ans=0.125
2023-11-21 09:05:48,782 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216050
2023-11-21 09:05:51,734 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11650, loss[loss=0.06139, simple_loss=0.07514, pruned_loss=0.01268, audio_tagging_loss=0.01114, over 16750.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09664, pruned_loss=0.01706, audio_tagging_loss=0.00955, over 3046896.59 frames. ], batch size: 66, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:06:02,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1440333.3333333333, ans=0.125
2023-11-21 09:06:06,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1440400.0, ans=0.0
2023-11-21 09:06:22,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1440466.6666666667, ans=0.05
2023-11-21 09:06:36,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1440533.3333333333, ans=0.125
2023-11-21 09:06:45,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1440600.0, ans=0.125
2023-11-21 09:06:54,290 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216100
2023-11-21 09:06:56,646 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11700, loss[loss=0.06977, simple_loss=0.09248, pruned_loss=0.01436, audio_tagging_loss=0.00917, over 15047.00 frames. ], tot_loss[loss=0.07497, simple_loss=0.0966, pruned_loss=0.01704, audio_tagging_loss=0.009625, over 3044272.77 frames. ], batch size: 60, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:07:13,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1440733.3333333333, ans=0.2
2023-11-21 09:07:38,773 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.900e+01 8.159e+01 8.719e+01 9.346e+01 1.199e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 09:07:57,235 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216150
2023-11-21 09:07:59,724 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11750, loss[loss=0.05609, simple_loss=0.06776, pruned_loss=0.009661, audio_tagging_loss=0.01254, over 14679.00 frames. ], tot_loss[loss=0.07483, simple_loss=0.09616, pruned_loss=0.01702, audio_tagging_loss=0.009734, over 3047288.60 frames. ], batch size: 54, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:08:16,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1441066.6666666667, ans=0.125
2023-11-21 09:08:25,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1441133.3333333333, ans=0.125
2023-11-21 09:08:29,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1441133.3333333333, ans=0.0
2023-11-21 09:08:47,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1441200.0, ans=0.2
2023-11-21 09:08:52,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1441266.6666666667, ans=0.0
2023-11-21 09:09:01,162 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216200
2023-11-21 09:09:04,068 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11800, loss[loss=0.08651, simple_loss=0.1126, pruned_loss=0.02201, audio_tagging_loss=0.008195, over 14925.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09609, pruned_loss=0.01711, audio_tagging_loss=0.009702, over 3049399.61 frames. ], batch size: 57, lr: 3.76e-03, grad_scale: 16.0
2023-11-21 09:09:08,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1441333.3333333333, ans=0.125
2023-11-21 09:09:17,762 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.71 vs. limit=6.0
2023-11-21 09:09:21,197 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.17 vs. limit=15.0
2023-11-21 09:09:27,717 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.21 vs. limit=15.0
2023-11-21 09:09:28,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1441400.0, ans=0.125
2023-11-21 09:09:45,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1441533.3333333333, ans=15.0
2023-11-21 09:09:48,127 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.548e+01 8.145e+01 8.665e+01 9.541e+01 1.453e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 09:09:48,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1441533.3333333333, ans=0.125
2023-11-21 09:09:55,513 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.11 vs. limit=12.0
2023-11-21 09:10:03,174 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=6.168e-02
2023-11-21 09:10:07,218 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.44 vs. limit=6.0
2023-11-21 09:10:07,373 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.54 vs. limit=15.0
2023-11-21 09:10:07,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216250
2023-11-21 09:10:10,305 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11850, loss[loss=0.06212, simple_loss=0.077, pruned_loss=0.01283, audio_tagging_loss=0.01079, over 16438.00 frames. ], tot_loss[loss=0.07503, simple_loss=0.09614, pruned_loss=0.01719, audio_tagging_loss=0.009773, over 3043666.39 frames. ], batch size: 61, lr: 3.76e-03, grad_scale: 16.0
2023-11-21 09:10:16,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1441666.6666666667, ans=0.0
2023-11-21 09:10:16,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1441666.6666666667, ans=0.1
2023-11-21 09:10:26,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1441733.3333333333, ans=0.1
2023-11-21 09:10:32,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1441733.3333333333, ans=0.0
2023-11-21 09:10:44,898 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.98 vs. limit=15.0
2023-11-21 09:10:48,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1441866.6666666667, ans=0.125
2023-11-21 09:11:11,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216300
2023-11-21 09:11:14,256 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11900, loss[loss=0.08221, simple_loss=0.1101, pruned_loss=0.01533, audio_tagging_loss=0.01182, over 15959.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09619, pruned_loss=0.01704, audio_tagging_loss=0.009781, over 3048109.90 frames. ], batch size: 59, lr: 3.76e-03, grad_scale: 16.0
2023-11-21 09:11:47,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1442133.3333333333, ans=0.1
2023-11-21 09:11:52,989 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.33 vs. limit=22.5
2023-11-21 09:11:54,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1442200.0, ans=0.125
2023-11-21 09:11:58,209 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.062e+01 8.187e+01 8.956e+01 9.688e+01 1.192e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-21 09:12:15,542 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216350
2023-11-21 09:12:18,005 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 11950, loss[loss=0.07978, simple_loss=0.09609, pruned_loss=0.02095, audio_tagging_loss=0.01079, over 15027.00 frames. ], tot_loss[loss=0.07518, simple_loss=0.09634, pruned_loss=0.01711, audio_tagging_loss=0.009898, over 3043088.79 frames. ], batch size: 57, lr: 3.76e-03, grad_scale: 16.0
2023-11-21 09:12:20,555 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:12:34,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1442400.0, ans=0.09899494936611666
2023-11-21 09:12:44,119 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.86 vs. limit=15.0
2023-11-21 09:12:47,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1442466.6666666667, ans=0.09899494936611666
2023-11-21 09:12:56,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1442533.3333333333, ans=0.125
2023-11-21 09:13:14,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1442600.0, ans=0.125
2023-11-21 09:13:17,275 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216400
2023-11-21 09:13:19,969 INFO [train_asr.py:1221] (0/4) Epoch 18, batch 12000, loss[loss=0.0789, simple_loss=0.09618, pruned_loss=0.02194, audio_tagging_loss=0.008868, over 14850.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09632, pruned_loss=0.01711, audio_tagging_loss=0.01006, over 3039682.53 frames. ], batch size: 57, lr: 3.76e-03, grad_scale: 32.0
2023-11-21 09:13:19,971 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 09:14:01,468 INFO [train_asr.py:1253] (0/4) Epoch 18, validation: loss=0.06063, simple_loss=0.05245, pruned_loss=0.005328, audio_tagging_loss=0.02908, over 4681554.00 frames. 
2023-11-21 09:14:01,469 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 09:14:08,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1442666.6666666667, ans=0.0
2023-11-21 09:14:17,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1442733.3333333333, ans=0.0
2023-11-21 09:14:25,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1442800.0, ans=0.95
2023-11-21 09:14:29,203 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-18.pt
2023-11-21 09:15:03,555 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 0, loss[loss=0.1097, simple_loss=0.1282, pruned_loss=0.02155, audio_tagging_loss=0.02402, over 15927.00 frames. ], tot_loss[loss=0.1097, simple_loss=0.1282, pruned_loss=0.02155, audio_tagging_loss=0.02402, over 15927.00 frames. ], batch size: 59, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:15:03,560 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 09:15:25,288 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9784, 3.2137, 2.8756, 3.1310, 3.4381, 2.7278, 3.4003, 2.7488],
       device='cuda:0')
2023-11-21 09:15:39,004 INFO [train_asr.py:1253] (0/4) Epoch 19, validation: loss=0.05975, simple_loss=0.05244, pruned_loss=0.005316, audio_tagging_loss=0.02822, over 4681554.00 frames. 
2023-11-21 09:15:39,005 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 09:15:45,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1442820.0, ans=0.125
2023-11-21 09:15:46,885 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.94 vs. limit=15.0
2023-11-21 09:15:52,326 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 8.140e+01 8.995e+01 9.841e+01 1.386e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-21 09:16:08,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1442953.3333333333, ans=0.125
2023-11-21 09:16:11,948 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216450
2023-11-21 09:16:12,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1442953.3333333333, ans=0.125
2023-11-21 09:16:24,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1443020.0, ans=0.125
2023-11-21 09:16:38,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1443086.6666666667, ans=0.1
2023-11-21 09:16:43,267 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 50, loss[loss=0.07343, simple_loss=0.08686, pruned_loss=0.01205, audio_tagging_loss=0.01795, over 14370.00 frames. ], tot_loss[loss=0.08101, simple_loss=0.09223, pruned_loss=0.01572, audio_tagging_loss=0.01918, over 689138.72 frames. ], batch size: 55, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:16:48,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1443153.3333333333, ans=0.125
2023-11-21 09:16:50,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1443153.3333333333, ans=0.125
2023-11-21 09:16:57,669 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.76 vs. limit=12.0
2023-11-21 09:17:04,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1443220.0, ans=0.125
2023-11-21 09:17:04,933 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:17:06,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1443220.0, ans=0.0
2023-11-21 09:17:15,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216500
2023-11-21 09:17:18,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1443286.6666666667, ans=0.125
2023-11-21 09:17:34,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1443420.0, ans=0.1
2023-11-21 09:17:38,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1443420.0, ans=0.125
2023-11-21 09:17:48,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1443486.6666666667, ans=0.125
2023-11-21 09:17:49,158 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 100, loss[loss=0.07302, simple_loss=0.08298, pruned_loss=0.01713, audio_tagging_loss=0.01441, over 15099.00 frames. ], tot_loss[loss=0.08082, simple_loss=0.09422, pruned_loss=0.01572, audio_tagging_loss=0.01798, over 1209507.34 frames. ], batch size: 59, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:17:53,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1443486.6666666667, ans=0.125
2023-11-21 09:17:59,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1443486.6666666667, ans=0.0
2023-11-21 09:18:00,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1443553.3333333333, ans=0.1
2023-11-21 09:18:02,448 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.866e+01 9.575e+01 1.050e+02 1.381e+02, threshold=1.915e+02, percent-clipped=0.0
2023-11-21 09:18:16,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1443620.0, ans=0.125
2023-11-21 09:18:16,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1443620.0, ans=0.0
2023-11-21 09:18:19,808 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216550
2023-11-21 09:18:27,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1443686.6666666667, ans=0.125
2023-11-21 09:18:49,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1443753.3333333333, ans=0.125
2023-11-21 09:18:52,671 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 150, loss[loss=0.05019, simple_loss=0.05045, pruned_loss=0.008658, audio_tagging_loss=0.01631, over 15324.00 frames. ], tot_loss[loss=0.07924, simple_loss=0.09408, pruned_loss=0.01602, audio_tagging_loss=0.01618, over 1618844.84 frames. ], batch size: 61, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:19:13,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1443886.6666666667, ans=0.125
2023-11-21 09:19:18,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1443953.3333333333, ans=0.0
2023-11-21 09:19:20,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1443953.3333333333, ans=0.0
2023-11-21 09:19:25,154 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216600
2023-11-21 09:19:33,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1444020.0, ans=0.0
2023-11-21 09:19:43,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1444086.6666666667, ans=0.125
2023-11-21 09:19:51,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1444086.6666666667, ans=0.125
2023-11-21 09:19:56,869 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 200, loss[loss=0.08245, simple_loss=0.1102, pruned_loss=0.0173, audio_tagging_loss=0.01005, over 14671.00 frames. ], tot_loss[loss=0.078, simple_loss=0.09474, pruned_loss=0.01635, audio_tagging_loss=0.01428, over 1936899.11 frames. ], batch size: 56, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:20:12,113 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.223e+01 8.596e+01 9.461e+01 1.153e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-21 09:20:13,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1444220.0, ans=0.125
2023-11-21 09:20:20,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=1444220.0, ans=0.025
2023-11-21 09:20:23,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1444286.6666666667, ans=0.0
2023-11-21 09:20:27,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1444286.6666666667, ans=10.0
2023-11-21 09:20:29,870 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216650
2023-11-21 09:20:42,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1444353.3333333333, ans=0.2
2023-11-21 09:20:49,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1444420.0, ans=0.125
2023-11-21 09:20:53,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1444420.0, ans=0.125
2023-11-21 09:21:02,396 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 250, loss[loss=0.06798, simple_loss=0.09002, pruned_loss=0.01547, audio_tagging_loss=0.0075, over 15366.00 frames. ], tot_loss[loss=0.07855, simple_loss=0.09759, pruned_loss=0.01695, audio_tagging_loss=0.01281, over 2183309.11 frames. ], batch size: 57, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:21:19,679 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.96 vs. limit=22.5
2023-11-21 09:21:30,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1444620.0, ans=0.125
2023-11-21 09:21:33,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216700
2023-11-21 09:21:44,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1444686.6666666667, ans=0.1
2023-11-21 09:21:55,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1444753.3333333333, ans=0.95
2023-11-21 09:21:57,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1444753.3333333333, ans=0.0
2023-11-21 09:22:06,608 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 300, loss[loss=0.08024, simple_loss=0.1083, pruned_loss=0.01824, audio_tagging_loss=0.007828, over 16032.00 frames. ], tot_loss[loss=0.07765, simple_loss=0.09752, pruned_loss=0.01707, audio_tagging_loss=0.01182, over 2379505.01 frames. ], batch size: 57, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:22:06,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1444820.0, ans=0.125
2023-11-21 09:22:19,962 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.255e+01 8.343e+01 8.887e+01 9.711e+01 1.312e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 09:22:38,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-21 09:22:38,912 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216750
2023-11-21 09:22:41,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1444953.3333333333, ans=0.1
2023-11-21 09:22:41,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1444953.3333333333, ans=0.125
2023-11-21 09:22:42,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1444953.3333333333, ans=0.125
2023-11-21 09:22:48,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1445020.0, ans=0.0
2023-11-21 09:22:59,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1445086.6666666667, ans=0.0
2023-11-21 09:23:09,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.83 vs. limit=15.0
2023-11-21 09:23:09,906 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 350, loss[loss=0.06867, simple_loss=0.09332, pruned_loss=0.01647, audio_tagging_loss=0.005536, over 15152.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09814, pruned_loss=0.01695, audio_tagging_loss=0.01111, over 2529126.11 frames. ], batch size: 56, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:23:25,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1445220.0, ans=0.05
2023-11-21 09:23:37,924 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.33 vs. limit=22.5
2023-11-21 09:23:42,999 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216800
2023-11-21 09:23:45,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1445286.6666666667, ans=0.125
2023-11-21 09:23:46,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=1445286.6666666667, ans=15.0
2023-11-21 09:24:00,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1445420.0, ans=0.0
2023-11-21 09:24:15,662 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 400, loss[loss=0.07831, simple_loss=0.09439, pruned_loss=0.01775, audio_tagging_loss=0.01336, over 14895.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.09654, pruned_loss=0.01672, audio_tagging_loss=0.01085, over 2638086.94 frames. ], batch size: 55, lr: 3.66e-03, grad_scale: 32.0
2023-11-21 09:24:29,288 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.137e+01 8.063e+01 8.732e+01 9.531e+01 1.428e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 09:24:31,185 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=12.0
2023-11-21 09:24:36,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1445553.3333333333, ans=0.2
2023-11-21 09:24:45,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1445620.0, ans=0.0
2023-11-21 09:24:47,455 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216850
2023-11-21 09:25:00,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1445686.6666666667, ans=0.1
2023-11-21 09:25:18,006 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.53 vs. limit=10.0
2023-11-21 09:25:19,747 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 450, loss[loss=0.09644, simple_loss=0.127, pruned_loss=0.02475, audio_tagging_loss=0.008171, over 14765.00 frames. ], tot_loss[loss=0.07588, simple_loss=0.09672, pruned_loss=0.01691, audio_tagging_loss=0.01061, over 2725294.79 frames. ], batch size: 55, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:25:20,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.87 vs. limit=8.0
2023-11-21 09:25:25,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1445820.0, ans=0.2
2023-11-21 09:25:26,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1445820.0, ans=0.0
2023-11-21 09:25:35,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1445886.6666666667, ans=0.1
2023-11-21 09:25:36,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1445886.6666666667, ans=0.125
2023-11-21 09:25:36,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1445886.6666666667, ans=0.125
2023-11-21 09:25:40,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1445886.6666666667, ans=0.2
2023-11-21 09:25:47,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1445953.3333333333, ans=0.0
2023-11-21 09:25:53,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216900
2023-11-21 09:25:57,230 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.30 vs. limit=15.0
2023-11-21 09:26:16,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1446086.6666666667, ans=0.125
2023-11-21 09:26:24,305 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 500, loss[loss=0.06386, simple_loss=0.07902, pruned_loss=0.01396, audio_tagging_loss=0.0104, over 15571.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09733, pruned_loss=0.01701, audio_tagging_loss=0.01036, over 2801754.38 frames. ], batch size: 59, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:26:41,045 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.147e+01 8.789e+01 9.630e+01 1.210e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 09:26:46,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1446220.0, ans=0.125
2023-11-21 09:26:51,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1446286.6666666667, ans=0.125
2023-11-21 09:26:54,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1446286.6666666667, ans=0.2
2023-11-21 09:26:56,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 216950
2023-11-21 09:27:01,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1446353.3333333333, ans=0.125
2023-11-21 09:27:05,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1446353.3333333333, ans=0.125
2023-11-21 09:27:13,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1446353.3333333333, ans=0.0
2023-11-21 09:27:16,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1446420.0, ans=0.125
2023-11-21 09:27:25,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1446420.0, ans=0.125
2023-11-21 09:27:29,718 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 550, loss[loss=0.07064, simple_loss=0.09144, pruned_loss=0.01681, audio_tagging_loss=0.008111, over 14690.00 frames. ], tot_loss[loss=0.07605, simple_loss=0.09725, pruned_loss=0.01715, audio_tagging_loss=0.01028, over 2858583.44 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:27:32,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1446486.6666666667, ans=0.125
2023-11-21 09:27:37,961 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.58 vs. limit=6.0
2023-11-21 09:27:57,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1446620.0, ans=0.025
2023-11-21 09:28:00,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217000
2023-11-21 09:28:02,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1446620.0, ans=0.125
2023-11-21 09:28:14,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1446686.6666666667, ans=0.125
2023-11-21 09:28:27,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1446753.3333333333, ans=0.1
2023-11-21 09:28:33,555 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 600, loss[loss=0.08842, simple_loss=0.1136, pruned_loss=0.02455, audio_tagging_loss=0.007072, over 14891.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09705, pruned_loss=0.01706, audio_tagging_loss=0.01012, over 2898825.37 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:28:38,186 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.71 vs. limit=15.0
2023-11-21 09:28:43,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1446820.0, ans=0.125
2023-11-21 09:28:49,014 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.367e+01 7.902e+01 8.372e+01 9.097e+01 1.242e+02, threshold=1.674e+02, percent-clipped=0.0
2023-11-21 09:28:52,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1446886.6666666667, ans=0.125
2023-11-21 09:28:53,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1446886.6666666667, ans=0.1
2023-11-21 09:29:07,207 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217050
2023-11-21 09:29:12,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1447020.0, ans=0.1
2023-11-21 09:29:20,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1447020.0, ans=0.0
2023-11-21 09:29:28,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1447086.6666666667, ans=0.125
2023-11-21 09:29:34,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1447086.6666666667, ans=0.2
2023-11-21 09:29:38,226 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 650, loss[loss=0.1056, simple_loss=0.1304, pruned_loss=0.03138, audio_tagging_loss=0.008995, over 15721.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09748, pruned_loss=0.01724, audio_tagging_loss=0.01012, over 2924824.26 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:29:48,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1447153.3333333333, ans=0.125
2023-11-21 09:29:48,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1447153.3333333333, ans=0.125
2023-11-21 09:30:04,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1447286.6666666667, ans=0.025
2023-11-21 09:30:10,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217100
2023-11-21 09:30:24,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1447353.3333333333, ans=0.125
2023-11-21 09:30:25,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1447353.3333333333, ans=0.07
2023-11-21 09:30:25,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1447353.3333333333, ans=0.125
2023-11-21 09:30:36,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1447420.0, ans=0.125
2023-11-21 09:30:44,266 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 700, loss[loss=0.05851, simple_loss=0.07482, pruned_loss=0.009253, audio_tagging_loss=0.01185, over 15798.00 frames. ], tot_loss[loss=0.07587, simple_loss=0.09734, pruned_loss=0.01717, audio_tagging_loss=0.01003, over 2956831.43 frames. ], batch size: 59, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:30:45,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1447486.6666666667, ans=0.0
2023-11-21 09:30:57,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1447553.3333333333, ans=0.125
2023-11-21 09:30:58,915 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.199e+01 8.920e+01 9.625e+01 1.204e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-21 09:31:02,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1447553.3333333333, ans=0.125
2023-11-21 09:31:14,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217150
2023-11-21 09:31:15,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1447620.0, ans=0.025
2023-11-21 09:31:19,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1447620.0, ans=0.1
2023-11-21 09:31:29,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1447686.6666666667, ans=0.04949747468305833
2023-11-21 09:31:33,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1447686.6666666667, ans=0.0
2023-11-21 09:31:47,780 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 750, loss[loss=0.06863, simple_loss=0.08312, pruned_loss=0.01678, audio_tagging_loss=0.0103, over 14507.00 frames. ], tot_loss[loss=0.07619, simple_loss=0.0977, pruned_loss=0.01734, audio_tagging_loss=0.01, over 2978643.40 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:31:49,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1447820.0, ans=0.0
2023-11-21 09:32:05,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1447886.6666666667, ans=0.125
2023-11-21 09:32:20,134 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217200
2023-11-21 09:32:28,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1448020.0, ans=0.0
2023-11-21 09:32:51,797 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 800, loss[loss=0.04834, simple_loss=0.05257, pruned_loss=0.009681, audio_tagging_loss=0.01238, over 13622.00 frames. ], tot_loss[loss=0.07602, simple_loss=0.09775, pruned_loss=0.01717, audio_tagging_loss=0.009981, over 2997185.19 frames. ], batch size: 54, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:32:59,125 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.55 vs. limit=15.0
2023-11-21 09:33:02,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1448153.3333333333, ans=0.125
2023-11-21 09:33:07,790 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.07 vs. limit=22.5
2023-11-21 09:33:08,185 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.219e+01 8.917e+01 9.925e+01 1.296e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-21 09:33:08,478 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:33:19,257 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.12 vs. limit=10.0
2023-11-21 09:33:24,907 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217250
2023-11-21 09:33:30,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1448353.3333333333, ans=0.125
2023-11-21 09:33:41,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1448353.3333333333, ans=0.2
2023-11-21 09:33:57,409 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 850, loss[loss=0.06858, simple_loss=0.0905, pruned_loss=0.01262, audio_tagging_loss=0.01071, over 14863.00 frames. ], tot_loss[loss=0.07595, simple_loss=0.09752, pruned_loss=0.01706, audio_tagging_loss=0.01013, over 3011636.39 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:34:14,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1448553.3333333333, ans=0.0
2023-11-21 09:34:21,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1448620.0, ans=0.125
2023-11-21 09:34:28,596 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217300
2023-11-21 09:34:32,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1448620.0, ans=0.125
2023-11-21 09:34:41,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1448686.6666666667, ans=0.125
2023-11-21 09:34:50,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1448753.3333333333, ans=0.125
2023-11-21 09:34:55,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1448753.3333333333, ans=0.2
2023-11-21 09:34:55,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1448753.3333333333, ans=0.125
2023-11-21 09:34:56,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1448753.3333333333, ans=0.2
2023-11-21 09:34:56,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1448753.3333333333, ans=0.0
2023-11-21 09:35:01,266 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 900, loss[loss=0.08488, simple_loss=0.111, pruned_loss=0.01993, audio_tagging_loss=0.009463, over 14241.00 frames. ], tot_loss[loss=0.07601, simple_loss=0.09743, pruned_loss=0.01706, audio_tagging_loss=0.01023, over 3018542.70 frames. ], batch size: 54, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:35:10,433 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.90 vs. limit=15.0
2023-11-21 09:35:15,737 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.060e+01 7.979e+01 8.801e+01 9.598e+01 1.201e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 09:35:33,205 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217350
2023-11-21 09:35:39,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1449020.0, ans=0.0
2023-11-21 09:35:43,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1449020.0, ans=0.0
2023-11-21 09:35:46,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1449020.0, ans=0.09899494936611666
2023-11-21 09:35:47,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1449020.0, ans=0.125
2023-11-21 09:35:52,238 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:35:54,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1449086.6666666667, ans=0.0
2023-11-21 09:36:04,295 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 950, loss[loss=0.06492, simple_loss=0.09016, pruned_loss=0.008186, audio_tagging_loss=0.01165, over 15740.00 frames. ], tot_loss[loss=0.07605, simple_loss=0.09796, pruned_loss=0.0171, audio_tagging_loss=0.009967, over 3023879.25 frames. ], batch size: 58, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:36:08,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1449153.3333333333, ans=0.1
2023-11-21 09:36:20,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1449220.0, ans=0.2
2023-11-21 09:36:23,309 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.22 vs. limit=6.0
2023-11-21 09:36:34,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1449286.6666666667, ans=0.015
2023-11-21 09:36:36,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217400
2023-11-21 09:36:39,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1449286.6666666667, ans=0.125
2023-11-21 09:36:43,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1449353.3333333333, ans=0.0
2023-11-21 09:36:49,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1449353.3333333333, ans=0.125
2023-11-21 09:37:00,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1449420.0, ans=0.1
2023-11-21 09:37:08,040 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1000, loss[loss=0.05366, simple_loss=0.06307, pruned_loss=0.01163, audio_tagging_loss=0.0105, over 14945.00 frames. ], tot_loss[loss=0.07576, simple_loss=0.09791, pruned_loss=0.01714, audio_tagging_loss=0.009667, over 3030527.20 frames. ], batch size: 58, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:37:13,190 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.55 vs. limit=6.0
2023-11-21 09:37:20,999 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.60 vs. limit=22.5
2023-11-21 09:37:25,113 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.132e+01 7.890e+01 8.682e+01 9.562e+01 1.074e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 09:37:26,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1449553.3333333333, ans=0.0
2023-11-21 09:37:27,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1449553.3333333333, ans=0.0
2023-11-21 09:37:34,496 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.65 vs. limit=15.0
2023-11-21 09:37:35,081 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 09:37:39,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217450
2023-11-21 09:37:58,469 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0
2023-11-21 09:38:04,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1449753.3333333333, ans=0.125
2023-11-21 09:38:11,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1449820.0, ans=0.125
2023-11-21 09:38:12,100 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1050, loss[loss=0.06459, simple_loss=0.08242, pruned_loss=0.01229, audio_tagging_loss=0.01109, over 14901.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.09751, pruned_loss=0.01715, audio_tagging_loss=0.009699, over 3035471.64 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:38:30,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1449886.6666666667, ans=0.0
2023-11-21 09:38:30,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1449886.6666666667, ans=0.125
2023-11-21 09:38:43,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217500
2023-11-21 09:38:43,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1449953.3333333333, ans=0.125
2023-11-21 09:39:00,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1450020.0, ans=0.125
2023-11-21 09:39:08,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1450086.6666666667, ans=0.2
2023-11-21 09:39:14,699 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1100, loss[loss=0.06302, simple_loss=0.07988, pruned_loss=0.01524, audio_tagging_loss=0.007843, over 15581.00 frames. ], tot_loss[loss=0.07554, simple_loss=0.09758, pruned_loss=0.01715, audio_tagging_loss=0.009597, over 3034197.02 frames. ], batch size: 59, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:39:18,234 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 09:39:27,859 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.55 vs. limit=6.0
2023-11-21 09:39:31,328 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.109e+01 8.788e+01 9.339e+01 1.148e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 09:39:47,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217550
2023-11-21 09:40:01,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1450353.3333333333, ans=0.2
2023-11-21 09:40:12,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.76 vs. limit=5.0
2023-11-21 09:40:18,399 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1150, loss[loss=0.07417, simple_loss=0.09103, pruned_loss=0.02014, audio_tagging_loss=0.008514, over 15479.00 frames. ], tot_loss[loss=0.07513, simple_loss=0.09699, pruned_loss=0.01706, audio_tagging_loss=0.009569, over 3033273.38 frames. ], batch size: 60, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:40:20,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1450486.6666666667, ans=0.0
2023-11-21 09:40:24,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1450486.6666666667, ans=0.2
2023-11-21 09:40:39,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1450553.3333333333, ans=0.125
2023-11-21 09:40:51,352 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217600
2023-11-21 09:40:51,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1450620.0, ans=0.0
2023-11-21 09:41:08,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.74 vs. limit=15.0
2023-11-21 09:41:23,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1450820.0, ans=0.07
2023-11-21 09:41:24,288 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1200, loss[loss=0.05991, simple_loss=0.07406, pruned_loss=0.01203, audio_tagging_loss=0.01085, over 15378.00 frames. ], tot_loss[loss=0.0746, simple_loss=0.09646, pruned_loss=0.01681, audio_tagging_loss=0.009562, over 3032002.88 frames. ], batch size: 58, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:41:24,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1450820.0, ans=0.125
2023-11-21 09:41:27,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1450820.0, ans=0.2
2023-11-21 09:41:40,314 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.126e+01 8.918e+01 9.759e+01 1.159e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-21 09:41:55,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217650
2023-11-21 09:42:28,257 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1250, loss[loss=0.0657, simple_loss=0.08599, pruned_loss=0.01228, audio_tagging_loss=0.01043, over 15754.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.09733, pruned_loss=0.01717, audio_tagging_loss=0.009509, over 3027823.58 frames. ], batch size: 58, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:43:00,170 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217700
2023-11-21 09:43:11,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1451353.3333333333, ans=0.125
2023-11-21 09:43:14,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1451353.3333333333, ans=0.0
2023-11-21 09:43:31,373 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1300, loss[loss=0.08874, simple_loss=0.1013, pruned_loss=0.02745, audio_tagging_loss=0.01066, over 14407.00 frames. ], tot_loss[loss=0.07521, simple_loss=0.09698, pruned_loss=0.01722, audio_tagging_loss=0.009496, over 3028017.62 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:43:48,500 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.540e+01 8.140e+01 8.673e+01 9.454e+01 1.388e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 09:43:50,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1451553.3333333333, ans=0.09899494936611666
2023-11-21 09:43:55,857 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.23 vs. limit=12.0
2023-11-21 09:44:03,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217750
2023-11-21 09:44:35,296 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1350, loss[loss=0.0681, simple_loss=0.08716, pruned_loss=0.0119, audio_tagging_loss=0.01262, over 13863.00 frames. ], tot_loss[loss=0.07529, simple_loss=0.09714, pruned_loss=0.01719, audio_tagging_loss=0.009527, over 3040042.66 frames. ], batch size: 55, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:44:45,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1451820.0, ans=0.125
2023-11-21 09:44:50,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1451886.6666666667, ans=0.1
2023-11-21 09:44:53,542 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.16 vs. limit=10.0
2023-11-21 09:45:00,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1451953.3333333333, ans=0.0
2023-11-21 09:45:06,570 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217800
2023-11-21 09:45:21,502 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 09:45:38,989 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1400, loss[loss=0.07469, simple_loss=0.09141, pruned_loss=0.02055, audio_tagging_loss=0.008429, over 15319.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09574, pruned_loss=0.0169, audio_tagging_loss=0.009638, over 3039283.99 frames. ], batch size: 56, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:45:56,178 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.285e+01 9.066e+01 9.966e+01 1.256e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-21 09:46:05,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1452286.6666666667, ans=0.125
2023-11-21 09:46:10,890 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217850
2023-11-21 09:46:11,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1452286.6666666667, ans=0.025
2023-11-21 09:46:18,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1452353.3333333333, ans=0.125
2023-11-21 09:46:24,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1452353.3333333333, ans=0.1
2023-11-21 09:46:33,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1452420.0, ans=0.125
2023-11-21 09:46:41,624 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1450, loss[loss=0.05512, simple_loss=0.06136, pruned_loss=0.01218, audio_tagging_loss=0.01227, over 15238.00 frames. ], tot_loss[loss=0.07561, simple_loss=0.09737, pruned_loss=0.01736, audio_tagging_loss=0.00957, over 3040769.78 frames. ], batch size: 57, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:46:45,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1452486.6666666667, ans=0.125
2023-11-21 09:46:45,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1452486.6666666667, ans=0.125
2023-11-21 09:46:49,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1452486.6666666667, ans=0.0
2023-11-21 09:46:52,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1452486.6666666667, ans=0.125
2023-11-21 09:46:56,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1452553.3333333333, ans=0.125
2023-11-21 09:46:57,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1452553.3333333333, ans=0.125
2023-11-21 09:47:14,047 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217900
2023-11-21 09:47:14,185 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:47:14,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1452620.0, ans=0.125
2023-11-21 09:47:45,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1452820.0, ans=0.125
2023-11-21 09:47:46,032 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1500, loss[loss=0.09152, simple_loss=0.132, pruned_loss=0.01921, audio_tagging_loss=0.006304, over 14948.00 frames. ], tot_loss[loss=0.07629, simple_loss=0.09819, pruned_loss=0.01746, audio_tagging_loss=0.009729, over 3043309.45 frames. ], batch size: 54, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:47:46,850 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.54 vs. limit=15.0
2023-11-21 09:47:53,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1452820.0, ans=0.1
2023-11-21 09:48:03,704 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.762e+01 8.061e+01 8.622e+01 9.502e+01 1.412e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-21 09:48:06,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1452886.6666666667, ans=0.0
2023-11-21 09:48:17,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 217950
2023-11-21 09:48:27,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=13.07 vs. limit=15.0
2023-11-21 09:48:30,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1453020.0, ans=0.125
2023-11-21 09:48:38,648 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.45 vs. limit=15.0
2023-11-21 09:48:49,040 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1550, loss[loss=0.07657, simple_loss=0.1005, pruned_loss=0.01814, audio_tagging_loss=0.008181, over 14941.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09696, pruned_loss=0.01713, audio_tagging_loss=0.009843, over 3046811.29 frames. ], batch size: 55, lr: 3.65e-03, grad_scale: 16.0
2023-11-21 09:49:08,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1453220.0, ans=0.125
2023-11-21 09:49:14,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1453286.6666666667, ans=0.125
2023-11-21 09:49:14,498 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.60 vs. limit=15.0
2023-11-21 09:49:15,745 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.02 vs. limit=10.0
2023-11-21 09:49:20,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1453286.6666666667, ans=0.125
2023-11-21 09:49:21,837 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218000
2023-11-21 09:49:23,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1453286.6666666667, ans=0.0
2023-11-21 09:49:33,658 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.99 vs. limit=15.0
2023-11-21 09:49:38,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1453353.3333333333, ans=0.0
2023-11-21 09:49:49,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1453420.0, ans=0.125
2023-11-21 09:49:53,169 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1600, loss[loss=0.06427, simple_loss=0.08839, pruned_loss=0.01228, audio_tagging_loss=0.007789, over 15988.00 frames. ], tot_loss[loss=0.07578, simple_loss=0.09742, pruned_loss=0.0172, audio_tagging_loss=0.009869, over 3045503.72 frames. ], batch size: 60, lr: 3.65e-03, grad_scale: 32.0
2023-11-21 09:49:53,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1453486.6666666667, ans=0.0
2023-11-21 09:50:11,475 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.287e+01 9.004e+01 9.532e+01 2.614e+02, threshold=1.801e+02, percent-clipped=1.0
2023-11-21 09:50:15,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1453553.3333333333, ans=0.125
2023-11-21 09:50:25,344 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218050
2023-11-21 09:50:34,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1453686.6666666667, ans=0.5
2023-11-21 09:50:34,964 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.20 vs. limit=12.0
2023-11-21 09:50:35,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1453686.6666666667, ans=0.0
2023-11-21 09:50:35,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1453686.6666666667, ans=0.125
2023-11-21 09:50:51,499 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.84 vs. limit=15.0
2023-11-21 09:50:57,614 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1650, loss[loss=0.09317, simple_loss=0.1238, pruned_loss=0.02312, audio_tagging_loss=0.008131, over 15733.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09787, pruned_loss=0.01727, audio_tagging_loss=0.00989, over 3053686.13 frames. ], batch size: 59, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:51:01,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1453820.0, ans=0.0
2023-11-21 09:51:26,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1453953.3333333333, ans=0.125
2023-11-21 09:51:29,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218100
2023-11-21 09:51:33,492 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.70 vs. limit=10.0
2023-11-21 09:51:38,401 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.37 vs. limit=12.0
2023-11-21 09:51:38,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.40 vs. limit=6.0
2023-11-21 09:51:58,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1454086.6666666667, ans=0.125
2023-11-21 09:52:01,494 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1700, loss[loss=0.07164, simple_loss=0.09204, pruned_loss=0.01453, audio_tagging_loss=0.01108, over 15062.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.0973, pruned_loss=0.01708, audio_tagging_loss=0.009892, over 3053179.46 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:52:14,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1454220.0, ans=0.07
2023-11-21 09:52:19,397 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.104e+01 8.594e+01 9.360e+01 1.130e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-21 09:52:33,607 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218150
2023-11-21 09:52:47,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=1454353.3333333333, ans=15.0
2023-11-21 09:52:58,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.14 vs. limit=15.0
2023-11-21 09:53:04,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1454486.6666666667, ans=0.2
2023-11-21 09:53:05,409 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1750, loss[loss=0.08302, simple_loss=0.1106, pruned_loss=0.01788, audio_tagging_loss=0.009842, over 15278.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09627, pruned_loss=0.01685, audio_tagging_loss=0.00992, over 3051674.95 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:53:35,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1454620.0, ans=0.0
2023-11-21 09:53:37,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218200
2023-11-21 09:53:52,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1454686.6666666667, ans=0.05
2023-11-21 09:53:53,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1454686.6666666667, ans=0.07
2023-11-21 09:53:59,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1454753.3333333333, ans=0.2
2023-11-21 09:54:08,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1454820.0, ans=0.2
2023-11-21 09:54:09,249 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1800, loss[loss=0.07264, simple_loss=0.09833, pruned_loss=0.01429, audio_tagging_loss=0.009186, over 14721.00 frames. ], tot_loss[loss=0.0747, simple_loss=0.09634, pruned_loss=0.01677, audio_tagging_loss=0.009767, over 3052595.90 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 09:54:12,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1454820.0, ans=0.2
2023-11-21 09:54:15,032 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 09:54:16,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1454820.0, ans=0.125
2023-11-21 09:54:28,213 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 7.963e+01 8.409e+01 9.319e+01 2.355e+02, threshold=1.682e+02, percent-clipped=0.0
2023-11-21 09:54:41,232 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218250
2023-11-21 09:55:13,326 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1850, loss[loss=0.0646, simple_loss=0.08647, pruned_loss=0.01033, audio_tagging_loss=0.01104, over 15271.00 frames. ], tot_loss[loss=0.07518, simple_loss=0.09708, pruned_loss=0.01703, audio_tagging_loss=0.00961, over 3050312.54 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 09:55:23,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1455153.3333333333, ans=0.0
2023-11-21 09:55:28,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1455220.0, ans=0.2
2023-11-21 09:55:45,232 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218300
2023-11-21 09:55:45,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1455286.6666666667, ans=0.0
2023-11-21 09:56:07,199 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0
2023-11-21 09:56:16,177 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1900, loss[loss=0.07111, simple_loss=0.09484, pruned_loss=0.01474, audio_tagging_loss=0.008955, over 15068.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09693, pruned_loss=0.0168, audio_tagging_loss=0.009543, over 3050910.26 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 09:56:28,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1455553.3333333333, ans=0.0
2023-11-21 09:56:35,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.90 vs. limit=6.0
2023-11-21 09:56:35,908 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.992e+01 8.400e+01 8.910e+01 9.667e+01 1.226e+02, threshold=1.782e+02, percent-clipped=1.0
2023-11-21 09:56:41,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1455620.0, ans=0.125
2023-11-21 09:56:48,875 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218350
2023-11-21 09:56:49,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1455620.0, ans=0.125
2023-11-21 09:57:12,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1455753.3333333333, ans=0.2
2023-11-21 09:57:17,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1455753.3333333333, ans=0.0
2023-11-21 09:57:20,924 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 1950, loss[loss=0.0734, simple_loss=0.09433, pruned_loss=0.01361, audio_tagging_loss=0.01263, over 14798.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09568, pruned_loss=0.01663, audio_tagging_loss=0.009573, over 3047143.33 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 09:57:26,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1455820.0, ans=0.125
2023-11-21 09:57:31,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.10 vs. limit=22.5
2023-11-21 09:57:41,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1455886.6666666667, ans=0.125
2023-11-21 09:57:43,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1455886.6666666667, ans=0.07
2023-11-21 09:57:52,293 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218400
2023-11-21 09:57:58,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1456020.0, ans=0.125
2023-11-21 09:58:00,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1456020.0, ans=0.125
2023-11-21 09:58:13,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1456086.6666666667, ans=0.2
2023-11-21 09:58:25,233 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2000, loss[loss=0.07186, simple_loss=0.08674, pruned_loss=0.01826, audio_tagging_loss=0.01023, over 14374.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09527, pruned_loss=0.01674, audio_tagging_loss=0.009597, over 3041845.07 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:58:43,537 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.049e+01 8.117e+01 8.612e+01 9.115e+01 1.173e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-21 09:58:57,071 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218450
2023-11-21 09:59:10,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1456353.3333333333, ans=0.1
2023-11-21 09:59:20,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1456420.0, ans=0.125
2023-11-21 09:59:21,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1456420.0, ans=0.125
2023-11-21 09:59:28,353 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2050, loss[loss=0.06899, simple_loss=0.08189, pruned_loss=0.01598, audio_tagging_loss=0.01207, over 14942.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09534, pruned_loss=0.01691, audio_tagging_loss=0.009664, over 3040229.65 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 09:59:35,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1456486.6666666667, ans=0.125
2023-11-21 09:59:40,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1456553.3333333333, ans=0.2
2023-11-21 09:59:43,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1456553.3333333333, ans=0.125
2023-11-21 09:59:57,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1456620.0, ans=0.2
2023-11-21 09:59:58,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1456620.0, ans=0.125
2023-11-21 10:00:00,943 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218500
2023-11-21 10:00:13,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1456686.6666666667, ans=0.0
2023-11-21 10:00:31,893 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2100, loss[loss=0.07482, simple_loss=0.09764, pruned_loss=0.01555, audio_tagging_loss=0.01044, over 16263.00 frames. ], tot_loss[loss=0.07381, simple_loss=0.09489, pruned_loss=0.01673, audio_tagging_loss=0.009641, over 3044295.05 frames. ], batch size: 61, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:00:47,458 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.65 vs. limit=22.5
2023-11-21 10:00:52,763 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.577e+01 8.022e+01 8.596e+01 9.289e+01 1.125e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-21 10:01:04,080 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218550
2023-11-21 10:01:24,914 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.47 vs. limit=12.0
2023-11-21 10:01:26,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1457086.6666666667, ans=0.125
2023-11-21 10:01:31,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1457086.6666666667, ans=0.125
2023-11-21 10:01:32,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1457086.6666666667, ans=0.125
2023-11-21 10:01:35,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1457153.3333333333, ans=0.125
2023-11-21 10:01:36,272 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2150, loss[loss=0.08876, simple_loss=0.1116, pruned_loss=0.02253, audio_tagging_loss=0.01045, over 14930.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09506, pruned_loss=0.01686, audio_tagging_loss=0.009702, over 3041075.02 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:01:46,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1457153.3333333333, ans=0.0
2023-11-21 10:01:59,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1457286.6666666667, ans=0.125
2023-11-21 10:02:01,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1457286.6666666667, ans=0.125
2023-11-21 10:02:06,996 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218600
2023-11-21 10:02:14,326 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:02:34,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1457420.0, ans=0.125
2023-11-21 10:02:37,314 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.25 vs. limit=15.0
2023-11-21 10:02:39,308 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2200, loss[loss=0.08645, simple_loss=0.1056, pruned_loss=0.02431, audio_tagging_loss=0.009366, over 15336.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09594, pruned_loss=0.01703, audio_tagging_loss=0.00966, over 3042261.83 frames. ], batch size: 54, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:02:40,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1457486.6666666667, ans=0.125
2023-11-21 10:02:59,799 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.151e+01 8.877e+01 9.686e+01 1.220e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 10:03:07,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1457620.0, ans=0.125
2023-11-21 10:03:12,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218650
2023-11-21 10:03:26,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1457686.6666666667, ans=0.125
2023-11-21 10:03:43,229 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2250, loss[loss=0.07467, simple_loss=0.09837, pruned_loss=0.01579, audio_tagging_loss=0.009695, over 15475.00 frames. ], tot_loss[loss=0.07532, simple_loss=0.09691, pruned_loss=0.01723, audio_tagging_loss=0.009639, over 3048254.15 frames. ], batch size: 58, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:03:52,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1457820.0, ans=0.125
2023-11-21 10:04:02,040 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.16 vs. limit=10.0
2023-11-21 10:04:11,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1457953.3333333333, ans=0.05
2023-11-21 10:04:16,173 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218700
2023-11-21 10:04:33,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1458086.6666666667, ans=0.0
2023-11-21 10:04:38,839 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0
2023-11-21 10:04:49,120 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2300, loss[loss=0.07846, simple_loss=0.09888, pruned_loss=0.01738, audio_tagging_loss=0.01164, over 15415.00 frames. ], tot_loss[loss=0.07497, simple_loss=0.09612, pruned_loss=0.01707, audio_tagging_loss=0.009837, over 3051740.94 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:05:10,244 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.111e+01 8.131e+01 8.990e+01 9.981e+01 1.492e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-21 10:05:20,295 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218750
2023-11-21 10:05:45,478 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:05:52,965 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2350, loss[loss=0.08605, simple_loss=0.1086, pruned_loss=0.02131, audio_tagging_loss=0.01041, over 15128.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09591, pruned_loss=0.01701, audio_tagging_loss=0.00995, over 3053772.83 frames. ], batch size: 57, lr: 3.64e-03, grad_scale: 8.0
2023-11-21 10:06:15,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1458553.3333333333, ans=0.125
2023-11-21 10:06:15,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1458553.3333333333, ans=0.1
2023-11-21 10:06:26,185 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218800
2023-11-21 10:06:39,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1458686.6666666667, ans=0.125
2023-11-21 10:06:42,073 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.42 vs. limit=15.0
2023-11-21 10:06:47,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1458753.3333333333, ans=0.2
2023-11-21 10:06:49,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1458753.3333333333, ans=0.04949747468305833
2023-11-21 10:06:54,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1458753.3333333333, ans=0.1
2023-11-21 10:06:57,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.78 vs. limit=22.5
2023-11-21 10:06:57,856 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2400, loss[loss=0.07517, simple_loss=0.1109, pruned_loss=0.01414, audio_tagging_loss=0.00559, over 14564.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09651, pruned_loss=0.01692, audio_tagging_loss=0.009943, over 3047960.15 frames. ], batch size: 56, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:06:59,723 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.38 vs. limit=12.0
2023-11-21 10:07:10,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1458886.6666666667, ans=0.125
2023-11-21 10:07:20,776 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.958e+01 7.990e+01 8.660e+01 9.489e+01 1.183e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 10:07:30,359 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.85 vs. limit=12.0
2023-11-21 10:07:30,975 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218850
2023-11-21 10:07:32,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1458953.3333333333, ans=0.0
2023-11-21 10:07:44,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1459020.0, ans=0.125
2023-11-21 10:07:44,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1459020.0, ans=10.0
2023-11-21 10:07:52,840 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.64 vs. limit=15.0
2023-11-21 10:07:58,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1459086.6666666667, ans=0.125
2023-11-21 10:08:03,728 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2450, loss[loss=0.05943, simple_loss=0.07245, pruned_loss=0.01256, audio_tagging_loss=0.01065, over 14546.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.09697, pruned_loss=0.01694, audio_tagging_loss=0.009928, over 3051641.61 frames. ], batch size: 54, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:08:35,267 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218900
2023-11-21 10:08:39,544 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.99 vs. limit=15.0
2023-11-21 10:08:44,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1459353.3333333333, ans=0.125
2023-11-21 10:09:08,363 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2500, loss[loss=0.07374, simple_loss=0.09864, pruned_loss=0.01385, audio_tagging_loss=0.01057, over 16334.00 frames. ], tot_loss[loss=0.07505, simple_loss=0.09644, pruned_loss=0.01692, audio_tagging_loss=0.009906, over 3042803.21 frames. ], batch size: 62, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:09:23,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1459553.3333333333, ans=0.0
2023-11-21 10:09:29,581 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.472e+01 8.078e+01 8.779e+01 9.710e+01 1.406e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-21 10:09:30,207 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.52 vs. limit=15.0
2023-11-21 10:09:41,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 218950
2023-11-21 10:09:53,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.65 vs. limit=6.0
2023-11-21 10:09:58,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1459686.6666666667, ans=0.2
2023-11-21 10:10:12,686 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2550, loss[loss=0.06941, simple_loss=0.09059, pruned_loss=0.01276, audio_tagging_loss=0.01135, over 16176.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09516, pruned_loss=0.01661, audio_tagging_loss=0.009882, over 3040801.92 frames. ], batch size: 61, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:10:17,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=1459820.0, ans=0.05
2023-11-21 10:10:33,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1459886.6666666667, ans=0.05
2023-11-21 10:10:45,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219000
2023-11-21 10:10:57,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1460020.0, ans=0.1
2023-11-21 10:11:01,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1460020.0, ans=0.125
2023-11-21 10:11:17,684 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2600, loss[loss=0.06405, simple_loss=0.08131, pruned_loss=0.01474, audio_tagging_loss=0.008652, over 15979.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09526, pruned_loss=0.01664, audio_tagging_loss=0.009741, over 3048415.81 frames. ], batch size: 59, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:11:33,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0
2023-11-21 10:11:39,747 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.684e+01 8.043e+01 8.700e+01 9.465e+01 1.335e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 10:11:41,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1460220.0, ans=0.125
2023-11-21 10:11:49,702 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219050
2023-11-21 10:11:52,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1460286.6666666667, ans=0.125
2023-11-21 10:11:52,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.07 vs. limit=6.0
2023-11-21 10:11:54,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1460353.3333333333, ans=0.0
2023-11-21 10:11:59,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1460353.3333333333, ans=0.0
2023-11-21 10:12:17,057 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.59 vs. limit=15.0
2023-11-21 10:12:19,364 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.78 vs. limit=6.0
2023-11-21 10:12:22,969 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2650, loss[loss=0.05366, simple_loss=0.06666, pruned_loss=0.009996, audio_tagging_loss=0.01034, over 16680.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09495, pruned_loss=0.01663, audio_tagging_loss=0.00964, over 3047058.60 frames. ], batch size: 65, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:12:55,062 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219100
2023-11-21 10:12:56,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1460620.0, ans=0.0
2023-11-21 10:13:08,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1460686.6666666667, ans=0.2
2023-11-21 10:13:10,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1460686.6666666667, ans=0.125
2023-11-21 10:13:10,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1460686.6666666667, ans=0.125
2023-11-21 10:13:14,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1460753.3333333333, ans=0.125
2023-11-21 10:13:14,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.49 vs. limit=15.0
2023-11-21 10:13:19,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1460753.3333333333, ans=0.95
2023-11-21 10:13:26,502 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2700, loss[loss=0.07681, simple_loss=0.09969, pruned_loss=0.01775, audio_tagging_loss=0.009215, over 15493.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.09583, pruned_loss=0.01688, audio_tagging_loss=0.009482, over 3046868.15 frames. ], batch size: 57, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:13:42,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1460886.6666666667, ans=0.1
2023-11-21 10:13:47,395 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.63 vs. limit=15.0
2023-11-21 10:13:48,922 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.184e+01 8.729e+01 9.389e+01 1.078e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 10:13:57,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1460953.3333333333, ans=0.125
2023-11-21 10:13:58,754 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219150
2023-11-21 10:14:31,247 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2750, loss[loss=0.06854, simple_loss=0.08999, pruned_loss=0.01742, audio_tagging_loss=0.006127, over 14295.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09538, pruned_loss=0.01661, audio_tagging_loss=0.009496, over 3047756.95 frames. ], batch size: 55, lr: 3.64e-03, grad_scale: 16.0
2023-11-21 10:14:35,880 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:15:02,930 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219200
2023-11-21 10:15:26,983 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:15:35,664 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2800, loss[loss=0.06532, simple_loss=0.08168, pruned_loss=0.0137, audio_tagging_loss=0.01078, over 15416.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09407, pruned_loss=0.01623, audio_tagging_loss=0.009569, over 3047187.74 frames. ], batch size: 58, lr: 3.64e-03, grad_scale: 32.0
2023-11-21 10:15:56,838 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.792e+01 8.334e+01 9.153e+01 1.021e+02 1.703e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-21 10:16:08,004 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219250
2023-11-21 10:16:10,540 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:16:17,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1461686.6666666667, ans=0.125
2023-11-21 10:16:30,204 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.45 vs. limit=15.0
2023-11-21 10:16:32,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1461753.3333333333, ans=0.125
2023-11-21 10:16:39,987 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2850, loss[loss=0.05559, simple_loss=0.07623, pruned_loss=0.01018, audio_tagging_loss=0.007287, over 14960.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09333, pruned_loss=0.01622, audio_tagging_loss=0.009563, over 3048126.39 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:16:44,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1461820.0, ans=0.125
2023-11-21 10:16:51,996 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:17:01,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1461886.6666666667, ans=22.5
2023-11-21 10:17:12,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219300
2023-11-21 10:17:45,357 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2900, loss[loss=0.06843, simple_loss=0.09575, pruned_loss=0.01259, audio_tagging_loss=0.007961, over 15556.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09538, pruned_loss=0.01669, audio_tagging_loss=0.009489, over 3049490.04 frames. ], batch size: 60, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:17:54,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1462153.3333333333, ans=0.125
2023-11-21 10:18:08,044 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.530e+01 8.282e+01 8.695e+01 9.389e+01 1.119e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 10:18:08,940 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.80 vs. limit=22.5
2023-11-21 10:18:17,315 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219350
2023-11-21 10:18:30,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1462353.3333333333, ans=0.2
2023-11-21 10:18:38,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1462420.0, ans=0.035
2023-11-21 10:18:49,480 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 2950, loss[loss=0.09205, simple_loss=0.1211, pruned_loss=0.02448, audio_tagging_loss=0.007041, over 14882.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.09699, pruned_loss=0.01704, audio_tagging_loss=0.009419, over 3052127.49 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:19:21,960 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219400
2023-11-21 10:19:26,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1462620.0, ans=0.125
2023-11-21 10:19:53,234 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3000, loss[loss=0.06762, simple_loss=0.0901, pruned_loss=0.01711, audio_tagging_loss=0.005461, over 14347.00 frames. ], tot_loss[loss=0.07552, simple_loss=0.09764, pruned_loss=0.0172, audio_tagging_loss=0.0095, over 3043810.82 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:19:53,237 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 10:20:13,295 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9797, 3.8145, 5.2060, 3.5009], device='cuda:0')
2023-11-21 10:20:32,681 INFO [train_asr.py:1253] (0/4) Epoch 19, validation: loss=0.05961, simple_loss=0.05235, pruned_loss=0.005214, audio_tagging_loss=0.02822, over 4681554.00 frames. 
2023-11-21 10:20:32,682 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 10:20:37,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1462820.0, ans=0.125
2023-11-21 10:20:54,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1462886.6666666667, ans=0.0
2023-11-21 10:20:54,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.12 vs. limit=22.5
2023-11-21 10:20:55,276 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.194e+01 8.910e+01 9.878e+01 1.451e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-21 10:21:00,701 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.09 vs. limit=10.0
2023-11-21 10:21:01,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1462953.3333333333, ans=0.125
2023-11-21 10:21:03,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219450
2023-11-21 10:21:06,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.12 vs. limit=15.0
2023-11-21 10:21:20,969 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.90 vs. limit=15.0
2023-11-21 10:21:28,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1463086.6666666667, ans=0.0
2023-11-21 10:21:37,031 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3050, loss[loss=0.07964, simple_loss=0.1111, pruned_loss=0.01558, audio_tagging_loss=0.00852, over 16562.00 frames. ], tot_loss[loss=0.07574, simple_loss=0.0978, pruned_loss=0.01727, audio_tagging_loss=0.009572, over 3043591.98 frames. ], batch size: 62, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:21:49,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1463220.0, ans=0.2
2023-11-21 10:21:53,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1463220.0, ans=0.0
2023-11-21 10:21:54,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1463220.0, ans=0.125
2023-11-21 10:21:58,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1463220.0, ans=0.1
2023-11-21 10:21:58,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1463220.0, ans=0.125
2023-11-21 10:22:09,578 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219500
2023-11-21 10:22:15,138 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:22:16,938 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.35 vs. limit=15.0
2023-11-21 10:22:21,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1463353.3333333333, ans=0.95
2023-11-21 10:22:40,802 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3100, loss[loss=0.05096, simple_loss=0.06092, pruned_loss=0.007928, audio_tagging_loss=0.01257, over 13575.00 frames. ], tot_loss[loss=0.07583, simple_loss=0.098, pruned_loss=0.01723, audio_tagging_loss=0.009597, over 3036735.21 frames. ], batch size: 53, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:22:41,555 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.18 vs. limit=15.0
2023-11-21 10:23:05,417 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.851e+01 8.202e+01 8.849e+01 9.632e+01 1.395e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 10:23:14,139 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219550
2023-11-21 10:23:14,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1463620.0, ans=0.125
2023-11-21 10:23:25,776 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.83 vs. limit=15.0
2023-11-21 10:23:46,191 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3150, loss[loss=0.1142, simple_loss=0.1427, pruned_loss=0.03711, audio_tagging_loss=0.005694, over 14615.00 frames. ], tot_loss[loss=0.07659, simple_loss=0.09879, pruned_loss=0.01753, audio_tagging_loss=0.00967, over 3036232.72 frames. ], batch size: 53, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:23:54,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1463820.0, ans=0.0
2023-11-21 10:24:17,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219600
2023-11-21 10:24:18,173 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.91 vs. limit=15.0
2023-11-21 10:24:50,990 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3200, loss[loss=0.07105, simple_loss=0.0898, pruned_loss=0.01777, audio_tagging_loss=0.008378, over 14287.00 frames. ], tot_loss[loss=0.07612, simple_loss=0.09813, pruned_loss=0.01731, audio_tagging_loss=0.009746, over 3046019.61 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:25:01,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1464153.3333333333, ans=0.2
2023-11-21 10:25:03,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1464220.0, ans=0.125
2023-11-21 10:25:06,886 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.92 vs. limit=22.5
2023-11-21 10:25:07,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1464220.0, ans=0.125
2023-11-21 10:25:12,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1464220.0, ans=0.125
2023-11-21 10:25:13,498 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.100e+01 8.202e+01 8.879e+01 9.509e+01 1.631e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 10:25:18,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1464286.6666666667, ans=0.1
2023-11-21 10:25:23,380 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219650
2023-11-21 10:25:23,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1464286.6666666667, ans=0.125
2023-11-21 10:25:25,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1464286.6666666667, ans=0.125
2023-11-21 10:25:44,011 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.20 vs. limit=10.0
2023-11-21 10:25:55,610 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3250, loss[loss=0.05179, simple_loss=0.04916, pruned_loss=0.01165, audio_tagging_loss=0.01555, over 14089.00 frames. ], tot_loss[loss=0.07542, simple_loss=0.09707, pruned_loss=0.01698, audio_tagging_loss=0.009901, over 3044465.27 frames. ], batch size: 58, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:25:59,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1464486.6666666667, ans=0.0
2023-11-21 10:26:19,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1464553.3333333333, ans=0.0
2023-11-21 10:26:25,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1464620.0, ans=0.125
2023-11-21 10:26:28,011 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219700
2023-11-21 10:26:31,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.84 vs. limit=12.0
2023-11-21 10:26:37,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1464686.6666666667, ans=0.0
2023-11-21 10:26:45,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1464753.3333333333, ans=0.0
2023-11-21 10:26:58,975 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3300, loss[loss=0.07562, simple_loss=0.0932, pruned_loss=0.01863, audio_tagging_loss=0.01038, over 14406.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09657, pruned_loss=0.01683, audio_tagging_loss=0.01001, over 3037695.16 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:27:03,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1464820.0, ans=0.1
2023-11-21 10:27:06,167 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 10:27:11,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1464820.0, ans=0.125
2023-11-21 10:27:13,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1464886.6666666667, ans=0.2
2023-11-21 10:27:24,141 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.952e+01 8.128e+01 8.736e+01 9.388e+01 1.641e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 10:27:24,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.70 vs. limit=15.0
2023-11-21 10:27:26,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=11.24 vs. limit=12.0
2023-11-21 10:27:30,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1464953.3333333333, ans=0.125
2023-11-21 10:27:32,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219750
2023-11-21 10:27:46,036 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.99 vs. limit=15.0
2023-11-21 10:27:58,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1465086.6666666667, ans=0.125
2023-11-21 10:28:06,497 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3350, loss[loss=0.08201, simple_loss=0.1107, pruned_loss=0.01827, audio_tagging_loss=0.008402, over 15379.00 frames. ], tot_loss[loss=0.07528, simple_loss=0.09714, pruned_loss=0.01682, audio_tagging_loss=0.009894, over 3046614.00 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:28:08,376 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.49 vs. limit=15.0
2023-11-21 10:28:37,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219800
2023-11-21 10:28:40,217 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.99 vs. limit=15.0
2023-11-21 10:28:55,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1465353.3333333333, ans=0.1
2023-11-21 10:29:11,219 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3400, loss[loss=0.08257, simple_loss=0.1101, pruned_loss=0.02028, audio_tagging_loss=0.007268, over 16530.00 frames. ], tot_loss[loss=0.0757, simple_loss=0.09767, pruned_loss=0.01709, audio_tagging_loss=0.009782, over 3045235.43 frames. ], batch size: 58, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:29:34,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1465553.3333333333, ans=0.07
2023-11-21 10:29:35,133 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.502e+01 9.051e+01 9.796e+01 1.205e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-21 10:29:39,450 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.56 vs. limit=15.0
2023-11-21 10:29:44,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1465620.0, ans=0.1
2023-11-21 10:29:45,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219850
2023-11-21 10:29:56,651 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.57 vs. limit=15.0
2023-11-21 10:30:15,867 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3450, loss[loss=0.09493, simple_loss=0.1374, pruned_loss=0.01936, audio_tagging_loss=0.006878, over 16181.00 frames. ], tot_loss[loss=0.07537, simple_loss=0.09718, pruned_loss=0.01708, audio_tagging_loss=0.009696, over 3044962.92 frames. ], batch size: 58, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:30:16,455 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.31 vs. limit=12.0
2023-11-21 10:30:22,499 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.68 vs. limit=12.0
2023-11-21 10:30:24,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1465820.0, ans=0.125
2023-11-21 10:30:49,328 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219900
2023-11-21 10:30:51,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1465953.3333333333, ans=0.125
2023-11-21 10:30:59,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1466020.0, ans=0.125
2023-11-21 10:31:22,093 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3500, loss[loss=0.07871, simple_loss=0.1069, pruned_loss=0.01603, audio_tagging_loss=0.009224, over 16297.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09662, pruned_loss=0.01694, audio_tagging_loss=0.009608, over 3055360.60 frames. ], batch size: 58, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:31:32,556 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.60 vs. limit=15.0
2023-11-21 10:31:44,241 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.587e+01 8.045e+01 8.820e+01 9.717e+01 1.244e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 10:31:52,914 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 219950
2023-11-21 10:31:54,005 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:32:15,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1466420.0, ans=0.05
2023-11-21 10:32:20,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1466420.0, ans=0.1
2023-11-21 10:32:21,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1466420.0, ans=0.0
2023-11-21 10:32:26,280 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3550, loss[loss=0.06503, simple_loss=0.08079, pruned_loss=0.01472, audio_tagging_loss=0.009905, over 14348.00 frames. ], tot_loss[loss=0.07442, simple_loss=0.09586, pruned_loss=0.0169, audio_tagging_loss=0.009588, over 3050778.40 frames. ], batch size: 54, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:32:32,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1466486.6666666667, ans=0.0
2023-11-21 10:32:46,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.37 vs. limit=15.0
2023-11-21 10:32:48,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1466553.3333333333, ans=0.2
2023-11-21 10:32:55,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1466620.0, ans=0.125
2023-11-21 10:32:58,994 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220000
2023-11-21 10:33:00,522 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-220000.pt
2023-11-21 10:33:09,896 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.83 vs. limit=15.0
2023-11-21 10:33:34,063 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3600, loss[loss=0.08252, simple_loss=0.1063, pruned_loss=0.01882, audio_tagging_loss=0.01054, over 14740.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09472, pruned_loss=0.01665, audio_tagging_loss=0.009632, over 3036852.09 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 32.0
2023-11-21 10:33:56,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1466886.6666666667, ans=0.0
2023-11-21 10:34:00,209 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.124e+01 7.919e+01 8.702e+01 9.494e+01 1.216e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 10:34:07,757 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220050
2023-11-21 10:34:33,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1467086.6666666667, ans=0.125
2023-11-21 10:34:40,916 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3650, loss[loss=0.08586, simple_loss=0.1083, pruned_loss=0.02193, audio_tagging_loss=0.009787, over 15677.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09465, pruned_loss=0.01659, audio_tagging_loss=0.009581, over 3039259.36 frames. ], batch size: 60, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:34:47,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1467153.3333333333, ans=0.125
2023-11-21 10:34:59,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1467220.0, ans=0.125
2023-11-21 10:35:08,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1467286.6666666667, ans=0.125
2023-11-21 10:35:12,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220100
2023-11-21 10:35:14,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1467286.6666666667, ans=0.125
2023-11-21 10:35:42,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1467420.0, ans=0.0
2023-11-21 10:35:44,962 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3700, loss[loss=0.06456, simple_loss=0.08143, pruned_loss=0.01224, audio_tagging_loss=0.0116, over 15919.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09576, pruned_loss=0.01682, audio_tagging_loss=0.00952, over 3045748.81 frames. ], batch size: 61, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:35:50,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1467486.6666666667, ans=0.2
2023-11-21 10:35:53,056 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.40 vs. limit=22.5
2023-11-21 10:35:57,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1467553.3333333333, ans=0.125
2023-11-21 10:36:08,757 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.463e+01 8.151e+01 8.696e+01 9.414e+01 1.695e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 10:36:17,764 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220150
2023-11-21 10:36:23,437 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.11 vs. limit=22.5
2023-11-21 10:36:24,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1467686.6666666667, ans=0.1
2023-11-21 10:36:45,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1467753.3333333333, ans=0.125
2023-11-21 10:36:49,677 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3750, loss[loss=0.08818, simple_loss=0.1093, pruned_loss=0.02147, audio_tagging_loss=0.01208, over 15333.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09688, pruned_loss=0.01705, audio_tagging_loss=0.009523, over 3047575.07 frames. ], batch size: 56, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:37:14,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1467886.6666666667, ans=0.125
2023-11-21 10:37:23,658 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220200
2023-11-21 10:37:35,335 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:37:57,254 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3800, loss[loss=0.05084, simple_loss=0.06129, pruned_loss=0.008562, audio_tagging_loss=0.01163, over 16841.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09616, pruned_loss=0.01683, audio_tagging_loss=0.009591, over 3044741.64 frames. ], batch size: 64, lr: 3.63e-03, grad_scale: 8.0
2023-11-21 10:38:07,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1468153.3333333333, ans=0.09899494936611666
2023-11-21 10:38:08,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.54 vs. limit=22.5
2023-11-21 10:38:18,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1468220.0, ans=0.0
2023-11-21 10:38:19,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1468220.0, ans=0.2
2023-11-21 10:38:22,689 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.633e+01 8.173e+01 8.825e+01 9.624e+01 1.863e+02, threshold=1.765e+02, percent-clipped=1.0
2023-11-21 10:38:26,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1468286.6666666667, ans=0.125
2023-11-21 10:38:28,977 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220250
2023-11-21 10:39:01,414 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3850, loss[loss=0.07561, simple_loss=0.09867, pruned_loss=0.0149, audio_tagging_loss=0.01137, over 15190.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09647, pruned_loss=0.01666, audio_tagging_loss=0.009646, over 3036384.75 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 8.0
2023-11-21 10:39:07,458 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.46 vs. limit=15.0
2023-11-21 10:39:27,896 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.70 vs. limit=15.0
2023-11-21 10:39:28,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1468620.0, ans=0.0
2023-11-21 10:39:34,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220300
2023-11-21 10:39:38,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1468620.0, ans=0.1
2023-11-21 10:39:43,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1468686.6666666667, ans=0.125
2023-11-21 10:39:59,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1468753.3333333333, ans=0.125
2023-11-21 10:39:59,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1468753.3333333333, ans=0.1
2023-11-21 10:40:06,840 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3900, loss[loss=0.0713, simple_loss=0.09419, pruned_loss=0.01419, audio_tagging_loss=0.01002, over 14971.00 frames. ], tot_loss[loss=0.07471, simple_loss=0.09657, pruned_loss=0.01673, audio_tagging_loss=0.009697, over 3036470.02 frames. ], batch size: 55, lr: 3.63e-03, grad_scale: 8.0
2023-11-21 10:40:34,094 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.413e+01 7.996e+01 8.695e+01 9.569e+01 1.427e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 10:40:40,573 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220350
2023-11-21 10:40:51,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1469020.0, ans=0.1
2023-11-21 10:41:05,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1469086.6666666667, ans=0.125
2023-11-21 10:41:08,829 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.97 vs. limit=15.0
2023-11-21 10:41:13,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1469153.3333333333, ans=0.0
2023-11-21 10:41:13,861 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 3950, loss[loss=0.08511, simple_loss=0.1137, pruned_loss=0.02095, audio_tagging_loss=0.007297, over 15662.00 frames. ], tot_loss[loss=0.07471, simple_loss=0.09647, pruned_loss=0.01673, audio_tagging_loss=0.009738, over 3041757.20 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 8.0
2023-11-21 10:41:18,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.54 vs. limit=12.0
2023-11-21 10:41:39,049 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.964e-01
2023-11-21 10:41:46,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220400
2023-11-21 10:41:54,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1469353.3333333333, ans=0.125
2023-11-21 10:41:59,762 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.09 vs. limit=15.0
2023-11-21 10:42:01,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1469353.3333333333, ans=0.125
2023-11-21 10:42:10,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1469420.0, ans=0.125
2023-11-21 10:42:11,081 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.44 vs. limit=22.5
2023-11-21 10:42:15,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1469420.0, ans=0.2
2023-11-21 10:42:19,398 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4000, loss[loss=0.08658, simple_loss=0.1201, pruned_loss=0.0175, audio_tagging_loss=0.009026, over 15865.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09763, pruned_loss=0.01703, audio_tagging_loss=0.00979, over 3044725.89 frames. ], batch size: 57, lr: 3.63e-03, grad_scale: 16.0
2023-11-21 10:42:44,798 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 8.206e+01 8.898e+01 9.673e+01 1.616e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 10:42:52,356 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220450
2023-11-21 10:42:59,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1469686.6666666667, ans=0.2
2023-11-21 10:43:01,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1469686.6666666667, ans=0.0
2023-11-21 10:43:21,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1469753.3333333333, ans=0.2
2023-11-21 10:43:24,147 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4050, loss[loss=0.06851, simple_loss=0.08207, pruned_loss=0.01682, audio_tagging_loss=0.01066, over 14593.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.09747, pruned_loss=0.01699, audio_tagging_loss=0.009876, over 3037801.77 frames. ], batch size: 57, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:43:26,792 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:43:35,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1469886.6666666667, ans=0.125
2023-11-21 10:43:47,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1469886.6666666667, ans=0.125
2023-11-21 10:43:56,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220500
2023-11-21 10:44:26,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1470086.6666666667, ans=0.2
2023-11-21 10:44:29,554 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4100, loss[loss=0.08093, simple_loss=0.105, pruned_loss=0.01841, audio_tagging_loss=0.01002, over 14624.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09669, pruned_loss=0.01691, audio_tagging_loss=0.009943, over 3034968.57 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:44:36,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1470153.3333333333, ans=0.2
2023-11-21 10:44:39,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1470153.3333333333, ans=0.2
2023-11-21 10:44:40,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1470153.3333333333, ans=0.1
2023-11-21 10:44:55,160 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.123e+01 8.556e+01 9.611e+01 1.349e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 10:45:02,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220550
2023-11-21 10:45:32,344 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.24 vs. limit=22.5
2023-11-21 10:45:34,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.61 vs. limit=15.0
2023-11-21 10:45:35,379 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4150, loss[loss=0.07988, simple_loss=0.1003, pruned_loss=0.02144, audio_tagging_loss=0.008289, over 14744.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.09713, pruned_loss=0.01702, audio_tagging_loss=0.009809, over 3039816.90 frames. ], batch size: 56, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:45:42,145 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.66 vs. limit=15.0
2023-11-21 10:45:57,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1470553.3333333333, ans=0.125
2023-11-21 10:46:00,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1470620.0, ans=0.1
2023-11-21 10:46:06,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1470620.0, ans=0.0
2023-11-21 10:46:07,504 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220600
2023-11-21 10:46:22,617 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:46:31,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1470753.3333333333, ans=0.1
2023-11-21 10:46:37,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1470753.3333333333, ans=0.125
2023-11-21 10:46:39,655 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4200, loss[loss=0.0775, simple_loss=0.0961, pruned_loss=0.01906, audio_tagging_loss=0.01038, over 15621.00 frames. ], tot_loss[loss=0.07516, simple_loss=0.09712, pruned_loss=0.01693, audio_tagging_loss=0.00967, over 3044951.19 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:46:42,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1470820.0, ans=0.05
2023-11-21 10:46:52,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1470886.6666666667, ans=0.0
2023-11-21 10:47:03,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1470886.6666666667, ans=0.125
2023-11-21 10:47:06,455 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.111e+01 8.714e+01 9.250e+01 1.135e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 10:47:06,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1470953.3333333333, ans=0.0
2023-11-21 10:47:12,890 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220650
2023-11-21 10:47:44,734 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4250, loss[loss=0.0696, simple_loss=0.09576, pruned_loss=0.01417, audio_tagging_loss=0.007545, over 15584.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09775, pruned_loss=0.01714, audio_tagging_loss=0.009608, over 3049231.61 frames. ], batch size: 56, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:47:44,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1471153.3333333333, ans=0.0
2023-11-21 10:47:50,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1471153.3333333333, ans=0.125
2023-11-21 10:48:14,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1471286.6666666667, ans=0.09899494936611666
2023-11-21 10:48:17,295 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220700
2023-11-21 10:48:42,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1471420.0, ans=0.0
2023-11-21 10:48:50,612 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4300, loss[loss=0.08162, simple_loss=0.1079, pruned_loss=0.01884, audio_tagging_loss=0.008822, over 16593.00 frames. ], tot_loss[loss=0.07581, simple_loss=0.09813, pruned_loss=0.01725, audio_tagging_loss=0.009501, over 3050394.55 frames. ], batch size: 60, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:48:55,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1471486.6666666667, ans=0.04949747468305833
2023-11-21 10:49:10,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1471553.3333333333, ans=0.125
2023-11-21 10:49:12,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1471553.3333333333, ans=0.125
2023-11-21 10:49:12,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1471553.3333333333, ans=0.0
2023-11-21 10:49:14,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1471620.0, ans=0.125
2023-11-21 10:49:15,451 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.061e+01 8.689e+01 9.437e+01 1.140e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 10:49:15,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1471620.0, ans=0.0
2023-11-21 10:49:22,372 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220750
2023-11-21 10:49:35,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1471686.6666666667, ans=0.125
2023-11-21 10:49:40,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1471686.6666666667, ans=0.0
2023-11-21 10:49:42,980 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.41 vs. limit=15.0
2023-11-21 10:49:54,718 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4350, loss[loss=0.0785, simple_loss=0.1006, pruned_loss=0.02012, audio_tagging_loss=0.008098, over 13575.00 frames. ], tot_loss[loss=0.07522, simple_loss=0.09737, pruned_loss=0.01707, audio_tagging_loss=0.00946, over 3048751.44 frames. ], batch size: 52, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:50:02,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1471820.0, ans=0.125
2023-11-21 10:50:23,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1471953.3333333333, ans=0.125
2023-11-21 10:50:28,501 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220800
2023-11-21 10:50:31,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1471953.3333333333, ans=0.0
2023-11-21 10:50:56,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1472086.6666666667, ans=0.0
2023-11-21 10:51:00,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1472153.3333333333, ans=0.0
2023-11-21 10:51:00,752 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4400, loss[loss=0.06994, simple_loss=0.09018, pruned_loss=0.01374, audio_tagging_loss=0.01111, over 16033.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09795, pruned_loss=0.01715, audio_tagging_loss=0.009405, over 3049166.65 frames. ], batch size: 59, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 10:51:06,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1472153.3333333333, ans=0.09899494936611666
2023-11-21 10:51:26,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1472286.6666666667, ans=0.1
2023-11-21 10:51:27,109 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.481e+01 8.151e+01 8.586e+01 9.427e+01 1.147e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-21 10:51:33,379 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220850
2023-11-21 10:51:44,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1472353.3333333333, ans=0.2
2023-11-21 10:52:06,471 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4450, loss[loss=0.0756, simple_loss=0.09364, pruned_loss=0.01549, audio_tagging_loss=0.01328, over 14981.00 frames. ], tot_loss[loss=0.07628, simple_loss=0.0991, pruned_loss=0.01741, audio_tagging_loss=0.009321, over 3048415.34 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 10:52:23,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1472553.3333333333, ans=0.1
2023-11-21 10:52:38,065 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220900
2023-11-21 10:52:56,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1472686.6666666667, ans=0.1
2023-11-21 10:53:11,328 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4500, loss[loss=0.08096, simple_loss=0.1089, pruned_loss=0.01892, audio_tagging_loss=0.007589, over 16515.00 frames. ], tot_loss[loss=0.07626, simple_loss=0.09914, pruned_loss=0.01735, audio_tagging_loss=0.009337, over 3062992.21 frames. ], batch size: 62, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 10:53:38,068 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.114e+01 8.781e+01 9.396e+01 1.182e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-21 10:53:45,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 220950
2023-11-21 10:54:08,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.42 vs. limit=15.0
2023-11-21 10:54:11,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1473086.6666666667, ans=10.0
2023-11-21 10:54:16,246 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4550, loss[loss=0.07039, simple_loss=0.08792, pruned_loss=0.01899, audio_tagging_loss=0.007444, over 14935.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09788, pruned_loss=0.01706, audio_tagging_loss=0.009385, over 3055056.76 frames. ], batch size: 56, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 10:54:19,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1473153.3333333333, ans=0.125
2023-11-21 10:54:33,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1473220.0, ans=0.1
2023-11-21 10:54:37,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.87 vs. limit=15.0
2023-11-21 10:54:40,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1473220.0, ans=0.1
2023-11-21 10:54:49,104 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221000
2023-11-21 10:54:49,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1473286.6666666667, ans=0.125
2023-11-21 10:54:50,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1473286.6666666667, ans=0.2
2023-11-21 10:54:50,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1473286.6666666667, ans=0.1
2023-11-21 10:54:55,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1473353.3333333333, ans=0.0
2023-11-21 10:55:00,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1473353.3333333333, ans=0.125
2023-11-21 10:55:00,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1473353.3333333333, ans=0.125
2023-11-21 10:55:05,205 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 10:55:12,619 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.66 vs. limit=6.0
2023-11-21 10:55:22,531 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4600, loss[loss=0.09629, simple_loss=0.1274, pruned_loss=0.02369, audio_tagging_loss=0.008885, over 14603.00 frames. ], tot_loss[loss=0.07516, simple_loss=0.09727, pruned_loss=0.01696, audio_tagging_loss=0.009561, over 3052916.69 frames. ], batch size: 54, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:55:22,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1473486.6666666667, ans=10.0
2023-11-21 10:55:30,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1473486.6666666667, ans=0.0
2023-11-21 10:55:39,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1473553.3333333333, ans=0.0
2023-11-21 10:55:48,764 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.454e+01 8.163e+01 8.661e+01 9.433e+01 1.192e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 10:55:53,762 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221050
2023-11-21 10:56:03,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1473686.6666666667, ans=0.0
2023-11-21 10:56:03,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1473686.6666666667, ans=0.125
2023-11-21 10:56:12,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1473686.6666666667, ans=0.125
2023-11-21 10:56:20,131 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.12 vs. limit=22.5
2023-11-21 10:56:27,010 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4650, loss[loss=0.04936, simple_loss=0.05047, pruned_loss=0.01089, audio_tagging_loss=0.01324, over 14424.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09711, pruned_loss=0.01715, audio_tagging_loss=0.009674, over 3057450.35 frames. ], batch size: 59, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:56:35,009 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.71 vs. limit=15.0
2023-11-21 10:56:39,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1473886.6666666667, ans=0.125
2023-11-21 10:56:54,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1473953.3333333333, ans=0.2
2023-11-21 10:56:59,985 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221100
2023-11-21 10:57:08,616 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.54 vs. limit=15.0
2023-11-21 10:57:09,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1474020.0, ans=0.0
2023-11-21 10:57:21,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1474086.6666666667, ans=0.1
2023-11-21 10:57:25,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1474086.6666666667, ans=0.0
2023-11-21 10:57:31,384 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4700, loss[loss=0.08583, simple_loss=0.1093, pruned_loss=0.02281, audio_tagging_loss=0.008358, over 16838.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.0967, pruned_loss=0.01705, audio_tagging_loss=0.009702, over 3051682.10 frames. ], batch size: 61, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:57:35,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=4.28 vs. limit=15.0
2023-11-21 10:57:36,981 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.67 vs. limit=15.0
2023-11-21 10:57:48,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1474220.0, ans=0.125
2023-11-21 10:57:53,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1474220.0, ans=0.125
2023-11-21 10:57:59,446 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.726e+01 8.412e+01 8.839e+01 9.940e+01 1.268e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 10:58:04,076 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.87 vs. limit=22.5
2023-11-21 10:58:04,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221150
2023-11-21 10:58:15,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1474353.3333333333, ans=0.1
2023-11-21 10:58:35,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1474420.0, ans=0.0
2023-11-21 10:58:37,198 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4750, loss[loss=0.1048, simple_loss=0.1357, pruned_loss=0.02683, audio_tagging_loss=0.01013, over 13918.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09715, pruned_loss=0.01702, audio_tagging_loss=0.009866, over 3044254.86 frames. ], batch size: 52, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 10:58:49,904 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.63 vs. limit=15.0
2023-11-21 10:58:51,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1474553.3333333333, ans=0.0
2023-11-21 10:59:09,406 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221200
2023-11-21 10:59:43,347 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4800, loss[loss=0.08401, simple_loss=0.1039, pruned_loss=0.01599, audio_tagging_loss=0.01609, over 15584.00 frames. ], tot_loss[loss=0.07602, simple_loss=0.09783, pruned_loss=0.0172, audio_tagging_loss=0.00991, over 3047117.60 frames. ], batch size: 58, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 10:59:54,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1474886.6666666667, ans=0.035
2023-11-21 10:59:57,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1474886.6666666667, ans=0.1
2023-11-21 10:59:58,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1474886.6666666667, ans=0.2
2023-11-21 11:00:02,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=1474886.6666666667, ans=0.025
2023-11-21 11:00:02,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1474886.6666666667, ans=0.125
2023-11-21 11:00:06,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1474886.6666666667, ans=0.125
2023-11-21 11:00:07,818 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.98 vs. limit=6.0
2023-11-21 11:00:10,187 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.162e+01 8.841e+01 9.779e+01 1.301e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 11:00:15,879 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221250
2023-11-21 11:00:16,134 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:00:20,709 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.76 vs. limit=15.0
2023-11-21 11:00:45,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1475086.6666666667, ans=0.125
2023-11-21 11:00:47,736 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4850, loss[loss=0.0851, simple_loss=0.1062, pruned_loss=0.02427, audio_tagging_loss=0.007719, over 16737.00 frames. ], tot_loss[loss=0.07612, simple_loss=0.09768, pruned_loss=0.01736, audio_tagging_loss=0.009919, over 3043121.39 frames. ], batch size: 61, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:00:52,319 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.66 vs. limit=15.0
2023-11-21 11:01:11,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1475220.0, ans=0.125
2023-11-21 11:01:21,610 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221300
2023-11-21 11:01:21,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1475286.6666666667, ans=0.125
2023-11-21 11:01:31,968 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.53 vs. limit=15.0
2023-11-21 11:01:37,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1475353.3333333333, ans=0.125
2023-11-21 11:01:52,958 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4900, loss[loss=0.08703, simple_loss=0.1043, pruned_loss=0.02579, audio_tagging_loss=0.009069, over 14307.00 frames. ], tot_loss[loss=0.0764, simple_loss=0.09839, pruned_loss=0.01748, audio_tagging_loss=0.009735, over 3042810.10 frames. ], batch size: 54, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:02:00,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1475486.6666666667, ans=0.0
2023-11-21 11:02:10,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1475553.3333333333, ans=0.0
2023-11-21 11:02:18,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1475620.0, ans=0.1
2023-11-21 11:02:19,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1475620.0, ans=0.0
2023-11-21 11:02:20,760 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 8.034e+01 8.624e+01 9.399e+01 1.492e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-21 11:02:24,596 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221350
2023-11-21 11:02:57,371 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 4950, loss[loss=0.07123, simple_loss=0.09924, pruned_loss=0.01363, audio_tagging_loss=0.007986, over 15260.00 frames. ], tot_loss[loss=0.0759, simple_loss=0.09781, pruned_loss=0.01733, audio_tagging_loss=0.009665, over 3040659.42 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:03:06,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1475820.0, ans=0.125
2023-11-21 11:03:07,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1475820.0, ans=0.0
2023-11-21 11:03:15,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1475886.6666666667, ans=0.0
2023-11-21 11:03:19,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1475886.6666666667, ans=0.125
2023-11-21 11:03:29,686 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221400
2023-11-21 11:03:45,807 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:03:49,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1476086.6666666667, ans=0.125
2023-11-21 11:03:54,329 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.18 vs. limit=8.0
2023-11-21 11:03:56,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1476086.6666666667, ans=0.125
2023-11-21 11:04:02,076 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5000, loss[loss=0.08238, simple_loss=0.1116, pruned_loss=0.018, audio_tagging_loss=0.008572, over 15399.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09763, pruned_loss=0.01703, audio_tagging_loss=0.009589, over 3040690.95 frames. ], batch size: 54, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:04:08,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1476153.3333333333, ans=0.5
2023-11-21 11:04:20,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.67 vs. limit=15.0
2023-11-21 11:04:23,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1476220.0, ans=0.1
2023-11-21 11:04:29,665 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.95 vs. limit=22.5
2023-11-21 11:04:31,466 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.334e+01 8.366e+01 9.036e+01 1.004e+02 1.239e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 11:04:33,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1476286.6666666667, ans=0.2
2023-11-21 11:04:35,414 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221450
2023-11-21 11:04:38,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1476286.6666666667, ans=0.0
2023-11-21 11:04:39,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1476286.6666666667, ans=0.125
2023-11-21 11:04:44,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1476353.3333333333, ans=0.125
2023-11-21 11:04:53,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1476420.0, ans=0.0
2023-11-21 11:05:04,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1476420.0, ans=0.1
2023-11-21 11:05:07,096 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5050, loss[loss=0.07763, simple_loss=0.1014, pruned_loss=0.0171, audio_tagging_loss=0.009858, over 14573.00 frames. ], tot_loss[loss=0.07505, simple_loss=0.09744, pruned_loss=0.01682, audio_tagging_loss=0.009502, over 3043910.84 frames. ], batch size: 52, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:05:39,939 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221500
2023-11-21 11:05:51,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1476686.6666666667, ans=0.1
2023-11-21 11:05:55,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1476686.6666666667, ans=0.125
2023-11-21 11:05:58,902 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:06:09,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1476753.3333333333, ans=0.1
2023-11-21 11:06:09,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1476753.3333333333, ans=0.0
2023-11-21 11:06:12,966 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5100, loss[loss=0.05376, simple_loss=0.07402, pruned_loss=0.008933, audio_tagging_loss=0.007811, over 15227.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09621, pruned_loss=0.01643, audio_tagging_loss=0.009547, over 3037266.84 frames. ], batch size: 56, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:06:15,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1476820.0, ans=0.0
2023-11-21 11:06:36,247 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=8.780e-02
2023-11-21 11:06:37,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1476953.3333333333, ans=0.125
2023-11-21 11:06:40,837 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.039e+01 8.516e+01 9.144e+01 1.339e+02, threshold=1.703e+02, percent-clipped=0.0
2023-11-21 11:06:45,269 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221550
2023-11-21 11:06:51,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1477020.0, ans=0.0
2023-11-21 11:07:01,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1477020.0, ans=0.0
2023-11-21 11:07:18,251 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5150, loss[loss=0.08162, simple_loss=0.1089, pruned_loss=0.01846, audio_tagging_loss=0.008724, over 14394.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09574, pruned_loss=0.01639, audio_tagging_loss=0.009635, over 3038786.18 frames. ], batch size: 55, lr: 3.62e-03, grad_scale: 16.0
2023-11-21 11:07:46,852 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.20 vs. limit=22.5
2023-11-21 11:07:51,158 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221600
2023-11-21 11:07:54,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1477286.6666666667, ans=0.125
2023-11-21 11:08:23,357 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5200, loss[loss=0.06763, simple_loss=0.09133, pruned_loss=0.01254, audio_tagging_loss=0.009429, over 15394.00 frames. ], tot_loss[loss=0.07391, simple_loss=0.09585, pruned_loss=0.01644, audio_tagging_loss=0.00954, over 3043132.40 frames. ], batch size: 58, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 11:08:47,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1477553.3333333333, ans=0.0
2023-11-21 11:08:51,494 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.310e+01 8.141e+01 8.886e+01 9.437e+01 1.227e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 11:08:53,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1477620.0, ans=0.1
2023-11-21 11:08:55,948 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221650
2023-11-21 11:08:59,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1477620.0, ans=0.0
2023-11-21 11:09:28,801 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5250, loss[loss=0.08228, simple_loss=0.09858, pruned_loss=0.02149, audio_tagging_loss=0.01149, over 15730.00 frames. ], tot_loss[loss=0.07442, simple_loss=0.09642, pruned_loss=0.01667, audio_tagging_loss=0.009542, over 3042475.76 frames. ], batch size: 58, lr: 3.62e-03, grad_scale: 32.0
2023-11-21 11:09:33,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1477820.0, ans=0.125
2023-11-21 11:09:58,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1477953.3333333333, ans=0.2
2023-11-21 11:10:00,078 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221700
2023-11-21 11:10:04,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1477953.3333333333, ans=0.125
2023-11-21 11:10:15,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1478020.0, ans=0.035
2023-11-21 11:10:24,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1478086.6666666667, ans=0.125
2023-11-21 11:10:32,519 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5300, loss[loss=0.07213, simple_loss=0.1031, pruned_loss=0.01273, audio_tagging_loss=0.00788, over 15109.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.09638, pruned_loss=0.01665, audio_tagging_loss=0.00948, over 3045446.77 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:10:38,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1478153.3333333333, ans=0.1
2023-11-21 11:10:50,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1478220.0, ans=0.0
2023-11-21 11:11:01,615 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.106e+01 8.119e+01 8.657e+01 9.101e+01 1.165e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 11:11:06,039 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221750
2023-11-21 11:11:25,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1478420.0, ans=0.1
2023-11-21 11:11:25,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1478420.0, ans=0.2
2023-11-21 11:11:28,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1478420.0, ans=0.125
2023-11-21 11:11:36,915 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5350, loss[loss=0.07745, simple_loss=0.1001, pruned_loss=0.01615, audio_tagging_loss=0.01127, over 14926.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09715, pruned_loss=0.01687, audio_tagging_loss=0.00953, over 3045361.92 frames. ], batch size: 54, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:11:49,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1478553.3333333333, ans=0.5
2023-11-21 11:12:09,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221800
2023-11-21 11:12:13,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1478620.0, ans=0.09899494936611666
2023-11-21 11:12:17,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1478686.6666666667, ans=0.125
2023-11-21 11:12:35,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1478753.3333333333, ans=0.2
2023-11-21 11:12:42,763 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5400, loss[loss=0.07819, simple_loss=0.09455, pruned_loss=0.02047, audio_tagging_loss=0.01044, over 14303.00 frames. ], tot_loss[loss=0.07481, simple_loss=0.09686, pruned_loss=0.01682, audio_tagging_loss=0.009562, over 3044082.44 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:12:44,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1478820.0, ans=0.125
2023-11-21 11:12:45,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1478820.0, ans=0.0
2023-11-21 11:12:48,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.58 vs. limit=10.0
2023-11-21 11:13:02,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1478886.6666666667, ans=0.2
2023-11-21 11:13:11,526 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.155e+01 8.706e+01 9.355e+01 1.764e+02, threshold=1.741e+02, percent-clipped=1.0
2023-11-21 11:13:14,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221850
2023-11-21 11:13:29,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1479020.0, ans=0.125
2023-11-21 11:13:40,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1479086.6666666667, ans=0.125
2023-11-21 11:13:42,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1479086.6666666667, ans=0.0
2023-11-21 11:13:43,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1479086.6666666667, ans=0.0
2023-11-21 11:13:46,683 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5450, loss[loss=0.08397, simple_loss=0.1143, pruned_loss=0.0186, audio_tagging_loss=0.008214, over 15613.00 frames. ], tot_loss[loss=0.07503, simple_loss=0.09684, pruned_loss=0.01698, audio_tagging_loss=0.009639, over 3043798.21 frames. ], batch size: 56, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:14:02,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1479220.0, ans=0.125
2023-11-21 11:14:06,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1479220.0, ans=0.125
2023-11-21 11:14:08,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1479220.0, ans=0.0
2023-11-21 11:14:13,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1479286.6666666667, ans=0.05
2023-11-21 11:14:19,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221900
2023-11-21 11:14:36,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1479420.0, ans=0.125
2023-11-21 11:14:40,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1479420.0, ans=0.0
2023-11-21 11:14:50,981 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5500, loss[loss=0.07217, simple_loss=0.09622, pruned_loss=0.01412, audio_tagging_loss=0.009933, over 15233.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09637, pruned_loss=0.01688, audio_tagging_loss=0.009689, over 3041660.74 frames. ], batch size: 54, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:15:07,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1479553.3333333333, ans=0.125
2023-11-21 11:15:15,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1479553.3333333333, ans=0.0
2023-11-21 11:15:21,344 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.777e+01 8.123e+01 9.057e+01 9.951e+01 1.227e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-21 11:15:23,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 221950
2023-11-21 11:15:36,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1479686.6666666667, ans=0.0
2023-11-21 11:15:47,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1479753.3333333333, ans=0.1
2023-11-21 11:15:56,279 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5550, loss[loss=0.08681, simple_loss=0.1156, pruned_loss=0.01963, audio_tagging_loss=0.009352, over 15480.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09757, pruned_loss=0.01707, audio_tagging_loss=0.00959, over 3043603.81 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:16:02,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1479820.0, ans=0.1
2023-11-21 11:16:07,202 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.66 vs. limit=22.5
2023-11-21 11:16:27,020 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222000
2023-11-21 11:16:49,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1480086.6666666667, ans=0.1
2023-11-21 11:16:50,881 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.30 vs. limit=22.5
2023-11-21 11:16:57,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1480086.6666666667, ans=0.0
2023-11-21 11:17:00,034 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5600, loss[loss=0.0731, simple_loss=0.09084, pruned_loss=0.01557, audio_tagging_loss=0.01211, over 16093.00 frames. ], tot_loss[loss=0.07525, simple_loss=0.09735, pruned_loss=0.01687, audio_tagging_loss=0.0097, over 3046214.27 frames. ], batch size: 60, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:17:01,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1480153.3333333333, ans=0.1
2023-11-21 11:17:10,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1480153.3333333333, ans=0.0
2023-11-21 11:17:11,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1480220.0, ans=0.1
2023-11-21 11:17:13,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1480220.0, ans=0.1
2023-11-21 11:17:14,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1480220.0, ans=0.0
2023-11-21 11:17:30,542 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.739e+01 7.938e+01 8.660e+01 9.316e+01 1.150e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 11:17:31,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222050
2023-11-21 11:17:44,698 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 11:18:02,767 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5650, loss[loss=0.08726, simple_loss=0.1087, pruned_loss=0.02494, audio_tagging_loss=0.007944, over 14943.00 frames. ], tot_loss[loss=0.07512, simple_loss=0.09689, pruned_loss=0.01686, audio_tagging_loss=0.009813, over 3049736.61 frames. ], batch size: 54, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:18:06,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1480486.6666666667, ans=0.0
2023-11-21 11:18:35,906 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222100
2023-11-21 11:18:37,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1480620.0, ans=0.125
2023-11-21 11:18:52,355 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.41 vs. limit=15.0
2023-11-21 11:19:07,299 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5700, loss[loss=0.06725, simple_loss=0.07865, pruned_loss=0.01785, audio_tagging_loss=0.01007, over 15230.00 frames. ], tot_loss[loss=0.07539, simple_loss=0.09697, pruned_loss=0.0171, audio_tagging_loss=0.009807, over 3046866.91 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:19:28,324 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.60 vs. limit=6.0
2023-11-21 11:19:37,371 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.400e+01 9.034e+01 9.970e+01 1.319e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 11:19:37,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1480953.3333333333, ans=0.0
2023-11-21 11:19:38,769 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222150
2023-11-21 11:20:11,600 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5750, loss[loss=0.06833, simple_loss=0.09264, pruned_loss=0.01335, audio_tagging_loss=0.008659, over 14347.00 frames. ], tot_loss[loss=0.07562, simple_loss=0.09713, pruned_loss=0.01736, audio_tagging_loss=0.00969, over 3046226.60 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:20:43,237 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222200
2023-11-21 11:20:58,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1481353.3333333333, ans=0.125
2023-11-21 11:21:11,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1481420.0, ans=0.0
2023-11-21 11:21:14,442 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.51 vs. limit=15.0
2023-11-21 11:21:14,847 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5800, loss[loss=0.06609, simple_loss=0.0795, pruned_loss=0.01501, audio_tagging_loss=0.01133, over 14790.00 frames. ], tot_loss[loss=0.07462, simple_loss=0.09571, pruned_loss=0.01707, audio_tagging_loss=0.009699, over 3044336.48 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:21:21,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1481486.6666666667, ans=0.0
2023-11-21 11:21:46,273 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.129e+01 8.053e+01 8.570e+01 9.375e+01 1.179e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 11:21:47,712 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222250
2023-11-21 11:21:56,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1481686.6666666667, ans=0.125
2023-11-21 11:22:01,519 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.91 vs. limit=10.0
2023-11-21 11:22:17,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1481820.0, ans=0.035
2023-11-21 11:22:17,893 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.07 vs. limit=10.0
2023-11-21 11:22:18,591 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5850, loss[loss=0.0914, simple_loss=0.1191, pruned_loss=0.02343, audio_tagging_loss=0.008427, over 14175.00 frames. ], tot_loss[loss=0.07499, simple_loss=0.09658, pruned_loss=0.01722, audio_tagging_loss=0.009476, over 3042714.43 frames. ], batch size: 54, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:22:50,183 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222300
2023-11-21 11:22:51,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1481953.3333333333, ans=0.125
2023-11-21 11:23:16,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1482086.6666666667, ans=0.0
2023-11-21 11:23:22,448 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5900, loss[loss=0.08065, simple_loss=0.1017, pruned_loss=0.0213, audio_tagging_loss=0.008509, over 15028.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.09657, pruned_loss=0.01709, audio_tagging_loss=0.009464, over 3039756.01 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:23:26,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1482153.3333333333, ans=0.0
2023-11-21 11:23:51,658 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.554e+01 8.184e+01 8.891e+01 9.441e+01 1.396e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 11:23:52,980 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222350
2023-11-21 11:23:54,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1482286.6666666667, ans=0.1
2023-11-21 11:23:57,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1482286.6666666667, ans=0.125
2023-11-21 11:24:01,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1482353.3333333333, ans=0.125
2023-11-21 11:24:17,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1482420.0, ans=0.2
2023-11-21 11:24:18,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1482420.0, ans=0.0
2023-11-21 11:24:24,463 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 5950, loss[loss=0.06622, simple_loss=0.09236, pruned_loss=0.01245, audio_tagging_loss=0.007593, over 15896.00 frames. ], tot_loss[loss=0.07515, simple_loss=0.09726, pruned_loss=0.01711, audio_tagging_loss=0.009411, over 3045757.39 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:24:24,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1482486.6666666667, ans=0.0
2023-11-21 11:24:42,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1482553.3333333333, ans=0.2
2023-11-21 11:24:50,833 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.69 vs. limit=22.5
2023-11-21 11:24:57,001 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222400
2023-11-21 11:25:02,453 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=8.004e-02
2023-11-21 11:25:08,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1482686.6666666667, ans=0.125
2023-11-21 11:25:28,516 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6000, loss[loss=0.0771, simple_loss=0.1117, pruned_loss=0.01404, audio_tagging_loss=0.007206, over 15155.00 frames. ], tot_loss[loss=0.07486, simple_loss=0.09675, pruned_loss=0.01697, audio_tagging_loss=0.009512, over 3039781.02 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:25:28,519 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 11:26:03,357 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.3985, 3.1906, 2.8583, 2.7803, 3.4202, 3.5239, 3.1601, 3.6538],
       device='cuda:0')
2023-11-21 11:26:10,017 INFO [train_asr.py:1253] (0/4) Epoch 19, validation: loss=0.05983, simple_loss=0.05236, pruned_loss=0.005293, audio_tagging_loss=0.02835, over 4681554.00 frames. 
2023-11-21 11:26:10,018 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 11:26:12,154 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.42 vs. limit=8.0
2023-11-21 11:26:25,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1482886.6666666667, ans=0.0
2023-11-21 11:26:40,199 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.474e+01 7.915e+01 8.568e+01 9.449e+01 1.142e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 11:26:41,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222450
2023-11-21 11:26:49,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1483020.0, ans=0.1
2023-11-21 11:26:52,200 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.65 vs. limit=15.0
2023-11-21 11:26:55,914 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 11:27:12,939 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6050, loss[loss=0.03744, simple_loss=0.04274, pruned_loss=0.006117, audio_tagging_loss=0.00995, over 14361.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.09625, pruned_loss=0.01686, audio_tagging_loss=0.009482, over 3037539.90 frames. ], batch size: 55, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:27:28,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1483220.0, ans=0.125
2023-11-21 11:27:37,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1483286.6666666667, ans=0.2
2023-11-21 11:27:45,767 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222500
2023-11-21 11:28:14,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1483420.0, ans=0.1
2023-11-21 11:28:16,920 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6100, loss[loss=0.07722, simple_loss=0.09995, pruned_loss=0.01596, audio_tagging_loss=0.01129, over 15880.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.09614, pruned_loss=0.01693, audio_tagging_loss=0.009483, over 3039008.99 frames. ], batch size: 60, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:28:24,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1483486.6666666667, ans=0.125
2023-11-21 11:28:32,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1483553.3333333333, ans=0.125
2023-11-21 11:28:45,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1483620.0, ans=0.125
2023-11-21 11:28:47,597 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.060e+01 8.165e+01 8.686e+01 9.448e+01 1.325e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 11:28:48,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222550
2023-11-21 11:29:07,776 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.26 vs. limit=6.0
2023-11-21 11:29:15,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1483753.3333333333, ans=0.1
2023-11-21 11:29:17,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1483753.3333333333, ans=0.2
2023-11-21 11:29:21,024 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6150, loss[loss=0.04954, simple_loss=0.05955, pruned_loss=0.006012, audio_tagging_loss=0.01376, over 15321.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09665, pruned_loss=0.01688, audio_tagging_loss=0.009531, over 3041879.00 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:29:39,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1483886.6666666667, ans=0.125
2023-11-21 11:29:50,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1483953.3333333333, ans=0.0
2023-11-21 11:29:52,912 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222600
2023-11-21 11:30:00,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1484020.0, ans=0.125
2023-11-21 11:30:05,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1484020.0, ans=0.2
2023-11-21 11:30:08,443 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.88 vs. limit=6.0
2023-11-21 11:30:14,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1484086.6666666667, ans=0.125
2023-11-21 11:30:24,566 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.45 vs. limit=15.0
2023-11-21 11:30:25,228 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6200, loss[loss=0.05344, simple_loss=0.05882, pruned_loss=0.01204, audio_tagging_loss=0.01198, over 15304.00 frames. ], tot_loss[loss=0.07537, simple_loss=0.09748, pruned_loss=0.01712, audio_tagging_loss=0.009512, over 3048636.18 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:30:53,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1484286.6666666667, ans=0.125
2023-11-21 11:30:55,994 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.122e+01 8.795e+01 9.552e+01 1.328e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 11:30:57,324 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222650
2023-11-21 11:31:04,841 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.52 vs. limit=22.5
2023-11-21 11:31:10,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1484353.3333333333, ans=0.125
2023-11-21 11:31:10,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1484353.3333333333, ans=0.125
2023-11-21 11:31:28,750 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6250, loss[loss=0.07188, simple_loss=0.08632, pruned_loss=0.01739, audio_tagging_loss=0.01133, over 15504.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09715, pruned_loss=0.01713, audio_tagging_loss=0.009629, over 3047192.92 frames. ], batch size: 59, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:32:01,088 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222700
2023-11-21 11:32:06,517 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.76 vs. limit=12.0
2023-11-21 11:32:07,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1484686.6666666667, ans=0.1
2023-11-21 11:32:33,433 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6300, loss[loss=0.1001, simple_loss=0.1317, pruned_loss=0.0257, audio_tagging_loss=0.008528, over 14780.00 frames. ], tot_loss[loss=0.07608, simple_loss=0.09838, pruned_loss=0.01728, audio_tagging_loss=0.009607, over 3049475.10 frames. ], batch size: 58, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:32:50,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1484886.6666666667, ans=0.1
2023-11-21 11:33:03,726 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.093e+01 8.788e+01 9.540e+01 1.292e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 11:33:04,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1484953.3333333333, ans=0.125
2023-11-21 11:33:05,073 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222750
2023-11-21 11:33:20,437 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.72 vs. limit=15.0
2023-11-21 11:33:37,372 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6350, loss[loss=0.09671, simple_loss=0.1295, pruned_loss=0.02306, audio_tagging_loss=0.00891, over 14916.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09826, pruned_loss=0.01717, audio_tagging_loss=0.009801, over 3053521.23 frames. ], batch size: 57, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:34:04,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1485286.6666666667, ans=0.1
2023-11-21 11:34:09,954 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222800
2023-11-21 11:34:16,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1485353.3333333333, ans=10.0
2023-11-21 11:34:37,128 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.89 vs. limit=15.0
2023-11-21 11:34:41,328 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.79 vs. limit=15.0
2023-11-21 11:34:41,965 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6400, loss[loss=0.08332, simple_loss=0.1093, pruned_loss=0.02022, audio_tagging_loss=0.008468, over 13717.00 frames. ], tot_loss[loss=0.0756, simple_loss=0.09736, pruned_loss=0.01705, audio_tagging_loss=0.009869, over 3046106.33 frames. ], batch size: 53, lr: 3.61e-03, grad_scale: 32.0
2023-11-21 11:34:42,613 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.00 vs. limit=10.0
2023-11-21 11:34:52,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1485486.6666666667, ans=0.125
2023-11-21 11:35:12,457 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.327e+01 9.131e+01 1.015e+02 1.534e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-21 11:35:13,769 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222850
2023-11-21 11:35:39,286 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.21 vs. limit=22.5
2023-11-21 11:35:42,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1485753.3333333333, ans=0.125
2023-11-21 11:35:45,840 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.92 vs. limit=15.0
2023-11-21 11:35:46,478 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6450, loss[loss=0.1041, simple_loss=0.1347, pruned_loss=0.0266, audio_tagging_loss=0.01015, over 14333.00 frames. ], tot_loss[loss=0.07564, simple_loss=0.09729, pruned_loss=0.01715, audio_tagging_loss=0.009842, over 3042381.77 frames. ], batch size: 52, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:36:02,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1485886.6666666667, ans=0.0
2023-11-21 11:36:13,573 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.59 vs. limit=12.0
2023-11-21 11:36:14,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1485953.3333333333, ans=0.0
2023-11-21 11:36:18,310 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222900
2023-11-21 11:36:18,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1485953.3333333333, ans=0.125
2023-11-21 11:36:44,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1486086.6666666667, ans=0.0
2023-11-21 11:36:45,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1486086.6666666667, ans=0.0
2023-11-21 11:36:49,992 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6500, loss[loss=0.07014, simple_loss=0.0914, pruned_loss=0.01467, audio_tagging_loss=0.009765, over 14446.00 frames. ], tot_loss[loss=0.0761, simple_loss=0.09788, pruned_loss=0.01736, audio_tagging_loss=0.009797, over 3044579.06 frames. ], batch size: 54, lr: 3.61e-03, grad_scale: 16.0
2023-11-21 11:37:03,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1486220.0, ans=0.125
2023-11-21 11:37:07,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1486220.0, ans=0.0
2023-11-21 11:37:23,025 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.393e+01 8.000e+01 8.623e+01 9.429e+01 1.650e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-21 11:37:23,175 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 222950
2023-11-21 11:37:27,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1486286.6666666667, ans=0.0
2023-11-21 11:37:37,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1486353.3333333333, ans=0.1
2023-11-21 11:37:44,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1486420.0, ans=0.125
2023-11-21 11:37:52,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1486420.0, ans=0.0
2023-11-21 11:37:54,471 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6550, loss[loss=0.06302, simple_loss=0.07912, pruned_loss=0.0132, audio_tagging_loss=0.01027, over 14557.00 frames. ], tot_loss[loss=0.07545, simple_loss=0.09743, pruned_loss=0.01708, audio_tagging_loss=0.009658, over 3046805.22 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:38:17,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1486553.3333333333, ans=0.125
2023-11-21 11:38:25,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1486620.0, ans=0.125
2023-11-21 11:38:26,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1486620.0, ans=0.125
2023-11-21 11:38:27,319 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223000
2023-11-21 11:38:27,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1486620.0, ans=0.0
2023-11-21 11:38:31,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1486620.0, ans=0.1
2023-11-21 11:38:32,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1486686.6666666667, ans=0.0
2023-11-21 11:38:50,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1486753.3333333333, ans=0.125
2023-11-21 11:38:53,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.41 vs. limit=15.0
2023-11-21 11:39:00,738 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6600, loss[loss=0.0779, simple_loss=0.1107, pruned_loss=0.01466, audio_tagging_loss=0.0079, over 14463.00 frames. ], tot_loss[loss=0.07514, simple_loss=0.0972, pruned_loss=0.01701, audio_tagging_loss=0.009526, over 3046067.12 frames. ], batch size: 54, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:39:13,427 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:39:14,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1486886.6666666667, ans=0.125
2023-11-21 11:39:26,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1486953.3333333333, ans=0.0
2023-11-21 11:39:29,303 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:39:32,147 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.360e+01 8.932e+01 9.626e+01 1.286e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 11:39:32,299 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223050
2023-11-21 11:39:54,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1487086.6666666667, ans=0.125
2023-11-21 11:40:04,836 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6650, loss[loss=0.06822, simple_loss=0.09444, pruned_loss=0.01157, audio_tagging_loss=0.00943, over 14564.00 frames. ], tot_loss[loss=0.07527, simple_loss=0.09742, pruned_loss=0.01704, audio_tagging_loss=0.009519, over 3045588.79 frames. ], batch size: 55, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:40:18,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.92 vs. limit=15.0
2023-11-21 11:40:21,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1487220.0, ans=0.125
2023-11-21 11:40:25,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1487220.0, ans=0.125
2023-11-21 11:40:36,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1487286.6666666667, ans=15.0
2023-11-21 11:40:37,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223100
2023-11-21 11:40:55,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1487420.0, ans=0.2
2023-11-21 11:40:56,359 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.92 vs. limit=22.5
2023-11-21 11:41:09,301 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6700, loss[loss=0.06459, simple_loss=0.07738, pruned_loss=0.01566, audio_tagging_loss=0.01023, over 14785.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09683, pruned_loss=0.01689, audio_tagging_loss=0.009451, over 3044967.34 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:41:31,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1487553.3333333333, ans=0.125
2023-11-21 11:41:42,374 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.137e+01 8.757e+01 9.597e+01 1.374e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 11:41:42,513 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223150
2023-11-21 11:42:12,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1487753.3333333333, ans=0.025
2023-11-21 11:42:14,882 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6750, loss[loss=0.07622, simple_loss=0.1071, pruned_loss=0.01478, audio_tagging_loss=0.007875, over 15018.00 frames. ], tot_loss[loss=0.07526, simple_loss=0.09756, pruned_loss=0.01705, audio_tagging_loss=0.009431, over 3043731.49 frames. ], batch size: 55, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:42:20,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1487820.0, ans=0.125
2023-11-21 11:42:21,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1487820.0, ans=0.0
2023-11-21 11:42:24,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1487820.0, ans=0.0
2023-11-21 11:42:25,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1487820.0, ans=0.0
2023-11-21 11:42:34,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1487886.6666666667, ans=0.1
2023-11-21 11:42:46,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223200
2023-11-21 11:42:55,259 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.36 vs. limit=15.0
2023-11-21 11:43:01,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1488020.0, ans=0.025
2023-11-21 11:43:09,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1488086.6666666667, ans=0.09899494936611666
2023-11-21 11:43:18,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1488086.6666666667, ans=0.125
2023-11-21 11:43:20,322 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6800, loss[loss=0.06135, simple_loss=0.08198, pruned_loss=0.008834, audio_tagging_loss=0.01153, over 15533.00 frames. ], tot_loss[loss=0.07525, simple_loss=0.09724, pruned_loss=0.01711, audio_tagging_loss=0.009529, over 3033879.34 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:43:24,755 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.20 vs. limit=15.0
2023-11-21 11:43:36,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1488220.0, ans=0.1
2023-11-21 11:43:37,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1488220.0, ans=0.0
2023-11-21 11:43:51,767 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.809e+01 8.121e+01 8.832e+01 9.588e+01 1.152e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 11:43:51,930 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223250
2023-11-21 11:44:06,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1488353.3333333333, ans=0.125
2023-11-21 11:44:13,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1488420.0, ans=0.125
2023-11-21 11:44:19,149 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.02 vs. limit=22.5
2023-11-21 11:44:23,471 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6850, loss[loss=0.06766, simple_loss=0.0831, pruned_loss=0.01571, audio_tagging_loss=0.0104, over 15365.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09778, pruned_loss=0.01721, audio_tagging_loss=0.009535, over 3028727.32 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:44:44,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1488553.3333333333, ans=0.04949747468305833
2023-11-21 11:44:49,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1488620.0, ans=0.09899494936611666
2023-11-21 11:44:56,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223300
2023-11-21 11:45:17,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1488753.3333333333, ans=0.0
2023-11-21 11:45:29,257 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6900, loss[loss=0.06355, simple_loss=0.07922, pruned_loss=0.01492, audio_tagging_loss=0.009017, over 15939.00 frames. ], tot_loss[loss=0.07538, simple_loss=0.09732, pruned_loss=0.01715, audio_tagging_loss=0.00956, over 3040293.56 frames. ], batch size: 60, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:45:33,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1488820.0, ans=0.125
2023-11-21 11:45:35,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1488820.0, ans=0.125
2023-11-21 11:45:38,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1488820.0, ans=0.125
2023-11-21 11:45:42,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=1488886.6666666667, ans=0.2
2023-11-21 11:45:44,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.17 vs. limit=22.5
2023-11-21 11:45:46,658 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.77 vs. limit=6.0
2023-11-21 11:45:49,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1488886.6666666667, ans=0.125
2023-11-21 11:45:52,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.41 vs. limit=15.0
2023-11-21 11:45:54,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1488953.3333333333, ans=0.125
2023-11-21 11:45:56,443 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.30 vs. limit=15.0
2023-11-21 11:45:57,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1488953.3333333333, ans=0.2
2023-11-21 11:46:00,719 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 8.015e+01 8.658e+01 9.630e+01 1.322e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 11:46:00,879 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223350
2023-11-21 11:46:05,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1489020.0, ans=0.125
2023-11-21 11:46:05,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1489020.0, ans=0.0
2023-11-21 11:46:19,835 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 11:46:34,104 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 6950, loss[loss=0.07566, simple_loss=0.09778, pruned_loss=0.01586, audio_tagging_loss=0.01091, over 14465.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09727, pruned_loss=0.01694, audio_tagging_loss=0.009526, over 3048326.93 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:46:42,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1489153.3333333333, ans=0.0
2023-11-21 11:46:50,704 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.68 vs. limit=15.0
2023-11-21 11:47:00,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1489286.6666666667, ans=0.1
2023-11-21 11:47:02,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.94 vs. limit=22.5
2023-11-21 11:47:05,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1489286.6666666667, ans=0.125
2023-11-21 11:47:06,142 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223400
2023-11-21 11:47:07,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1489286.6666666667, ans=0.125
2023-11-21 11:47:12,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1489353.3333333333, ans=0.07
2023-11-21 11:47:20,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1489353.3333333333, ans=0.04949747468305833
2023-11-21 11:47:33,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1489420.0, ans=0.125
2023-11-21 11:47:35,297 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.17 vs. limit=15.0
2023-11-21 11:47:38,463 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7000, loss[loss=0.07577, simple_loss=0.09593, pruned_loss=0.01923, audio_tagging_loss=0.008584, over 15471.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.0964, pruned_loss=0.01675, audio_tagging_loss=0.009569, over 3052642.80 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:47:49,023 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:47:49,474 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.55 vs. limit=15.0
2023-11-21 11:48:10,557 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 7.954e+01 8.531e+01 9.123e+01 1.113e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 11:48:10,700 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223450
2023-11-21 11:48:10,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1489620.0, ans=0.1
2023-11-21 11:48:26,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1489686.6666666667, ans=0.125
2023-11-21 11:48:31,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1489753.3333333333, ans=15.0
2023-11-21 11:48:35,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1489753.3333333333, ans=0.125
2023-11-21 11:48:42,250 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7050, loss[loss=0.06506, simple_loss=0.08279, pruned_loss=0.01392, audio_tagging_loss=0.009742, over 16121.00 frames. ], tot_loss[loss=0.07363, simple_loss=0.095, pruned_loss=0.01645, audio_tagging_loss=0.009685, over 3058546.90 frames. ], batch size: 62, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:48:53,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1489820.0, ans=0.04949747468305833
2023-11-21 11:48:59,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1489886.6666666667, ans=0.0
2023-11-21 11:49:13,867 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223500
2023-11-21 11:49:15,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1489953.3333333333, ans=0.125
2023-11-21 11:49:39,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1490086.6666666667, ans=0.1
2023-11-21 11:49:46,560 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7100, loss[loss=0.111, simple_loss=0.1465, pruned_loss=0.03146, audio_tagging_loss=0.00629, over 15967.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.0949, pruned_loss=0.01664, audio_tagging_loss=0.009848, over 3052709.95 frames. ], batch size: 55, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:49:59,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1490220.0, ans=0.1
2023-11-21 11:50:06,424 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=15.0
2023-11-21 11:50:16,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1490286.6666666667, ans=0.0
2023-11-21 11:50:16,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1490286.6666666667, ans=0.125
2023-11-21 11:50:17,521 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223550
2023-11-21 11:50:17,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1490286.6666666667, ans=0.125
2023-11-21 11:50:19,152 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.463e+01 8.140e+01 8.741e+01 9.619e+01 1.480e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 11:50:42,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1490420.0, ans=0.125
2023-11-21 11:50:43,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1490420.0, ans=0.1
2023-11-21 11:50:49,639 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7150, loss[loss=0.102, simple_loss=0.1337, pruned_loss=0.0243, audio_tagging_loss=0.01082, over 14660.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09449, pruned_loss=0.01663, audio_tagging_loss=0.009995, over 3050487.11 frames. ], batch size: 55, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:50:49,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1490486.6666666667, ans=0.125
2023-11-21 11:51:09,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1490553.3333333333, ans=0.125
2023-11-21 11:51:22,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223600
2023-11-21 11:51:46,516 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.11 vs. limit=12.0
2023-11-21 11:51:47,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1490753.3333333333, ans=0.0
2023-11-21 11:51:54,001 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7200, loss[loss=0.07199, simple_loss=0.08858, pruned_loss=0.01827, audio_tagging_loss=0.009427, over 14769.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.0954, pruned_loss=0.01677, audio_tagging_loss=0.009976, over 3043484.73 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:52:26,371 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223650
2023-11-21 11:52:26,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1490953.3333333333, ans=0.125
2023-11-21 11:52:27,424 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.414e+01 8.953e+01 9.739e+01 1.219e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-21 11:52:36,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1491020.0, ans=0.0
2023-11-21 11:52:39,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1491020.0, ans=0.0
2023-11-21 11:52:49,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1491086.6666666667, ans=0.125
2023-11-21 11:52:58,685 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7250, loss[loss=0.06842, simple_loss=0.09767, pruned_loss=0.01199, audio_tagging_loss=0.007596, over 15338.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09653, pruned_loss=0.01679, audio_tagging_loss=0.009959, over 3045067.21 frames. ], batch size: 55, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:53:16,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1491220.0, ans=0.0
2023-11-21 11:53:16,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1491220.0, ans=0.0
2023-11-21 11:53:19,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1491220.0, ans=0.125
2023-11-21 11:53:26,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1491286.6666666667, ans=0.2
2023-11-21 11:53:30,034 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223700
2023-11-21 11:54:02,166 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7300, loss[loss=0.07949, simple_loss=0.1074, pruned_loss=0.01833, audio_tagging_loss=0.007439, over 15172.00 frames. ], tot_loss[loss=0.07501, simple_loss=0.09678, pruned_loss=0.01677, audio_tagging_loss=0.00985, over 3041750.05 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:54:12,035 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 11:54:27,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1491620.0, ans=0.125
2023-11-21 11:54:29,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1491620.0, ans=0.125
2023-11-21 11:54:34,376 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223750
2023-11-21 11:54:35,393 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.195e+01 8.693e+01 9.496e+01 1.146e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 11:54:42,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1491686.6666666667, ans=0.1
2023-11-21 11:55:05,929 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7350, loss[loss=0.08571, simple_loss=0.1146, pruned_loss=0.02217, audio_tagging_loss=0.006252, over 15910.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09598, pruned_loss=0.01683, audio_tagging_loss=0.009737, over 3038232.07 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 11:55:09,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.26 vs. limit=12.0
2023-11-21 11:55:12,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1491820.0, ans=0.125
2023-11-21 11:55:21,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1491886.6666666667, ans=0.0
2023-11-21 11:55:25,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1491886.6666666667, ans=0.95
2023-11-21 11:55:36,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1491953.3333333333, ans=0.125
2023-11-21 11:55:38,306 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223800
2023-11-21 11:55:41,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1491953.3333333333, ans=0.025
2023-11-21 11:55:46,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.79 vs. limit=15.0
2023-11-21 11:55:57,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=11.93 vs. limit=12.0
2023-11-21 11:56:01,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1492086.6666666667, ans=0.125
2023-11-21 11:56:03,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1492086.6666666667, ans=0.0
2023-11-21 11:56:11,496 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7400, loss[loss=0.07617, simple_loss=0.08954, pruned_loss=0.01894, audio_tagging_loss=0.01246, over 15271.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09591, pruned_loss=0.01654, audio_tagging_loss=0.009627, over 3041374.23 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:56:43,087 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223850
2023-11-21 11:56:43,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1492286.6666666667, ans=0.0
2023-11-21 11:56:45,438 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 7.994e+01 8.788e+01 9.602e+01 2.162e+02, threshold=1.758e+02, percent-clipped=1.0
2023-11-21 11:56:53,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1492353.3333333333, ans=0.1
2023-11-21 11:57:09,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1492420.0, ans=0.035
2023-11-21 11:57:10,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1492420.0, ans=0.2
2023-11-21 11:57:15,944 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7450, loss[loss=0.06124, simple_loss=0.07968, pruned_loss=0.01034, audio_tagging_loss=0.01106, over 15038.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09686, pruned_loss=0.0168, audio_tagging_loss=0.00952, over 3037712.63 frames. ], batch size: 58, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:57:32,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1492553.3333333333, ans=0.125
2023-11-21 11:57:33,291 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.61 vs. limit=10.0
2023-11-21 11:57:43,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1492620.0, ans=0.07
2023-11-21 11:57:43,741 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.45 vs. limit=5.0
2023-11-21 11:57:48,272 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223900
2023-11-21 11:57:56,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1492686.6666666667, ans=0.125
2023-11-21 11:57:59,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1492686.6666666667, ans=0.125
2023-11-21 11:58:19,359 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7500, loss[loss=0.06374, simple_loss=0.07672, pruned_loss=0.01561, audio_tagging_loss=0.00977, over 14407.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.09743, pruned_loss=0.01686, audio_tagging_loss=0.009386, over 3048603.62 frames. ], batch size: 54, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:58:24,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1492820.0, ans=0.125
2023-11-21 11:58:25,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1492820.0, ans=0.2
2023-11-21 11:58:29,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1492820.0, ans=0.0
2023-11-21 11:58:48,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1492953.3333333333, ans=0.125
2023-11-21 11:58:51,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 223950
2023-11-21 11:58:52,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1492953.3333333333, ans=0.125
2023-11-21 11:58:54,579 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.233e+01 8.004e+01 8.893e+01 9.495e+01 1.256e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 11:59:17,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1493086.6666666667, ans=0.125
2023-11-21 11:59:24,166 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7550, loss[loss=0.0621, simple_loss=0.08455, pruned_loss=0.01265, audio_tagging_loss=0.007174, over 15678.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.09798, pruned_loss=0.01704, audio_tagging_loss=0.009372, over 3056921.80 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 16.0
2023-11-21 11:59:25,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1493153.3333333333, ans=0.09899494936611666
2023-11-21 11:59:28,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1493153.3333333333, ans=0.125
2023-11-21 11:59:33,227 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.89 vs. limit=15.0
2023-11-21 11:59:55,186 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224000
2023-11-21 11:59:55,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1493286.6666666667, ans=0.125
2023-11-21 11:59:56,632 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-224000.pt
2023-11-21 12:00:00,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.61 vs. limit=10.0
2023-11-21 12:00:15,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1493353.3333333333, ans=0.1
2023-11-21 12:00:22,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1493420.0, ans=0.125
2023-11-21 12:00:29,971 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7600, loss[loss=0.08648, simple_loss=0.1095, pruned_loss=0.0207, audio_tagging_loss=0.01104, over 15749.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.097, pruned_loss=0.01686, audio_tagging_loss=0.009432, over 3056539.25 frames. ], batch size: 59, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 12:00:52,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1493553.3333333333, ans=0.125
2023-11-21 12:01:02,923 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224050
2023-11-21 12:01:05,232 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 8.103e+01 8.665e+01 9.260e+01 1.158e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 12:01:16,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.21 vs. limit=15.0
2023-11-21 12:01:27,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1493753.3333333333, ans=0.125
2023-11-21 12:01:34,047 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7650, loss[loss=0.06948, simple_loss=0.0908, pruned_loss=0.01609, audio_tagging_loss=0.00799, over 14976.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09724, pruned_loss=0.017, audio_tagging_loss=0.009479, over 3049892.79 frames. ], batch size: 56, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 12:01:38,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1493820.0, ans=0.125
2023-11-21 12:01:39,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1493820.0, ans=0.2
2023-11-21 12:01:55,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1493886.6666666667, ans=0.09899494936611666
2023-11-21 12:02:06,330 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224100
2023-11-21 12:02:12,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1494020.0, ans=0.125
2023-11-21 12:02:20,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1494020.0, ans=0.1
2023-11-21 12:02:22,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1494020.0, ans=0.125
2023-11-21 12:02:22,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1494020.0, ans=0.125
2023-11-21 12:02:24,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1494086.6666666667, ans=0.125
2023-11-21 12:02:38,306 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7700, loss[loss=0.04223, simple_loss=0.04736, pruned_loss=0.00573, audio_tagging_loss=0.01282, over 14994.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.09752, pruned_loss=0.01709, audio_tagging_loss=0.009496, over 3049405.76 frames. ], batch size: 57, lr: 3.60e-03, grad_scale: 32.0
2023-11-21 12:02:45,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1494153.3333333333, ans=0.07
2023-11-21 12:03:09,803 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224150
2023-11-21 12:03:10,232 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.80 vs. limit=15.0
2023-11-21 12:03:12,164 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.308e+01 8.854e+01 9.589e+01 1.258e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 12:03:12,731 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.81 vs. limit=15.0
2023-11-21 12:03:26,931 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:03:41,688 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7750, loss[loss=0.06915, simple_loss=0.08742, pruned_loss=0.01432, audio_tagging_loss=0.01112, over 14966.00 frames. ], tot_loss[loss=0.0753, simple_loss=0.09748, pruned_loss=0.01698, audio_tagging_loss=0.009576, over 3046013.19 frames. ], batch size: 55, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:03:41,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1494486.6666666667, ans=0.125
2023-11-21 12:03:44,773 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.46 vs. limit=22.5
2023-11-21 12:04:10,853 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.89 vs. limit=12.0
2023-11-21 12:04:12,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.74 vs. limit=22.5
2023-11-21 12:04:14,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224200
2023-11-21 12:04:15,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1494620.0, ans=0.125
2023-11-21 12:04:26,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1494686.6666666667, ans=0.125
2023-11-21 12:04:45,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1494820.0, ans=0.125
2023-11-21 12:04:45,895 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7800, loss[loss=0.08183, simple_loss=0.1049, pruned_loss=0.01758, audio_tagging_loss=0.01181, over 14362.00 frames. ], tot_loss[loss=0.0752, simple_loss=0.09728, pruned_loss=0.01688, audio_tagging_loss=0.00968, over 3046145.78 frames. ], batch size: 55, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:04:55,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1494820.0, ans=0.0
2023-11-21 12:05:02,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1494886.6666666667, ans=0.125
2023-11-21 12:05:07,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1494886.6666666667, ans=0.125
2023-11-21 12:05:18,287 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224250
2023-11-21 12:05:21,778 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.303e+01 8.766e+01 9.637e+01 1.238e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 12:05:37,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1495086.6666666667, ans=0.125
2023-11-21 12:05:44,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1495086.6666666667, ans=0.0
2023-11-21 12:05:45,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1495086.6666666667, ans=0.125
2023-11-21 12:05:50,440 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7850, loss[loss=0.07897, simple_loss=0.1108, pruned_loss=0.01728, audio_tagging_loss=0.006309, over 15215.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09777, pruned_loss=0.01693, audio_tagging_loss=0.00966, over 3052939.27 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:06:16,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1495286.6666666667, ans=0.0
2023-11-21 12:06:21,457 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224300
2023-11-21 12:06:25,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1495286.6666666667, ans=0.0
2023-11-21 12:06:50,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1495420.0, ans=0.2
2023-11-21 12:06:53,987 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7900, loss[loss=0.1037, simple_loss=0.1343, pruned_loss=0.02695, audio_tagging_loss=0.00961, over 16035.00 frames. ], tot_loss[loss=0.07514, simple_loss=0.09695, pruned_loss=0.01684, audio_tagging_loss=0.009832, over 3056341.71 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:06:55,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1495486.6666666667, ans=0.125
2023-11-21 12:07:06,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1495553.3333333333, ans=0.125
2023-11-21 12:07:12,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1495553.3333333333, ans=0.1
2023-11-21 12:07:16,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1495553.3333333333, ans=0.0
2023-11-21 12:07:22,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1495620.0, ans=0.125
2023-11-21 12:07:26,395 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224350
2023-11-21 12:07:27,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1495620.0, ans=10.0
2023-11-21 12:07:29,890 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 8.237e+01 8.708e+01 9.758e+01 1.228e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 12:07:56,990 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 7950, loss[loss=0.1013, simple_loss=0.1339, pruned_loss=0.02607, audio_tagging_loss=0.008274, over 15406.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09627, pruned_loss=0.0167, audio_tagging_loss=0.009896, over 3047340.32 frames. ], batch size: 54, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:08:14,105 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:08:29,280 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224400
2023-11-21 12:08:40,977 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.29 vs. limit=6.0
2023-11-21 12:08:42,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1496020.0, ans=0.2
2023-11-21 12:08:45,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1496020.0, ans=0.2
2023-11-21 12:08:58,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1496086.6666666667, ans=0.2
2023-11-21 12:09:01,119 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8000, loss[loss=0.08147, simple_loss=0.1073, pruned_loss=0.01741, audio_tagging_loss=0.0104, over 16831.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09625, pruned_loss=0.01672, audio_tagging_loss=0.009914, over 3043557.78 frames. ], batch size: 62, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:09:08,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1496153.3333333333, ans=0.125
2023-11-21 12:09:10,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1496153.3333333333, ans=0.125
2023-11-21 12:09:15,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1496220.0, ans=0.0
2023-11-21 12:09:29,029 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.15 vs. limit=15.0
2023-11-21 12:09:32,088 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224450
2023-11-21 12:09:35,518 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.187e+01 8.210e+01 8.871e+01 9.654e+01 1.306e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 12:10:01,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1496420.0, ans=0.07
2023-11-21 12:10:01,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1496420.0, ans=0.0
2023-11-21 12:10:04,410 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8050, loss[loss=0.06761, simple_loss=0.07601, pruned_loss=0.01919, audio_tagging_loss=0.01042, over 14629.00 frames. ], tot_loss[loss=0.07438, simple_loss=0.09557, pruned_loss=0.01665, audio_tagging_loss=0.009947, over 3042716.10 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:10:14,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1496486.6666666667, ans=0.125
2023-11-21 12:10:20,164 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.86 vs. limit=10.0
2023-11-21 12:10:36,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224500
2023-11-21 12:10:46,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1496686.6666666667, ans=0.0
2023-11-21 12:10:55,808 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.99 vs. limit=12.0
2023-11-21 12:11:07,059 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8100, loss[loss=0.07549, simple_loss=0.1049, pruned_loss=0.01444, audio_tagging_loss=0.008576, over 15373.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09594, pruned_loss=0.01678, audio_tagging_loss=0.009804, over 3041795.75 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:11:11,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1496820.0, ans=0.125
2023-11-21 12:11:14,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1496820.0, ans=0.2
2023-11-21 12:11:22,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1496886.6666666667, ans=0.09899494936611666
2023-11-21 12:11:39,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224550
2023-11-21 12:11:43,197 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.553e+01 8.024e+01 8.529e+01 9.304e+01 1.158e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 12:12:10,666 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8150, loss[loss=0.07179, simple_loss=0.1025, pruned_loss=0.01172, audio_tagging_loss=0.008844, over 14346.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09626, pruned_loss=0.01673, audio_tagging_loss=0.009693, over 3046837.73 frames. ], batch size: 55, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:12:28,623 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.86 vs. limit=6.0
2023-11-21 12:12:42,554 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224600
2023-11-21 12:13:15,673 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8200, loss[loss=0.07018, simple_loss=0.09213, pruned_loss=0.01388, audio_tagging_loss=0.01023, over 14892.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09666, pruned_loss=0.01667, audio_tagging_loss=0.009443, over 3044722.54 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:13:16,937 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:13:18,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1497486.6666666667, ans=0.07
2023-11-21 12:13:19,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1497486.6666666667, ans=0.5
2023-11-21 12:13:21,216 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.74 vs. limit=15.0
2023-11-21 12:13:36,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1497553.3333333333, ans=0.125
2023-11-21 12:13:46,783 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224650
2023-11-21 12:13:49,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=1497620.0, ans=0.2
2023-11-21 12:13:52,056 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.039e+01 8.805e+01 9.521e+01 1.121e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 12:13:58,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1497686.6666666667, ans=0.125
2023-11-21 12:13:59,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1497686.6666666667, ans=0.125
2023-11-21 12:14:08,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1497753.3333333333, ans=0.0
2023-11-21 12:14:12,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1497753.3333333333, ans=0.0
2023-11-21 12:14:18,864 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8250, loss[loss=0.08532, simple_loss=0.1087, pruned_loss=0.01868, audio_tagging_loss=0.01228, over 16270.00 frames. ], tot_loss[loss=0.07478, simple_loss=0.09724, pruned_loss=0.01681, audio_tagging_loss=0.009352, over 3051049.82 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:14:47,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1497953.3333333333, ans=0.04949747468305833
2023-11-21 12:14:51,170 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224700
2023-11-21 12:14:53,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1497953.3333333333, ans=0.125
2023-11-21 12:14:54,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.12 vs. limit=22.5
2023-11-21 12:14:59,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1498020.0, ans=0.0
2023-11-21 12:15:18,809 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0
2023-11-21 12:15:22,103 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8300, loss[loss=0.07644, simple_loss=0.09107, pruned_loss=0.01675, audio_tagging_loss=0.01415, over 15358.00 frames. ], tot_loss[loss=0.07472, simple_loss=0.09675, pruned_loss=0.01696, audio_tagging_loss=0.009386, over 3058291.23 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 8.0
2023-11-21 12:15:26,589 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.35 vs. limit=15.0
2023-11-21 12:15:49,834 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.32 vs. limit=22.5
2023-11-21 12:15:54,066 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224750
2023-11-21 12:16:00,005 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.024e+01 7.847e+01 8.529e+01 9.488e+01 1.210e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 12:16:03,203 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.85 vs. limit=22.5
2023-11-21 12:16:05,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1498353.3333333333, ans=0.0
2023-11-21 12:16:11,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1498420.0, ans=0.2
2023-11-21 12:16:24,968 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:16:27,136 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8350, loss[loss=0.08697, simple_loss=0.1128, pruned_loss=0.02082, audio_tagging_loss=0.009751, over 15678.00 frames. ], tot_loss[loss=0.07522, simple_loss=0.09767, pruned_loss=0.01705, audio_tagging_loss=0.009328, over 3058759.51 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 8.0
2023-11-21 12:16:50,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1498620.0, ans=0.125
2023-11-21 12:16:50,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1498620.0, ans=0.0
2023-11-21 12:16:58,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224800
2023-11-21 12:17:18,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1498753.3333333333, ans=0.0
2023-11-21 12:17:19,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1498753.3333333333, ans=0.125
2023-11-21 12:17:30,430 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8400, loss[loss=0.06566, simple_loss=0.0839, pruned_loss=0.01245, audio_tagging_loss=0.01125, over 15426.00 frames. ], tot_loss[loss=0.07508, simple_loss=0.0974, pruned_loss=0.01698, audio_tagging_loss=0.009401, over 3052341.13 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:17:39,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1498820.0, ans=0.125
2023-11-21 12:18:03,069 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224850
2023-11-21 12:18:04,750 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.54 vs. limit=15.0
2023-11-21 12:18:09,645 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.960e+01 8.252e+01 9.012e+01 9.570e+01 1.298e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-21 12:18:09,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1499020.0, ans=0.125
2023-11-21 12:18:22,123 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:18:32,635 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.37 vs. limit=15.0
2023-11-21 12:18:34,200 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8450, loss[loss=0.07049, simple_loss=0.09097, pruned_loss=0.01406, audio_tagging_loss=0.01095, over 15569.00 frames. ], tot_loss[loss=0.07592, simple_loss=0.09859, pruned_loss=0.01723, audio_tagging_loss=0.00939, over 3054436.84 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:18:36,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1499153.3333333333, ans=0.1
2023-11-21 12:18:43,582 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.54 vs. limit=10.0
2023-11-21 12:18:45,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1499153.3333333333, ans=0.0
2023-11-21 12:18:52,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.85 vs. limit=22.5
2023-11-21 12:18:53,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1499220.0, ans=0.125
2023-11-21 12:18:53,735 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.62 vs. limit=15.0
2023-11-21 12:18:54,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1499220.0, ans=0.125
2023-11-21 12:19:02,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1499286.6666666667, ans=0.0
2023-11-21 12:19:03,897 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.02 vs. limit=22.5
2023-11-21 12:19:06,952 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224900
2023-11-21 12:19:39,592 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8500, loss[loss=0.07964, simple_loss=0.1061, pruned_loss=0.01721, audio_tagging_loss=0.009358, over 15443.00 frames. ], tot_loss[loss=0.07569, simple_loss=0.09828, pruned_loss=0.01717, audio_tagging_loss=0.009387, over 3061237.02 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:19:49,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1499486.6666666667, ans=0.125
2023-11-21 12:19:51,682 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.63 vs. limit=15.0
2023-11-21 12:20:10,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1499620.0, ans=0.0
2023-11-21 12:20:10,977 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 224950
2023-11-21 12:20:12,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1499620.0, ans=0.125
2023-11-21 12:20:13,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1499620.0, ans=0.5
2023-11-21 12:20:17,533 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.782e+01 8.392e+01 8.931e+01 9.765e+01 1.184e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 12:20:26,185 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.32 vs. limit=10.0
2023-11-21 12:20:32,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1499753.3333333333, ans=0.1
2023-11-21 12:20:43,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1499820.0, ans=0.1
2023-11-21 12:20:43,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1499820.0, ans=0.125
2023-11-21 12:20:44,393 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8550, loss[loss=0.08087, simple_loss=0.1179, pruned_loss=0.01455, audio_tagging_loss=0.007343, over 15460.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09805, pruned_loss=0.01711, audio_tagging_loss=0.009407, over 3057441.46 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:21:00,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1499886.6666666667, ans=0.0
2023-11-21 12:21:15,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1499953.3333333333, ans=0.125
2023-11-21 12:21:16,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225000
2023-11-21 12:21:19,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.63 vs. limit=15.0
2023-11-21 12:21:20,347 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.41 vs. limit=15.0
2023-11-21 12:21:35,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1500086.6666666667, ans=0.2
2023-11-21 12:21:36,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1500086.6666666667, ans=0.1
2023-11-21 12:21:44,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1500086.6666666667, ans=0.125
2023-11-21 12:21:48,080 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8600, loss[loss=0.09319, simple_loss=0.129, pruned_loss=0.02004, audio_tagging_loss=0.00867, over 16292.00 frames. ], tot_loss[loss=0.07587, simple_loss=0.09851, pruned_loss=0.01723, audio_tagging_loss=0.009382, over 3054274.95 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:21:48,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1500153.3333333333, ans=0.2
2023-11-21 12:21:53,655 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.63 vs. limit=12.0
2023-11-21 12:21:53,682 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.73 vs. limit=15.0
2023-11-21 12:22:21,325 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225050
2023-11-21 12:22:27,332 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.106e+01 8.713e+01 9.527e+01 1.348e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 12:22:33,659 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:22:52,961 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8650, loss[loss=0.08575, simple_loss=0.09844, pruned_loss=0.0258, audio_tagging_loss=0.01073, over 14942.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09807, pruned_loss=0.01724, audio_tagging_loss=0.00944, over 3052806.80 frames. ], batch size: 57, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:23:14,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1500553.3333333333, ans=0.1
2023-11-21 12:23:15,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1500553.3333333333, ans=0.0
2023-11-21 12:23:25,147 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225100
2023-11-21 12:23:36,651 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.23 vs. limit=15.0
2023-11-21 12:23:41,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1500686.6666666667, ans=0.125
2023-11-21 12:23:44,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1500753.3333333333, ans=0.0
2023-11-21 12:23:57,075 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8700, loss[loss=0.06772, simple_loss=0.07927, pruned_loss=0.01593, audio_tagging_loss=0.01216, over 14431.00 frames. ], tot_loss[loss=0.07534, simple_loss=0.09735, pruned_loss=0.01712, audio_tagging_loss=0.009541, over 3048390.29 frames. ], batch size: 56, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:24:21,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1500953.3333333333, ans=0.125
2023-11-21 12:24:29,476 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225150
2023-11-21 12:24:35,994 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.442e+01 8.072e+01 8.843e+01 9.223e+01 1.204e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 12:24:37,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1501020.0, ans=0.1
2023-11-21 12:24:50,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1501086.6666666667, ans=0.0
2023-11-21 12:24:59,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1501153.3333333333, ans=0.1
2023-11-21 12:25:00,557 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8750, loss[loss=0.08515, simple_loss=0.1176, pruned_loss=0.01823, audio_tagging_loss=0.008153, over 14657.00 frames. ], tot_loss[loss=0.07614, simple_loss=0.09831, pruned_loss=0.0174, audio_tagging_loss=0.009589, over 3052023.36 frames. ], batch size: 55, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:25:21,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1501220.0, ans=0.125
2023-11-21 12:25:24,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.48 vs. limit=15.0
2023-11-21 12:25:32,758 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225200
2023-11-21 12:25:48,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1501353.3333333333, ans=0.125
2023-11-21 12:26:05,339 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8800, loss[loss=0.06502, simple_loss=0.0813, pruned_loss=0.01413, audio_tagging_loss=0.01023, over 15466.00 frames. ], tot_loss[loss=0.07616, simple_loss=0.09814, pruned_loss=0.01739, audio_tagging_loss=0.009697, over 3052338.45 frames. ], batch size: 60, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:26:19,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1501553.3333333333, ans=0.125
2023-11-21 12:26:34,320 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.19 vs. limit=22.5
2023-11-21 12:26:37,327 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225250
2023-11-21 12:26:43,237 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.062e+01 8.940e+01 9.452e+01 1.337e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-21 12:27:09,826 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8850, loss[loss=0.08683, simple_loss=0.1158, pruned_loss=0.01855, audio_tagging_loss=0.01038, over 16486.00 frames. ], tot_loss[loss=0.07598, simple_loss=0.09788, pruned_loss=0.01733, audio_tagging_loss=0.009709, over 3049603.94 frames. ], batch size: 61, lr: 3.59e-03, grad_scale: 32.0
2023-11-21 12:27:22,948 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:27:24,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1501886.6666666667, ans=0.125
2023-11-21 12:27:38,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1501953.3333333333, ans=0.0
2023-11-21 12:27:42,722 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225300
2023-11-21 12:28:15,054 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8900, loss[loss=0.05722, simple_loss=0.07254, pruned_loss=0.01137, audio_tagging_loss=0.009585, over 15608.00 frames. ], tot_loss[loss=0.07551, simple_loss=0.09727, pruned_loss=0.01717, audio_tagging_loss=0.009707, over 3050484.54 frames. ], batch size: 58, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:28:47,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225350
2023-11-21 12:28:47,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1502286.6666666667, ans=0.0
2023-11-21 12:28:54,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1502353.3333333333, ans=0.0
2023-11-21 12:28:54,894 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.032e+01 8.160e+01 8.777e+01 9.579e+01 1.339e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 12:29:17,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1502420.0, ans=0.1
2023-11-21 12:29:20,397 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 8950, loss[loss=0.08289, simple_loss=0.09387, pruned_loss=0.0239, audio_tagging_loss=0.01205, over 16184.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09779, pruned_loss=0.01729, audio_tagging_loss=0.00954, over 3047901.58 frames. ], batch size: 62, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:29:36,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1502553.3333333333, ans=0.0
2023-11-21 12:29:49,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1502620.0, ans=0.0
2023-11-21 12:29:51,952 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225400
2023-11-21 12:29:55,996 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.32 vs. limit=6.0
2023-11-21 12:29:58,155 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:30:00,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1502686.6666666667, ans=0.2
2023-11-21 12:30:07,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.93 vs. limit=15.0
2023-11-21 12:30:11,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1502753.3333333333, ans=0.125
2023-11-21 12:30:15,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1502753.3333333333, ans=0.125
2023-11-21 12:30:25,211 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9000, loss[loss=0.07369, simple_loss=0.09661, pruned_loss=0.01841, audio_tagging_loss=0.006978, over 16902.00 frames. ], tot_loss[loss=0.07607, simple_loss=0.09854, pruned_loss=0.01734, audio_tagging_loss=0.009456, over 3056962.84 frames. ], batch size: 62, lr: 3.59e-03, grad_scale: 16.0
2023-11-21 12:30:25,214 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 12:31:03,328 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([4.5004, 3.8028, 4.2997, 3.4018], device='cuda:0')
2023-11-21 12:31:06,239 INFO [train_asr.py:1253] (0/4) Epoch 19, validation: loss=0.06044, simple_loss=0.05233, pruned_loss=0.005297, audio_tagging_loss=0.02898, over 4681554.00 frames. 
2023-11-21 12:31:06,240 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 12:31:12,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1502820.0, ans=0.125
2023-11-21 12:31:25,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1502886.6666666667, ans=0.0
2023-11-21 12:31:38,588 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225450
2023-11-21 12:31:38,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1502953.3333333333, ans=0.0
2023-11-21 12:31:45,865 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.028e+01 8.087e+01 8.929e+01 9.690e+01 1.260e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 12:31:58,092 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.45 vs. limit=15.0
2023-11-21 12:32:11,484 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9050, loss[loss=0.08092, simple_loss=0.1088, pruned_loss=0.01709, audio_tagging_loss=0.009447, over 16241.00 frames. ], tot_loss[loss=0.076, simple_loss=0.09847, pruned_loss=0.01737, audio_tagging_loss=0.009399, over 3064158.40 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:32:24,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1503220.0, ans=0.125
2023-11-21 12:32:27,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1503220.0, ans=0.2
2023-11-21 12:32:42,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225500
2023-11-21 12:32:43,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1503286.6666666667, ans=0.0
2023-11-21 12:32:49,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1503353.3333333333, ans=0.1
2023-11-21 12:32:56,345 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.60 vs. limit=15.0
2023-11-21 12:33:15,759 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9100, loss[loss=0.06528, simple_loss=0.08235, pruned_loss=0.01672, audio_tagging_loss=0.007385, over 14251.00 frames. ], tot_loss[loss=0.07514, simple_loss=0.09734, pruned_loss=0.01705, audio_tagging_loss=0.009419, over 3055296.07 frames. ], batch size: 55, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:33:25,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1503486.6666666667, ans=0.125
2023-11-21 12:33:30,103 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.98 vs. limit=15.0
2023-11-21 12:33:37,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1503553.3333333333, ans=0.5
2023-11-21 12:33:48,296 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225550
2023-11-21 12:33:52,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1503620.0, ans=0.125
2023-11-21 12:33:56,025 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.533e+01 8.001e+01 8.590e+01 9.455e+01 1.329e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-21 12:34:10,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1503753.3333333333, ans=0.0
2023-11-21 12:34:19,574 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9150, loss[loss=0.07993, simple_loss=0.1096, pruned_loss=0.01771, audio_tagging_loss=0.007439, over 15621.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09834, pruned_loss=0.01712, audio_tagging_loss=0.009265, over 3056107.44 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:34:41,399 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=3.145e-01
2023-11-21 12:34:52,113 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225600
2023-11-21 12:35:08,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1504020.0, ans=0.2
2023-11-21 12:35:12,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1504086.6666666667, ans=0.125
2023-11-21 12:35:12,710 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0
2023-11-21 12:35:18,118 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.54 vs. limit=22.5
2023-11-21 12:35:18,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1504086.6666666667, ans=0.125
2023-11-21 12:35:20,692 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:35:24,770 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9200, loss[loss=0.06956, simple_loss=0.07931, pruned_loss=0.01712, audio_tagging_loss=0.01278, over 15261.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09852, pruned_loss=0.01729, audio_tagging_loss=0.009363, over 3061518.98 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:35:26,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1504153.3333333333, ans=0.0
2023-11-21 12:35:27,715 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.51 vs. limit=22.5
2023-11-21 12:35:42,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1504220.0, ans=0.125
2023-11-21 12:35:54,313 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.45 vs. limit=6.0
2023-11-21 12:35:56,172 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225650
2023-11-21 12:36:03,561 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.007e+01 8.520e+01 9.210e+01 1.380e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-21 12:36:03,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1504353.3333333333, ans=10.0
2023-11-21 12:36:07,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1504353.3333333333, ans=0.125
2023-11-21 12:36:08,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1504353.3333333333, ans=0.0
2023-11-21 12:36:13,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1504353.3333333333, ans=0.0
2023-11-21 12:36:29,486 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9250, loss[loss=0.05777, simple_loss=0.06716, pruned_loss=0.01304, audio_tagging_loss=0.01115, over 14728.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.0974, pruned_loss=0.0172, audio_tagging_loss=0.009347, over 3062833.86 frames. ], batch size: 58, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:36:47,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1504553.3333333333, ans=0.0
2023-11-21 12:37:01,629 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225700
2023-11-21 12:37:05,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1504620.0, ans=0.125
2023-11-21 12:37:33,053 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9300, loss[loss=0.0623, simple_loss=0.0836, pruned_loss=0.01187, audio_tagging_loss=0.008636, over 15008.00 frames. ], tot_loss[loss=0.07501, simple_loss=0.09709, pruned_loss=0.01701, audio_tagging_loss=0.009453, over 3066890.01 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:37:34,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1504820.0, ans=0.125
2023-11-21 12:37:50,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1504886.6666666667, ans=0.0
2023-11-21 12:37:51,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1504886.6666666667, ans=0.125
2023-11-21 12:38:05,495 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225750
2023-11-21 12:38:05,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1504953.3333333333, ans=0.1
2023-11-21 12:38:12,764 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 7.914e+01 8.650e+01 9.354e+01 1.480e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 12:38:13,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1505020.0, ans=0.1
2023-11-21 12:38:30,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1505086.6666666667, ans=0.125
2023-11-21 12:38:31,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1505086.6666666667, ans=0.125
2023-11-21 12:38:35,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1505153.3333333333, ans=0.0
2023-11-21 12:38:37,021 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9350, loss[loss=0.07095, simple_loss=0.08906, pruned_loss=0.01635, audio_tagging_loss=0.01006, over 16167.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09628, pruned_loss=0.01684, audio_tagging_loss=0.009475, over 3063339.24 frames. ], batch size: 59, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:38:42,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1505153.3333333333, ans=0.125
2023-11-21 12:38:53,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1505220.0, ans=0.125
2023-11-21 12:39:09,036 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225800
2023-11-21 12:39:10,996 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:39:12,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.43 vs. limit=15.0
2023-11-21 12:39:13,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.whiten.whitening_limit, batch_count=1505286.6666666667, ans=12.0
2023-11-21 12:39:17,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1505353.3333333333, ans=0.0
2023-11-21 12:39:18,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1505353.3333333333, ans=0.1
2023-11-21 12:39:42,146 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9400, loss[loss=0.07781, simple_loss=0.09402, pruned_loss=0.01737, audio_tagging_loss=0.01343, over 15748.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09649, pruned_loss=0.01691, audio_tagging_loss=0.009574, over 3063663.23 frames. ], batch size: 61, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:39:52,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1505486.6666666667, ans=0.0
2023-11-21 12:40:03,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1505553.3333333333, ans=0.125
2023-11-21 12:40:10,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1505620.0, ans=0.125
2023-11-21 12:40:12,726 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225850
2023-11-21 12:40:22,995 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.778e+01 8.417e+01 9.121e+01 9.724e+01 1.188e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-21 12:40:24,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1505686.6666666667, ans=0.0
2023-11-21 12:40:37,495 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.56 vs. limit=12.0
2023-11-21 12:40:39,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1505753.3333333333, ans=0.0
2023-11-21 12:40:42,768 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:40:45,237 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9450, loss[loss=0.08276, simple_loss=0.1049, pruned_loss=0.01952, audio_tagging_loss=0.01081, over 16485.00 frames. ], tot_loss[loss=0.07549, simple_loss=0.09758, pruned_loss=0.01716, audio_tagging_loss=0.009546, over 3061361.86 frames. ], batch size: 60, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:40:49,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1505820.0, ans=0.125
2023-11-21 12:40:53,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1505820.0, ans=0.0
2023-11-21 12:40:54,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1505820.0, ans=0.05
2023-11-21 12:40:54,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1505820.0, ans=0.125
2023-11-21 12:40:56,963 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.58 vs. limit=22.5
2023-11-21 12:40:59,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1505886.6666666667, ans=0.125
2023-11-21 12:41:16,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1505953.3333333333, ans=0.125
2023-11-21 12:41:17,598 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225900
2023-11-21 12:41:48,211 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9500, loss[loss=0.07085, simple_loss=0.08633, pruned_loss=0.01543, audio_tagging_loss=0.01226, over 15801.00 frames. ], tot_loss[loss=0.0753, simple_loss=0.09712, pruned_loss=0.017, audio_tagging_loss=0.00974, over 3063527.25 frames. ], batch size: 60, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:41:52,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1506153.3333333333, ans=10.0
2023-11-21 12:41:54,601 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.01 vs. limit=15.0
2023-11-21 12:42:02,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1506220.0, ans=0.0
2023-11-21 12:42:11,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1506220.0, ans=0.125
2023-11-21 12:42:20,935 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 225950
2023-11-21 12:42:29,398 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.765e+01 8.362e+01 9.191e+01 9.965e+01 1.276e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-21 12:42:53,266 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9550, loss[loss=0.06724, simple_loss=0.08175, pruned_loss=0.01465, audio_tagging_loss=0.01172, over 15173.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09712, pruned_loss=0.01716, audio_tagging_loss=0.009864, over 3059320.71 frames. ], batch size: 58, lr: 3.58e-03, grad_scale: 16.0
2023-11-21 12:43:02,210 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:43:08,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1506553.3333333333, ans=0.125
2023-11-21 12:43:11,063 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0
2023-11-21 12:43:14,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1506553.3333333333, ans=0.125
2023-11-21 12:43:24,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226000
2023-11-21 12:43:47,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1506753.3333333333, ans=0.1
2023-11-21 12:43:49,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1506753.3333333333, ans=0.125
2023-11-21 12:43:51,355 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=9.759e-02
2023-11-21 12:43:57,190 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9600, loss[loss=0.09271, simple_loss=0.1211, pruned_loss=0.02446, audio_tagging_loss=0.007705, over 16212.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09691, pruned_loss=0.01705, audio_tagging_loss=0.009946, over 3058257.18 frames. ], batch size: 61, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:44:04,639 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:44:24,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1506953.3333333333, ans=0.0
2023-11-21 12:44:29,947 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226050
2023-11-21 12:44:38,340 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 7.983e+01 8.743e+01 9.613e+01 1.291e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 12:44:59,705 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.75 vs. limit=10.0
2023-11-21 12:45:00,266 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9650, loss[loss=0.07676, simple_loss=0.1096, pruned_loss=0.0147, audio_tagging_loss=0.007262, over 15068.00 frames. ], tot_loss[loss=0.07518, simple_loss=0.09663, pruned_loss=0.01699, audio_tagging_loss=0.00987, over 3054947.56 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:45:09,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1507153.3333333333, ans=0.125
2023-11-21 12:45:12,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1507220.0, ans=0.1
2023-11-21 12:45:12,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1507220.0, ans=0.1
2023-11-21 12:45:17,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1507220.0, ans=0.0
2023-11-21 12:45:33,242 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226100
2023-11-21 12:46:04,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1507486.6666666667, ans=0.125
2023-11-21 12:46:05,008 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9700, loss[loss=0.0844, simple_loss=0.1064, pruned_loss=0.02011, audio_tagging_loss=0.0111, over 15024.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09596, pruned_loss=0.01694, audio_tagging_loss=0.009805, over 3044462.01 frames. ], batch size: 58, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:46:21,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.07 vs. limit=6.0
2023-11-21 12:46:23,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1507553.3333333333, ans=0.0
2023-11-21 12:46:36,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226150
2023-11-21 12:46:44,288 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.17 vs. limit=15.0
2023-11-21 12:46:46,071 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 7.992e+01 8.607e+01 9.446e+01 1.380e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 12:46:57,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.36 vs. limit=15.0
2023-11-21 12:47:07,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1507753.3333333333, ans=0.0
2023-11-21 12:47:09,292 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9750, loss[loss=0.07906, simple_loss=0.09608, pruned_loss=0.01927, audio_tagging_loss=0.01174, over 15016.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09634, pruned_loss=0.01683, audio_tagging_loss=0.009625, over 3043690.22 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:47:17,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1507820.0, ans=0.125
2023-11-21 12:47:18,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1507820.0, ans=0.125
2023-11-21 12:47:41,375 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226200
2023-11-21 12:47:45,010 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.78 vs. limit=15.0
2023-11-21 12:47:54,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1508020.0, ans=0.1
2023-11-21 12:48:08,540 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 12:48:13,285 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9800, loss[loss=0.0803, simple_loss=0.1023, pruned_loss=0.01806, audio_tagging_loss=0.0111, over 15924.00 frames. ], tot_loss[loss=0.07466, simple_loss=0.09643, pruned_loss=0.01691, audio_tagging_loss=0.009533, over 3042214.43 frames. ], batch size: 60, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:48:29,651 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.27 vs. limit=15.0
2023-11-21 12:48:45,546 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226250
2023-11-21 12:48:53,959 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.494e+01 7.972e+01 8.418e+01 9.313e+01 1.119e+02, threshold=1.684e+02, percent-clipped=0.0
2023-11-21 12:48:58,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1508353.3333333333, ans=0.0
2023-11-21 12:49:04,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1508420.0, ans=0.1
2023-11-21 12:49:04,872 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.56 vs. limit=22.5
2023-11-21 12:49:05,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1508420.0, ans=0.0
2023-11-21 12:49:09,509 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:49:17,439 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9850, loss[loss=0.09864, simple_loss=0.1348, pruned_loss=0.0229, audio_tagging_loss=0.00833, over 14919.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.09647, pruned_loss=0.01708, audio_tagging_loss=0.00952, over 3035199.69 frames. ], batch size: 53, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:49:30,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.96 vs. limit=6.0
2023-11-21 12:49:36,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1508553.3333333333, ans=0.0
2023-11-21 12:49:41,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1508620.0, ans=0.0
2023-11-21 12:49:48,691 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226300
2023-11-21 12:49:55,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1508686.6666666667, ans=0.0
2023-11-21 12:50:10,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1508753.3333333333, ans=0.0
2023-11-21 12:50:15,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1508753.3333333333, ans=0.125
2023-11-21 12:50:17,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.60 vs. limit=6.0
2023-11-21 12:50:20,973 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9900, loss[loss=0.09068, simple_loss=0.1147, pruned_loss=0.02185, audio_tagging_loss=0.01147, over 15462.00 frames. ], tot_loss[loss=0.07493, simple_loss=0.09696, pruned_loss=0.01703, audio_tagging_loss=0.009417, over 3036610.15 frames. ], batch size: 58, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:50:26,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1508820.0, ans=0.0
2023-11-21 12:50:31,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1508820.0, ans=0.1
2023-11-21 12:50:47,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1508953.3333333333, ans=0.2
2023-11-21 12:50:51,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1508953.3333333333, ans=0.125
2023-11-21 12:50:53,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226350
2023-11-21 12:50:53,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1508953.3333333333, ans=0.0
2023-11-21 12:50:59,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1509020.0, ans=0.0
2023-11-21 12:51:02,739 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.161e+01 8.727e+01 9.211e+01 1.112e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 12:51:06,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1509020.0, ans=0.1
2023-11-21 12:51:25,639 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 9950, loss[loss=0.06733, simple_loss=0.08702, pruned_loss=0.01503, audio_tagging_loss=0.00879, over 16268.00 frames. ], tot_loss[loss=0.07475, simple_loss=0.09667, pruned_loss=0.01698, audio_tagging_loss=0.009435, over 3045777.31 frames. ], batch size: 60, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:51:27,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1509153.3333333333, ans=0.125
2023-11-21 12:51:48,786 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.82 vs. limit=15.0
2023-11-21 12:51:57,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226400
2023-11-21 12:52:13,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1509353.3333333333, ans=0.1
2023-11-21 12:52:15,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1509353.3333333333, ans=0.125
2023-11-21 12:52:26,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1509420.0, ans=0.2
2023-11-21 12:52:29,835 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10000, loss[loss=0.05452, simple_loss=0.0712, pruned_loss=0.01016, audio_tagging_loss=0.008766, over 15101.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09576, pruned_loss=0.01671, audio_tagging_loss=0.009473, over 3045932.41 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:52:31,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1509486.6666666667, ans=0.1
2023-11-21 12:53:01,809 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226450
2023-11-21 12:53:05,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1509620.0, ans=0.125
2023-11-21 12:53:10,190 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 7.892e+01 8.555e+01 9.256e+01 1.223e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 12:53:33,328 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10050, loss[loss=0.07674, simple_loss=0.1013, pruned_loss=0.01889, audio_tagging_loss=0.007218, over 15420.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09601, pruned_loss=0.01681, audio_tagging_loss=0.009453, over 3043757.91 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:53:55,418 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.51 vs. limit=12.0
2023-11-21 12:54:05,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226500
2023-11-21 12:54:14,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1510020.0, ans=0.015
2023-11-21 12:54:14,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1510020.0, ans=0.125
2023-11-21 12:54:21,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1510020.0, ans=0.1
2023-11-21 12:54:31,001 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.57 vs. limit=15.0
2023-11-21 12:54:36,863 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10100, loss[loss=0.07526, simple_loss=0.1052, pruned_loss=0.0154, audio_tagging_loss=0.007273, over 15589.00 frames. ], tot_loss[loss=0.07532, simple_loss=0.09764, pruned_loss=0.01707, audio_tagging_loss=0.009436, over 3047595.53 frames. ], batch size: 55, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:54:49,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1510220.0, ans=0.125
2023-11-21 12:54:49,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1510220.0, ans=0.125
2023-11-21 12:55:08,759 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226550
2023-11-21 12:55:13,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1510353.3333333333, ans=0.2
2023-11-21 12:55:17,039 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.133e+01 8.640e+01 9.422e+01 1.310e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 12:55:25,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1510353.3333333333, ans=0.125
2023-11-21 12:55:26,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1510420.0, ans=0.125
2023-11-21 12:55:27,656 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:55:35,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1510420.0, ans=0.2
2023-11-21 12:55:40,555 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10150, loss[loss=0.0619, simple_loss=0.07571, pruned_loss=0.01422, audio_tagging_loss=0.009823, over 14855.00 frames. ], tot_loss[loss=0.07543, simple_loss=0.09792, pruned_loss=0.01706, audio_tagging_loss=0.009408, over 3045179.10 frames. ], batch size: 57, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:56:09,688 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:56:12,184 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226600
2023-11-21 12:56:15,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1510620.0, ans=0.125
2023-11-21 12:56:42,064 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.55 vs. limit=10.0
2023-11-21 12:56:42,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1510753.3333333333, ans=0.0
2023-11-21 12:56:44,936 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10200, loss[loss=0.04763, simple_loss=0.0538, pruned_loss=0.007679, audio_tagging_loss=0.01304, over 15524.00 frames. ], tot_loss[loss=0.07511, simple_loss=0.09733, pruned_loss=0.01698, audio_tagging_loss=0.00947, over 3047803.45 frames. ], batch size: 60, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:56:45,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1510820.0, ans=0.1
2023-11-21 12:56:52,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1510820.0, ans=0.2
2023-11-21 12:57:07,502 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 12:57:10,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1510953.3333333333, ans=0.125
2023-11-21 12:57:13,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1510953.3333333333, ans=0.0
2023-11-21 12:57:16,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226650
2023-11-21 12:57:24,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1511020.0, ans=0.2
2023-11-21 12:57:26,000 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.532e+01 7.915e+01 8.561e+01 9.293e+01 1.189e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 12:57:47,644 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.21 vs. limit=22.5
2023-11-21 12:57:48,034 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10250, loss[loss=0.0768, simple_loss=0.1094, pruned_loss=0.01312, audio_tagging_loss=0.008966, over 15424.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09763, pruned_loss=0.01696, audio_tagging_loss=0.009464, over 3048613.00 frames. ], batch size: 56, lr: 3.58e-03, grad_scale: 32.0
2023-11-21 12:57:48,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1511153.3333333333, ans=0.1
2023-11-21 12:58:16,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1511286.6666666667, ans=0.2
2023-11-21 12:58:20,786 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226700
2023-11-21 12:58:25,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1511353.3333333333, ans=0.1
2023-11-21 12:58:27,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1511353.3333333333, ans=0.125
2023-11-21 12:58:29,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1511353.3333333333, ans=0.125
2023-11-21 12:58:30,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1511353.3333333333, ans=0.125
2023-11-21 12:58:30,993 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0
2023-11-21 12:58:51,838 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10300, loss[loss=0.09173, simple_loss=0.125, pruned_loss=0.02177, audio_tagging_loss=0.007449, over 14946.00 frames. ], tot_loss[loss=0.07572, simple_loss=0.09839, pruned_loss=0.01706, audio_tagging_loss=0.009467, over 3053825.14 frames. ], batch size: 54, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 12:59:03,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1511486.6666666667, ans=0.0
2023-11-21 12:59:24,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226750
2023-11-21 12:59:32,483 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.325e+01 8.857e+01 9.782e+01 1.240e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 12:59:38,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1511686.6666666667, ans=0.0
2023-11-21 12:59:55,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1511820.0, ans=0.0
2023-11-21 12:59:56,453 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10350, loss[loss=0.07278, simple_loss=0.1031, pruned_loss=0.01242, audio_tagging_loss=0.008807, over 15142.00 frames. ], tot_loss[loss=0.0768, simple_loss=0.0998, pruned_loss=0.01737, audio_tagging_loss=0.009521, over 3052607.84 frames. ], batch size: 54, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:00:11,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.72 vs. limit=15.0
2023-11-21 13:00:13,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_na.min_abs, batch_count=1511886.6666666667, ans=0.02
2023-11-21 13:00:27,948 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226800
2023-11-21 13:00:38,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1512020.0, ans=0.1
2023-11-21 13:00:45,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1512020.0, ans=0.125
2023-11-21 13:00:47,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=1512086.6666666667, ans=0.025
2023-11-21 13:00:48,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.03 vs. limit=15.0
2023-11-21 13:01:00,005 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10400, loss[loss=0.08619, simple_loss=0.1014, pruned_loss=0.02219, audio_tagging_loss=0.01329, over 16093.00 frames. ], tot_loss[loss=0.07641, simple_loss=0.09922, pruned_loss=0.01721, audio_tagging_loss=0.009589, over 3056556.12 frames. ], batch size: 60, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:01:05,717 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.46 vs. limit=6.0
2023-11-21 13:01:08,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1512153.3333333333, ans=0.1
2023-11-21 13:01:27,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1512286.6666666667, ans=0.2
2023-11-21 13:01:31,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1512286.6666666667, ans=0.0
2023-11-21 13:01:32,582 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226850
2023-11-21 13:01:41,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1512353.3333333333, ans=0.0
2023-11-21 13:01:42,512 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.081e+01 8.812e+01 9.594e+01 2.017e+02, threshold=1.762e+02, percent-clipped=1.0
2023-11-21 13:01:56,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1512420.0, ans=0.125
2023-11-21 13:02:02,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1512486.6666666667, ans=0.125
2023-11-21 13:02:03,533 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10450, loss[loss=0.06659, simple_loss=0.08287, pruned_loss=0.01613, audio_tagging_loss=0.009023, over 13515.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09804, pruned_loss=0.01697, audio_tagging_loss=0.009608, over 3041128.77 frames. ], batch size: 52, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:02:09,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1512486.6666666667, ans=0.0
2023-11-21 13:02:20,887 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.94 vs. limit=15.0
2023-11-21 13:02:31,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1512620.0, ans=0.125
2023-11-21 13:02:36,208 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226900
2023-11-21 13:02:40,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1512620.0, ans=0.125
2023-11-21 13:03:00,136 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=15.52 vs. limit=15.0
2023-11-21 13:03:05,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1512753.3333333333, ans=0.125
2023-11-21 13:03:05,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1512753.3333333333, ans=0.125
2023-11-21 13:03:08,434 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10500, loss[loss=0.07153, simple_loss=0.09873, pruned_loss=0.01221, audio_tagging_loss=0.009951, over 16211.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09721, pruned_loss=0.01682, audio_tagging_loss=0.009591, over 3046238.35 frames. ], batch size: 63, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:03:32,558 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.68 vs. limit=8.0
2023-11-21 13:03:35,484 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:03:38,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 226950
2023-11-21 13:03:40,771 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.53 vs. limit=15.0
2023-11-21 13:03:46,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1513020.0, ans=0.0
2023-11-21 13:03:47,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1513020.0, ans=0.125
2023-11-21 13:03:49,552 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.276e+01 8.777e+01 9.449e+01 1.286e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 13:04:00,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1513086.6666666667, ans=0.0
2023-11-21 13:04:05,427 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.45 vs. limit=15.0
2023-11-21 13:04:11,144 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10550, loss[loss=0.08506, simple_loss=0.1081, pruned_loss=0.0229, audio_tagging_loss=0.008123, over 15460.00 frames. ], tot_loss[loss=0.075, simple_loss=0.09725, pruned_loss=0.01695, audio_tagging_loss=0.009426, over 3044035.57 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:04:27,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1513220.0, ans=0.2
2023-11-21 13:04:41,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1513286.6666666667, ans=0.2
2023-11-21 13:04:43,723 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227000
2023-11-21 13:04:56,816 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.88 vs. limit=22.5
2023-11-21 13:05:02,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1513420.0, ans=0.125
2023-11-21 13:05:14,602 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10600, loss[loss=0.06214, simple_loss=0.07332, pruned_loss=0.0154, audio_tagging_loss=0.01008, over 15318.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09765, pruned_loss=0.01684, audio_tagging_loss=0.009397, over 3044323.09 frames. ], batch size: 61, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:05:25,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.51 vs. limit=6.0
2023-11-21 13:05:44,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1513620.0, ans=0.0
2023-11-21 13:05:47,392 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227050
2023-11-21 13:05:56,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1513686.6666666667, ans=0.1
2023-11-21 13:05:57,031 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.166e+01 8.856e+01 9.761e+01 1.795e+02, threshold=1.771e+02, percent-clipped=1.0
2023-11-21 13:06:01,428 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.48 vs. limit=22.5
2023-11-21 13:06:02,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1513686.6666666667, ans=0.125
2023-11-21 13:06:19,706 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10650, loss[loss=0.07133, simple_loss=0.09279, pruned_loss=0.01559, audio_tagging_loss=0.00935, over 16385.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09699, pruned_loss=0.01663, audio_tagging_loss=0.00938, over 3047074.53 frames. ], batch size: 63, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:06:20,457 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.81 vs. limit=15.0
2023-11-21 13:06:24,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1513820.0, ans=0.2
2023-11-21 13:06:26,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1513820.0, ans=0.125
2023-11-21 13:06:27,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1513820.0, ans=15.0
2023-11-21 13:06:36,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1513886.6666666667, ans=0.125
2023-11-21 13:06:50,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227100
2023-11-21 13:07:11,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1514086.6666666667, ans=0.2
2023-11-21 13:07:12,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1514086.6666666667, ans=0.09899494936611666
2023-11-21 13:07:22,144 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10700, loss[loss=0.08083, simple_loss=0.09957, pruned_loss=0.01914, audio_tagging_loss=0.01191, over 17024.00 frames. ], tot_loss[loss=0.07506, simple_loss=0.09803, pruned_loss=0.01674, audio_tagging_loss=0.009307, over 3053159.76 frames. ], batch size: 64, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:07:22,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1514153.3333333333, ans=0.125
2023-11-21 13:07:31,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1514153.3333333333, ans=0.0
2023-11-21 13:07:55,034 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227150
2023-11-21 13:07:57,674 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:08:00,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1514353.3333333333, ans=0.0
2023-11-21 13:08:03,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1514353.3333333333, ans=0.125
2023-11-21 13:08:04,773 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.492e+01 8.011e+01 8.687e+01 9.462e+01 1.243e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 13:08:25,498 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10750, loss[loss=0.07334, simple_loss=0.09031, pruned_loss=0.0192, audio_tagging_loss=0.00898, over 16482.00 frames. ], tot_loss[loss=0.07484, simple_loss=0.09765, pruned_loss=0.01672, audio_tagging_loss=0.009293, over 3050989.99 frames. ], batch size: 63, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:08:29,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1514486.6666666667, ans=0.1
2023-11-21 13:08:29,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1514486.6666666667, ans=0.125
2023-11-21 13:08:57,928 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227200
2023-11-21 13:08:59,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1514620.0, ans=0.125
2023-11-21 13:09:07,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1514686.6666666667, ans=0.0
2023-11-21 13:09:19,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1514753.3333333333, ans=0.125
2023-11-21 13:09:30,372 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10800, loss[loss=0.07991, simple_loss=0.09988, pruned_loss=0.019, audio_tagging_loss=0.01097, over 14762.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.09695, pruned_loss=0.01666, audio_tagging_loss=0.009416, over 3057153.08 frames. ], batch size: 53, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:09:30,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1514820.0, ans=0.2
2023-11-21 13:09:48,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=1514886.6666666667, ans=10.0
2023-11-21 13:10:00,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1514953.3333333333, ans=0.2
2023-11-21 13:10:01,073 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.36 vs. limit=6.0
2023-11-21 13:10:01,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227250
2023-11-21 13:10:13,784 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.538e+01 7.864e+01 8.499e+01 9.519e+01 1.176e+02, threshold=1.700e+02, percent-clipped=0.0
2023-11-21 13:10:17,060 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.04 vs. limit=15.0
2023-11-21 13:10:34,458 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10850, loss[loss=0.07032, simple_loss=0.08956, pruned_loss=0.01539, audio_tagging_loss=0.01015, over 15843.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09747, pruned_loss=0.01684, audio_tagging_loss=0.009404, over 3058340.22 frames. ], batch size: 61, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:10:44,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1515153.3333333333, ans=0.125
2023-11-21 13:10:58,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1515286.6666666667, ans=0.125
2023-11-21 13:11:04,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1515286.6666666667, ans=0.125
2023-11-21 13:11:06,249 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227300
2023-11-21 13:11:12,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1515353.3333333333, ans=0.125
2023-11-21 13:11:16,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1515353.3333333333, ans=0.1
2023-11-21 13:11:21,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1515353.3333333333, ans=0.125
2023-11-21 13:11:27,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1515420.0, ans=0.0
2023-11-21 13:11:33,395 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 13:11:38,247 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10900, loss[loss=0.05371, simple_loss=0.06467, pruned_loss=0.007773, audio_tagging_loss=0.0136, over 13489.00 frames. ], tot_loss[loss=0.07499, simple_loss=0.09743, pruned_loss=0.01674, audio_tagging_loss=0.009538, over 3057054.57 frames. ], batch size: 50, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:11:47,680 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.47 vs. limit=15.0
2023-11-21 13:11:51,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1515553.3333333333, ans=0.0
2023-11-21 13:12:01,685 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.41 vs. limit=15.0
2023-11-21 13:12:03,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1515620.0, ans=0.5
2023-11-21 13:12:11,308 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227350
2023-11-21 13:12:22,169 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.236e+01 8.108e+01 8.684e+01 9.152e+01 1.151e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 13:12:29,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1515753.3333333333, ans=0.0
2023-11-21 13:12:34,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1515753.3333333333, ans=0.125
2023-11-21 13:12:42,783 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 10950, loss[loss=0.06479, simple_loss=0.08128, pruned_loss=0.01513, audio_tagging_loss=0.009015, over 15087.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.09644, pruned_loss=0.0165, audio_tagging_loss=0.009595, over 3054138.85 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:12:51,246 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.83 vs. limit=15.0
2023-11-21 13:12:52,332 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.47 vs. limit=15.0
2023-11-21 13:12:54,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1515886.6666666667, ans=0.2
2023-11-21 13:13:06,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1515886.6666666667, ans=0.0
2023-11-21 13:13:14,792 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227400
2023-11-21 13:13:16,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1515953.3333333333, ans=10.0
2023-11-21 13:13:23,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1516020.0, ans=0.0
2023-11-21 13:13:33,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1516086.6666666667, ans=0.0
2023-11-21 13:13:36,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1516086.6666666667, ans=0.2
2023-11-21 13:13:48,224 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11000, loss[loss=0.07536, simple_loss=0.09991, pruned_loss=0.01566, audio_tagging_loss=0.009744, over 15038.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09677, pruned_loss=0.01651, audio_tagging_loss=0.009601, over 3058495.23 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:13:58,159 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 13:14:02,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.66 vs. limit=22.5
2023-11-21 13:14:04,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1516220.0, ans=0.09899494936611666
2023-11-21 13:14:20,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227450
2023-11-21 13:14:32,297 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.986e+01 8.066e+01 8.741e+01 9.512e+01 1.178e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 13:14:33,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1516353.3333333333, ans=0.0
2023-11-21 13:14:46,856 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:14:52,772 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11050, loss[loss=0.08516, simple_loss=0.1061, pruned_loss=0.02292, audio_tagging_loss=0.009192, over 14508.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09608, pruned_loss=0.01652, audio_tagging_loss=0.009641, over 3055614.08 frames. ], batch size: 55, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:14:54,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1516486.6666666667, ans=0.125
2023-11-21 13:14:57,010 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.76 vs. limit=15.0
2023-11-21 13:15:07,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1516553.3333333333, ans=0.0
2023-11-21 13:15:07,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1516553.3333333333, ans=0.0
2023-11-21 13:15:09,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1516553.3333333333, ans=0.1
2023-11-21 13:15:24,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1516620.0, ans=0.125
2023-11-21 13:15:25,294 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227500
2023-11-21 13:15:26,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1516620.0, ans=0.1
2023-11-21 13:15:37,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1516686.6666666667, ans=0.125
2023-11-21 13:15:48,127 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.73 vs. limit=10.0
2023-11-21 13:15:54,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1516753.3333333333, ans=0.07
2023-11-21 13:15:57,813 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11100, loss[loss=0.08457, simple_loss=0.1077, pruned_loss=0.01665, audio_tagging_loss=0.01408, over 15592.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09603, pruned_loss=0.01652, audio_tagging_loss=0.009804, over 3056829.08 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:16:09,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1516820.0, ans=0.1
2023-11-21 13:16:14,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1516886.6666666667, ans=0.125
2023-11-21 13:16:22,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1516886.6666666667, ans=0.125
2023-11-21 13:16:28,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1516953.3333333333, ans=0.125
2023-11-21 13:16:30,933 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227550
2023-11-21 13:16:35,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.27 vs. limit=15.0
2023-11-21 13:16:36,474 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-21 13:16:39,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1517020.0, ans=0.2
2023-11-21 13:16:41,856 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.109e+01 8.821e+01 9.588e+01 2.410e+02, threshold=1.764e+02, percent-clipped=1.0
2023-11-21 13:16:43,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1517020.0, ans=0.2
2023-11-21 13:16:53,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1517086.6666666667, ans=0.125
2023-11-21 13:16:59,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1517086.6666666667, ans=0.125
2023-11-21 13:17:03,334 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11150, loss[loss=0.08739, simple_loss=0.1164, pruned_loss=0.02013, audio_tagging_loss=0.00905, over 15737.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09632, pruned_loss=0.0168, audio_tagging_loss=0.009887, over 3054599.09 frames. ], batch size: 59, lr: 3.57e-03, grad_scale: 16.0
2023-11-21 13:17:17,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1517220.0, ans=0.2
2023-11-21 13:17:19,225 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.69 vs. limit=22.5
2023-11-21 13:17:22,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1517220.0, ans=0.0
2023-11-21 13:17:27,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1517286.6666666667, ans=0.0
2023-11-21 13:17:35,393 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227600
2023-11-21 13:17:55,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1517420.0, ans=0.0
2023-11-21 13:18:07,668 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11200, loss[loss=0.08045, simple_loss=0.1104, pruned_loss=0.0167, audio_tagging_loss=0.00852, over 15388.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09543, pruned_loss=0.01646, audio_tagging_loss=0.009946, over 3052975.24 frames. ], batch size: 56, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:18:13,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1517486.6666666667, ans=0.1
2023-11-21 13:18:33,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1517620.0, ans=0.5
2023-11-21 13:18:40,746 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227650
2023-11-21 13:18:44,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1517620.0, ans=0.125
2023-11-21 13:18:51,700 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.451e+01 8.180e+01 8.891e+01 9.538e+01 1.267e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 13:19:12,942 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11250, loss[loss=0.07779, simple_loss=0.1039, pruned_loss=0.01592, audio_tagging_loss=0.00994, over 15703.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09412, pruned_loss=0.01631, audio_tagging_loss=0.009913, over 3046134.01 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:19:14,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1517820.0, ans=0.125
2023-11-21 13:19:14,498 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:19:14,758 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.27 vs. limit=15.0
2023-11-21 13:19:21,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1517820.0, ans=0.125
2023-11-21 13:19:35,758 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.87 vs. limit=15.0
2023-11-21 13:19:36,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1517886.6666666667, ans=0.125
2023-11-21 13:19:40,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1517953.3333333333, ans=0.1
2023-11-21 13:19:45,508 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227700
2023-11-21 13:19:45,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1517953.3333333333, ans=0.1
2023-11-21 13:20:18,086 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11300, loss[loss=0.08649, simple_loss=0.1113, pruned_loss=0.02107, audio_tagging_loss=0.009783, over 15336.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.09466, pruned_loss=0.01639, audio_tagging_loss=0.009806, over 3050380.08 frames. ], batch size: 54, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:20:23,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.30 vs. limit=22.5
2023-11-21 13:20:34,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1518220.0, ans=0.125
2023-11-21 13:20:48,955 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227750
2023-11-21 13:21:01,536 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.039e+01 8.792e+01 9.406e+01 1.983e+02, threshold=1.758e+02, percent-clipped=1.0
2023-11-21 13:21:05,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1518353.3333333333, ans=0.1
2023-11-21 13:21:17,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1518420.0, ans=0.125
2023-11-21 13:21:21,399 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11350, loss[loss=0.06214, simple_loss=0.08508, pruned_loss=0.01286, audio_tagging_loss=0.006732, over 15955.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.0951, pruned_loss=0.01636, audio_tagging_loss=0.009625, over 3046866.83 frames. ], batch size: 60, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:21:34,889 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.48 vs. limit=15.0
2023-11-21 13:21:41,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1518553.3333333333, ans=0.125
2023-11-21 13:21:44,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1518553.3333333333, ans=0.125
2023-11-21 13:21:48,362 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.72 vs. limit=15.0
2023-11-21 13:21:54,330 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227800
2023-11-21 13:21:56,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.36 vs. limit=15.0
2023-11-21 13:22:03,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1518686.6666666667, ans=10.0
2023-11-21 13:22:13,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1518753.3333333333, ans=0.0
2023-11-21 13:22:26,003 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11400, loss[loss=0.06621, simple_loss=0.07803, pruned_loss=0.01545, audio_tagging_loss=0.01174, over 14940.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.0947, pruned_loss=0.0164, audio_tagging_loss=0.009631, over 3043041.78 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:22:31,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1518820.0, ans=0.1
2023-11-21 13:22:37,617 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.47 vs. limit=22.5
2023-11-21 13:22:39,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1518886.6666666667, ans=0.125
2023-11-21 13:22:58,227 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227850
2023-11-21 13:23:09,540 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.020e+01 8.128e+01 8.790e+01 9.462e+01 1.169e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 13:23:10,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1519020.0, ans=0.0
2023-11-21 13:23:31,003 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11450, loss[loss=0.07426, simple_loss=0.09052, pruned_loss=0.01891, audio_tagging_loss=0.01009, over 13032.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09495, pruned_loss=0.01641, audio_tagging_loss=0.009611, over 3044665.27 frames. ], batch size: 52, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:23:41,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1519153.3333333333, ans=0.125
2023-11-21 13:24:02,296 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227900
2023-11-21 13:24:03,044 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.14 vs. limit=15.0
2023-11-21 13:24:04,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1519286.6666666667, ans=0.125
2023-11-21 13:24:12,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1519353.3333333333, ans=0.125
2023-11-21 13:24:20,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1519353.3333333333, ans=0.125
2023-11-21 13:24:22,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1519420.0, ans=0.0
2023-11-21 13:24:34,774 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11500, loss[loss=0.07355, simple_loss=0.09103, pruned_loss=0.01896, audio_tagging_loss=0.009079, over 15588.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09472, pruned_loss=0.01644, audio_tagging_loss=0.009571, over 3054605.41 frames. ], batch size: 58, lr: 3.57e-03, grad_scale: 32.0
2023-11-21 13:24:42,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1519486.6666666667, ans=0.2
2023-11-21 13:24:47,479 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.25 vs. limit=6.0
2023-11-21 13:24:58,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1519553.3333333333, ans=0.125
2023-11-21 13:25:05,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1519620.0, ans=0.125
2023-11-21 13:25:07,791 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 227950
2023-11-21 13:25:10,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1519620.0, ans=0.1
2023-11-21 13:25:18,649 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.513e+01 8.149e+01 8.832e+01 9.671e+01 1.256e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 13:25:38,554 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11550, loss[loss=0.0872, simple_loss=0.1259, pruned_loss=0.0157, audio_tagging_loss=0.008549, over 15831.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09541, pruned_loss=0.01628, audio_tagging_loss=0.009494, over 3058392.02 frames. ], batch size: 57, lr: 3.56e-03, grad_scale: 16.0
2023-11-21 13:26:07,801 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.63 vs. limit=10.0
2023-11-21 13:26:10,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228000
2023-11-21 13:26:12,362 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-228000.pt
2023-11-21 13:26:20,424 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 13:26:23,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1520020.0, ans=0.125
2023-11-21 13:26:34,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1520086.6666666667, ans=0.2
2023-11-21 13:26:47,326 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11600, loss[loss=0.09624, simple_loss=0.134, pruned_loss=0.02153, audio_tagging_loss=0.007722, over 15611.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09622, pruned_loss=0.01655, audio_tagging_loss=0.009509, over 3055039.61 frames. ], batch size: 56, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:26:56,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1520153.3333333333, ans=0.0
2023-11-21 13:27:18,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228050
2023-11-21 13:27:28,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1520353.3333333333, ans=0.0
2023-11-21 13:27:32,541 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.674e+01 8.168e+01 8.737e+01 9.633e+01 1.247e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 13:27:38,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1520420.0, ans=0.125
2023-11-21 13:27:40,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1520420.0, ans=0.025
2023-11-21 13:27:42,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1520420.0, ans=0.125
2023-11-21 13:27:45,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1520420.0, ans=0.0
2023-11-21 13:27:51,770 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11650, loss[loss=0.08082, simple_loss=0.1071, pruned_loss=0.0156, audio_tagging_loss=0.01169, over 15813.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09541, pruned_loss=0.01648, audio_tagging_loss=0.009558, over 3049740.74 frames. ], batch size: 56, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:28:05,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1520553.3333333333, ans=0.125
2023-11-21 13:28:11,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1520553.3333333333, ans=0.125
2023-11-21 13:28:15,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1520553.3333333333, ans=0.125
2023-11-21 13:28:24,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228100
2023-11-21 13:28:25,251 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.98 vs. limit=22.5
2023-11-21 13:28:49,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1520753.3333333333, ans=0.125
2023-11-21 13:28:54,841 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11700, loss[loss=0.07648, simple_loss=0.09865, pruned_loss=0.01711, audio_tagging_loss=0.01005, over 15708.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09481, pruned_loss=0.01635, audio_tagging_loss=0.009638, over 3055814.54 frames. ], batch size: 60, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:28:55,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1520820.0, ans=0.125
2023-11-21 13:28:59,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1520820.0, ans=0.2
2023-11-21 13:29:00,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1520820.0, ans=0.2
2023-11-21 13:29:11,807 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=7.12 vs. limit=10.0
2023-11-21 13:29:14,954 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.85 vs. limit=12.0
2023-11-21 13:29:15,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1520886.6666666667, ans=0.1
2023-11-21 13:29:27,537 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228150
2023-11-21 13:29:38,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1521020.0, ans=0.0
2023-11-21 13:29:39,518 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.045e+01 8.764e+01 9.289e+01 1.171e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 13:29:50,642 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.37 vs. limit=15.0
2023-11-21 13:29:55,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1521086.6666666667, ans=0.0
2023-11-21 13:29:59,679 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11750, loss[loss=0.06881, simple_loss=0.09316, pruned_loss=0.01575, audio_tagging_loss=0.006481, over 15814.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09552, pruned_loss=0.0166, audio_tagging_loss=0.009538, over 3051954.94 frames. ], batch size: 60, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:30:18,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1521220.0, ans=0.0
2023-11-21 13:30:30,699 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228200
2023-11-21 13:30:30,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1521286.6666666667, ans=0.0
2023-11-21 13:30:42,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1521353.3333333333, ans=0.125
2023-11-21 13:30:54,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1521420.0, ans=0.125
2023-11-21 13:31:03,790 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11800, loss[loss=0.09639, simple_loss=0.1201, pruned_loss=0.0258, audio_tagging_loss=0.01056, over 15522.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09482, pruned_loss=0.01642, audio_tagging_loss=0.009531, over 3055147.95 frames. ], batch size: 54, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:31:23,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1521553.3333333333, ans=0.0
2023-11-21 13:31:26,890 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.64 vs. limit=22.5
2023-11-21 13:31:28,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1521620.0, ans=0.0
2023-11-21 13:31:29,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1521620.0, ans=0.125
2023-11-21 13:31:35,856 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228250
2023-11-21 13:31:48,739 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.292e+01 8.361e+01 9.006e+01 1.004e+02 1.264e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-21 13:32:07,052 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11850, loss[loss=0.09351, simple_loss=0.1244, pruned_loss=0.02189, audio_tagging_loss=0.009402, over 14855.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09585, pruned_loss=0.01671, audio_tagging_loss=0.009558, over 3053507.16 frames. ], batch size: 54, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:32:07,337 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:32:11,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1521820.0, ans=0.1
2023-11-21 13:32:25,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1521886.6666666667, ans=0.0
2023-11-21 13:32:40,341 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228300
2023-11-21 13:33:11,970 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11900, loss[loss=0.07792, simple_loss=0.1058, pruned_loss=0.01511, audio_tagging_loss=0.009936, over 15467.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09643, pruned_loss=0.01665, audio_tagging_loss=0.009584, over 3049481.62 frames. ], batch size: 56, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:33:15,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1522153.3333333333, ans=0.125
2023-11-21 13:33:26,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.36 vs. limit=10.0
2023-11-21 13:33:43,882 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228350
2023-11-21 13:33:45,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1522286.6666666667, ans=0.1
2023-11-21 13:33:56,597 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.221e+01 8.725e+01 9.501e+01 1.211e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 13:34:04,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1522420.0, ans=0.125
2023-11-21 13:34:16,710 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 11950, loss[loss=0.06944, simple_loss=0.09119, pruned_loss=0.01524, audio_tagging_loss=0.008612, over 14889.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09556, pruned_loss=0.01658, audio_tagging_loss=0.009764, over 3056542.73 frames. ], batch size: 55, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:34:16,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1522486.6666666667, ans=0.125
2023-11-21 13:34:37,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1522553.3333333333, ans=0.125
2023-11-21 13:34:43,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1522620.0, ans=0.1
2023-11-21 13:34:47,877 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228400
2023-11-21 13:34:54,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1522686.6666666667, ans=0.125
2023-11-21 13:34:59,706 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.11 vs. limit=22.5
2023-11-21 13:35:00,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1522686.6666666667, ans=0.125
2023-11-21 13:35:01,440 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:35:18,689 INFO [train_asr.py:1221] (0/4) Epoch 19, batch 12000, loss[loss=0.09403, simple_loss=0.1355, pruned_loss=0.01781, audio_tagging_loss=0.008474, over 15171.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09638, pruned_loss=0.01656, audio_tagging_loss=0.00978, over 3057708.81 frames. ], batch size: 54, lr: 3.56e-03, grad_scale: 32.0
2023-11-21 13:35:18,692 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 13:36:02,005 INFO [train_asr.py:1253] (0/4) Epoch 19, validation: loss=0.06018, simple_loss=0.05227, pruned_loss=0.005307, audio_tagging_loss=0.02874, over 4681554.00 frames. 
2023-11-21 13:36:02,006 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 13:36:06,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1522820.0, ans=0.0
2023-11-21 13:36:11,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1522820.0, ans=0.2
2023-11-21 13:36:17,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.01 vs. limit=22.5
2023-11-21 13:36:31,612 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-19.pt
2023-11-21 13:37:05,051 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 0, loss[loss=0.07557, simple_loss=0.0786, pruned_loss=0.01103, audio_tagging_loss=0.02524, over 14318.00 frames. ], tot_loss[loss=0.07557, simple_loss=0.0786, pruned_loss=0.01103, audio_tagging_loss=0.02524, over 14318.00 frames. ], batch size: 54, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:37:05,054 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 13:37:31,698 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7892, 5.8311, 5.8839, 5.8712], device='cuda:0')
2023-11-21 13:37:41,805 INFO [train_asr.py:1253] (0/4) Epoch 20, validation: loss=0.05938, simple_loss=0.0523, pruned_loss=0.005287, audio_tagging_loss=0.02794, over 4681554.00 frames. 
2023-11-21 13:37:41,806 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 13:37:43,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228450
2023-11-21 13:37:47,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.40 vs. limit=15.0
2023-11-21 13:37:55,095 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.213e+01 9.034e+01 9.702e+01 1.198e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-21 13:38:00,460 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:38:14,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1523113.3333333333, ans=0.125
2023-11-21 13:38:20,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1523180.0, ans=0.125
2023-11-21 13:38:24,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1523180.0, ans=0.05
2023-11-21 13:38:27,255 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.16 vs. limit=6.0
2023-11-21 13:38:35,654 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.42 vs. limit=6.0
2023-11-21 13:38:39,517 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.87 vs. limit=22.5
2023-11-21 13:38:44,850 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 50, loss[loss=0.07838, simple_loss=0.09162, pruned_loss=0.01492, audio_tagging_loss=0.01764, over 14254.00 frames. ], tot_loss[loss=0.08134, simple_loss=0.09241, pruned_loss=0.01639, audio_tagging_loss=0.01875, over 678731.28 frames. ], batch size: 53, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:38:46,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228500
2023-11-21 13:39:23,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1523513.3333333333, ans=0.125
2023-11-21 13:39:24,702 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.92 vs. limit=15.0
2023-11-21 13:39:49,080 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 100, loss[loss=0.06716, simple_loss=0.07023, pruned_loss=0.01202, audio_tagging_loss=0.02002, over 15181.00 frames. ], tot_loss[loss=0.08087, simple_loss=0.09308, pruned_loss=0.01604, audio_tagging_loss=0.01829, over 1205973.45 frames. ], batch size: 57, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:39:50,353 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228550
2023-11-21 13:39:50,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1523646.6666666667, ans=0.125
2023-11-21 13:40:01,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1523713.3333333333, ans=0.125
2023-11-21 13:40:04,259 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.140e+01 8.678e+01 9.460e+01 1.029e+02 1.398e+02, threshold=1.892e+02, percent-clipped=0.0
2023-11-21 13:40:13,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1523713.3333333333, ans=0.0
2023-11-21 13:40:23,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1523780.0, ans=0.1
2023-11-21 13:40:23,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1523780.0, ans=0.125
2023-11-21 13:40:42,405 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.29 vs. limit=15.0
2023-11-21 13:40:43,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1523913.3333333333, ans=0.125
2023-11-21 13:40:54,004 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 150, loss[loss=0.07316, simple_loss=0.08891, pruned_loss=0.01519, audio_tagging_loss=0.01352, over 15634.00 frames. ], tot_loss[loss=0.07997, simple_loss=0.09452, pruned_loss=0.01642, audio_tagging_loss=0.01629, over 1616897.07 frames. ], batch size: 61, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:40:55,989 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228600
2023-11-21 13:41:07,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1524046.6666666667, ans=0.1
2023-11-21 13:41:58,551 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 200, loss[loss=0.05734, simple_loss=0.06825, pruned_loss=0.01102, audio_tagging_loss=0.01219, over 15885.00 frames. ], tot_loss[loss=0.07841, simple_loss=0.09526, pruned_loss=0.0165, audio_tagging_loss=0.01428, over 1931869.56 frames. ], batch size: 60, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:41:59,800 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228650
2023-11-21 13:42:01,541 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.65 vs. limit=10.0
2023-11-21 13:42:01,952 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.84 vs. limit=8.0
2023-11-21 13:42:12,005 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.014e+01 8.304e+01 8.770e+01 9.399e+01 1.399e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 13:42:12,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1524380.0, ans=0.0
2023-11-21 13:42:25,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1524446.6666666667, ans=0.1
2023-11-21 13:42:30,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1524446.6666666667, ans=0.125
2023-11-21 13:42:41,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1524513.3333333333, ans=0.1
2023-11-21 13:42:53,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1524580.0, ans=0.0
2023-11-21 13:43:02,040 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 250, loss[loss=0.07397, simple_loss=0.09926, pruned_loss=0.01382, audio_tagging_loss=0.01052, over 15571.00 frames. ], tot_loss[loss=0.07623, simple_loss=0.09445, pruned_loss=0.01616, audio_tagging_loss=0.01285, over 2178065.13 frames. ], batch size: 57, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:43:03,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228700
2023-11-21 13:43:07,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1524646.6666666667, ans=0.125
2023-11-21 13:43:17,901 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.34 vs. limit=10.0
2023-11-21 13:43:19,347 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.42 vs. limit=8.0
2023-11-21 13:43:22,224 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:43:56,107 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.77 vs. limit=15.0
2023-11-21 13:44:07,328 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 300, loss[loss=0.07786, simple_loss=0.09893, pruned_loss=0.01668, audio_tagging_loss=0.01171, over 15888.00 frames. ], tot_loss[loss=0.07531, simple_loss=0.09459, pruned_loss=0.01622, audio_tagging_loss=0.0118, over 2370843.19 frames. ], batch size: 57, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:44:08,645 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228750
2023-11-21 13:44:21,332 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.166e+01 8.999e+01 9.797e+01 1.140e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-21 13:44:26,836 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.74 vs. limit=15.0
2023-11-21 13:44:44,948 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=14.23 vs. limit=15.0
2023-11-21 13:45:03,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1525246.6666666667, ans=0.125
2023-11-21 13:45:11,673 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 350, loss[loss=0.08622, simple_loss=0.1131, pruned_loss=0.0216, audio_tagging_loss=0.008082, over 15339.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.09506, pruned_loss=0.01628, audio_tagging_loss=0.01116, over 2516695.71 frames. ], batch size: 60, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:45:12,922 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228800
2023-11-21 13:45:32,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1525380.0, ans=0.0
2023-11-21 13:45:38,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1525446.6666666667, ans=0.1
2023-11-21 13:45:53,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1525513.3333333333, ans=0.025
2023-11-21 13:46:04,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1525580.0, ans=0.125
2023-11-21 13:46:15,856 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 400, loss[loss=0.08812, simple_loss=0.1189, pruned_loss=0.02032, audio_tagging_loss=0.00833, over 14880.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09392, pruned_loss=0.01609, audio_tagging_loss=0.01092, over 2633417.01 frames. ], batch size: 56, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:46:17,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228850
2023-11-21 13:46:20,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1525646.6666666667, ans=0.125
2023-11-21 13:46:22,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1525646.6666666667, ans=0.125
2023-11-21 13:46:26,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1525646.6666666667, ans=0.1
2023-11-21 13:46:26,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1525646.6666666667, ans=0.2
2023-11-21 13:46:30,451 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.562e+01 8.083e+01 8.649e+01 9.202e+01 1.163e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 13:46:35,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.74 vs. limit=22.5
2023-11-21 13:46:36,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1525713.3333333333, ans=0.125
2023-11-21 13:47:15,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1525913.3333333333, ans=0.125
2023-11-21 13:47:20,809 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 450, loss[loss=0.08788, simple_loss=0.1225, pruned_loss=0.01767, audio_tagging_loss=0.008949, over 14828.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09397, pruned_loss=0.01617, audio_tagging_loss=0.01044, over 2723291.53 frames. ], batch size: 55, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:47:21,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1525980.0, ans=0.125
2023-11-21 13:47:22,692 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228900
2023-11-21 13:47:22,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1525980.0, ans=0.2
2023-11-21 13:47:27,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1525980.0, ans=0.125
2023-11-21 13:47:50,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1526113.3333333333, ans=0.125
2023-11-21 13:47:55,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1526113.3333333333, ans=0.0
2023-11-21 13:47:56,631 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.37 vs. limit=6.0
2023-11-21 13:48:04,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1526180.0, ans=0.0
2023-11-21 13:48:24,758 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 500, loss[loss=0.06412, simple_loss=0.08115, pruned_loss=0.0133, audio_tagging_loss=0.01024, over 14310.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09468, pruned_loss=0.01629, audio_tagging_loss=0.01019, over 2795612.43 frames. ], batch size: 55, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:48:26,111 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 228950
2023-11-21 13:48:38,752 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.882e+01 7.917e+01 8.636e+01 9.467e+01 1.294e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-21 13:49:28,870 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 550, loss[loss=0.05397, simple_loss=0.07052, pruned_loss=0.009039, audio_tagging_loss=0.00967, over 15055.00 frames. ], tot_loss[loss=0.07389, simple_loss=0.0951, pruned_loss=0.01629, audio_tagging_loss=0.01006, over 2847272.73 frames. ], batch size: 57, lr: 3.47e-03, grad_scale: 32.0
2023-11-21 13:49:30,150 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229000
2023-11-21 13:49:33,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1526646.6666666667, ans=0.125
2023-11-21 13:50:01,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.88 vs. limit=15.0
2023-11-21 13:50:02,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1526780.0, ans=0.1
2023-11-21 13:50:06,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1526846.6666666667, ans=15.0
2023-11-21 13:50:30,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1526913.3333333333, ans=0.2
2023-11-21 13:50:33,695 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 600, loss[loss=0.07552, simple_loss=0.1098, pruned_loss=0.01502, audio_tagging_loss=0.005576, over 14454.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.09613, pruned_loss=0.01637, audio_tagging_loss=0.009843, over 2896937.12 frames. ], batch size: 54, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:50:35,088 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229050
2023-11-21 13:50:39,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1526980.0, ans=0.0
2023-11-21 13:50:47,732 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.121e+01 7.930e+01 8.613e+01 9.327e+01 1.199e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 13:50:53,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1527046.6666666667, ans=0.1
2023-11-21 13:51:07,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1527113.3333333333, ans=0.125
2023-11-21 13:51:08,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1527113.3333333333, ans=0.0
2023-11-21 13:51:12,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1527180.0, ans=0.07
2023-11-21 13:51:34,085 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.61 vs. limit=15.0
2023-11-21 13:51:38,003 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 650, loss[loss=0.05718, simple_loss=0.07278, pruned_loss=0.008868, audio_tagging_loss=0.01192, over 14780.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09636, pruned_loss=0.01646, audio_tagging_loss=0.009802, over 2930221.36 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:51:39,311 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229100
2023-11-21 13:51:56,219 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.35 vs. limit=6.0
2023-11-21 13:51:59,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1527380.0, ans=0.0
2023-11-21 13:52:14,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1527446.6666666667, ans=0.1
2023-11-21 13:52:41,378 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 700, loss[loss=0.07929, simple_loss=0.09308, pruned_loss=0.0217, audio_tagging_loss=0.01105, over 14544.00 frames. ], tot_loss[loss=0.07495, simple_loss=0.09705, pruned_loss=0.01673, audio_tagging_loss=0.009694, over 2958203.88 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:52:43,368 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229150
2023-11-21 13:52:54,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1527713.3333333333, ans=0.1
2023-11-21 13:52:55,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1527713.3333333333, ans=0.0
2023-11-21 13:52:56,214 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.031e+01 8.760e+01 9.624e+01 1.233e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 13:53:01,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1527713.3333333333, ans=0.2
2023-11-21 13:53:17,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1527780.0, ans=0.0
2023-11-21 13:53:45,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1527913.3333333333, ans=0.125
2023-11-21 13:53:47,362 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 750, loss[loss=0.09279, simple_loss=0.1265, pruned_loss=0.0208, audio_tagging_loss=0.008725, over 15964.00 frames. ], tot_loss[loss=0.07517, simple_loss=0.09721, pruned_loss=0.01688, audio_tagging_loss=0.009684, over 2976510.68 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:53:48,692 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229200
2023-11-21 13:54:18,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1528113.3333333333, ans=0.125
2023-11-21 13:54:20,141 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:54:52,196 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.67 vs. limit=15.0
2023-11-21 13:54:52,686 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 800, loss[loss=0.08132, simple_loss=0.1074, pruned_loss=0.01897, audio_tagging_loss=0.008663, over 15082.00 frames. ], tot_loss[loss=0.07544, simple_loss=0.09752, pruned_loss=0.01689, audio_tagging_loss=0.009784, over 2997498.47 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:54:54,010 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229250
2023-11-21 13:54:54,636 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.41 vs. limit=22.5
2023-11-21 13:55:06,304 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 8.111e+01 8.814e+01 9.591e+01 1.302e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 13:55:16,543 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.51 vs. limit=15.0
2023-11-21 13:55:21,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1528446.6666666667, ans=0.0
2023-11-21 13:55:35,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1528513.3333333333, ans=0.1
2023-11-21 13:55:44,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1528580.0, ans=0.1
2023-11-21 13:55:45,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1528580.0, ans=0.125
2023-11-21 13:55:55,495 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.00 vs. limit=15.0
2023-11-21 13:55:57,284 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 850, loss[loss=0.06881, simple_loss=0.08537, pruned_loss=0.01677, audio_tagging_loss=0.009359, over 14780.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09734, pruned_loss=0.01681, audio_tagging_loss=0.009848, over 3015683.16 frames. ], batch size: 58, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 13:55:58,572 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229300
2023-11-21 13:56:07,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1528646.6666666667, ans=10.0
2023-11-21 13:56:11,276 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.89 vs. limit=12.0
2023-11-21 13:56:12,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1528713.3333333333, ans=0.0
2023-11-21 13:56:30,013 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.35 vs. limit=6.0
2023-11-21 13:56:30,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1528780.0, ans=0.07
2023-11-21 13:56:36,381 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.56 vs. limit=15.0
2023-11-21 13:56:48,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.64 vs. limit=10.0
2023-11-21 13:57:02,865 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 900, loss[loss=0.0841, simple_loss=0.1071, pruned_loss=0.02033, audio_tagging_loss=0.01022, over 15149.00 frames. ], tot_loss[loss=0.07615, simple_loss=0.09851, pruned_loss=0.01703, audio_tagging_loss=0.009864, over 3025318.75 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 13:57:04,168 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229350
2023-11-21 13:57:14,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1528980.0, ans=0.1
2023-11-21 13:57:19,060 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.780e+01 8.022e+01 8.539e+01 9.514e+01 1.359e+02, threshold=1.708e+02, percent-clipped=0.0
2023-11-21 13:57:20,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1529046.6666666667, ans=0.0
2023-11-21 13:57:45,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1529180.0, ans=0.0
2023-11-21 13:57:58,085 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 13:57:59,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1529246.6666666667, ans=0.07
2023-11-21 13:58:08,188 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.23 vs. limit=15.0
2023-11-21 13:58:08,505 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 950, loss[loss=0.07674, simple_loss=0.1105, pruned_loss=0.01316, audio_tagging_loss=0.00833, over 15547.00 frames. ], tot_loss[loss=0.07549, simple_loss=0.09757, pruned_loss=0.01691, audio_tagging_loss=0.00979, over 3028151.33 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 13:58:09,964 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229400
2023-11-21 13:58:16,974 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.66 vs. limit=10.0
2023-11-21 13:58:29,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1529380.0, ans=0.125
2023-11-21 13:58:32,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.28 vs. limit=12.0
2023-11-21 13:58:32,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1529446.6666666667, ans=0.2
2023-11-21 13:58:33,225 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.67 vs. limit=15.0
2023-11-21 13:58:59,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1529580.0, ans=0.0
2023-11-21 13:58:59,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1529580.0, ans=0.125
2023-11-21 13:59:12,408 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1000, loss[loss=0.05688, simple_loss=0.07042, pruned_loss=0.009664, audio_tagging_loss=0.01201, over 14577.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.09797, pruned_loss=0.01684, audio_tagging_loss=0.009529, over 3039384.16 frames. ], batch size: 54, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 13:59:13,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229450
2023-11-21 13:59:27,764 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.761e+01 8.099e+01 8.790e+01 9.899e+01 1.209e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 13:59:37,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1529780.0, ans=0.125
2023-11-21 13:59:41,908 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:00:16,821 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1050, loss[loss=0.05259, simple_loss=0.06754, pruned_loss=0.01115, audio_tagging_loss=0.00767, over 14138.00 frames. ], tot_loss[loss=0.07487, simple_loss=0.09758, pruned_loss=0.0167, audio_tagging_loss=0.009389, over 3041651.68 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:00:18,146 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229500
2023-11-21 14:00:34,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1530046.6666666667, ans=0.125
2023-11-21 14:00:44,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1530113.3333333333, ans=0.125
2023-11-21 14:00:49,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1530113.3333333333, ans=0.125
2023-11-21 14:00:54,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1530180.0, ans=0.0
2023-11-21 14:01:05,224 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.94 vs. limit=22.5
2023-11-21 14:01:12,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1530246.6666666667, ans=0.0
2023-11-21 14:01:23,285 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1100, loss[loss=0.08561, simple_loss=0.1126, pruned_loss=0.02155, audio_tagging_loss=0.007762, over 15534.00 frames. ], tot_loss[loss=0.0746, simple_loss=0.09717, pruned_loss=0.01667, audio_tagging_loss=0.00935, over 3037701.99 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:01:24,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229550
2023-11-21 14:01:27,036 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:01:29,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1530313.3333333333, ans=0.0
2023-11-21 14:01:38,195 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.405e+01 8.211e+01 9.088e+01 9.888e+01 2.167e+02, threshold=1.818e+02, percent-clipped=1.0
2023-11-21 14:01:42,243 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:01:44,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.95 vs. limit=22.5
2023-11-21 14:02:24,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1530580.0, ans=0.125
2023-11-21 14:02:25,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1530580.0, ans=0.0
2023-11-21 14:02:28,033 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1150, loss[loss=0.08534, simple_loss=0.1179, pruned_loss=0.01778, audio_tagging_loss=0.008607, over 14845.00 frames. ], tot_loss[loss=0.07478, simple_loss=0.09741, pruned_loss=0.01672, audio_tagging_loss=0.009357, over 3041200.88 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:02:29,376 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229600
2023-11-21 14:02:59,905 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:03:05,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1530780.0, ans=0.125
2023-11-21 14:03:10,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1530846.6666666667, ans=0.1
2023-11-21 14:03:31,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1530980.0, ans=0.04949747468305833
2023-11-21 14:03:32,047 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1200, loss[loss=0.09023, simple_loss=0.1108, pruned_loss=0.02503, audio_tagging_loss=0.009804, over 13944.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09746, pruned_loss=0.01677, audio_tagging_loss=0.009263, over 3040161.05 frames. ], batch size: 53, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:03:33,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229650
2023-11-21 14:03:37,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1530980.0, ans=0.1
2023-11-21 14:03:48,979 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.148e+01 8.693e+01 9.555e+01 1.704e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 14:03:49,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.14 vs. limit=22.5
2023-11-21 14:04:06,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1531113.3333333333, ans=0.125
2023-11-21 14:04:38,662 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1250, loss[loss=0.09071, simple_loss=0.1137, pruned_loss=0.02547, audio_tagging_loss=0.008376, over 15959.00 frames. ], tot_loss[loss=0.07477, simple_loss=0.09718, pruned_loss=0.01687, audio_tagging_loss=0.009309, over 3045613.06 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:04:40,035 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229700
2023-11-21 14:04:58,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1531380.0, ans=0.0
2023-11-21 14:05:03,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1531446.6666666667, ans=0.0
2023-11-21 14:05:09,836 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:05:11,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1531446.6666666667, ans=0.2
2023-11-21 14:05:43,345 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1300, loss[loss=0.06879, simple_loss=0.08675, pruned_loss=0.01742, audio_tagging_loss=0.007999, over 14730.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.09681, pruned_loss=0.01673, audio_tagging_loss=0.009433, over 3045357.68 frames. ], batch size: 59, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:05:44,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229750
2023-11-21 14:05:49,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1531646.6666666667, ans=0.07
2023-11-21 14:05:58,186 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.180e+01 8.668e+01 9.567e+01 1.145e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 14:06:10,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1531780.0, ans=0.025
2023-11-21 14:06:13,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1531780.0, ans=0.125
2023-11-21 14:06:21,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1531780.0, ans=0.0
2023-11-21 14:06:39,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1531913.3333333333, ans=0.05
2023-11-21 14:06:48,137 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1350, loss[loss=0.08206, simple_loss=0.1124, pruned_loss=0.01626, audio_tagging_loss=0.009611, over 15165.00 frames. ], tot_loss[loss=0.0746, simple_loss=0.09668, pruned_loss=0.01673, audio_tagging_loss=0.009528, over 3044059.76 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:06:49,457 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229800
2023-11-21 14:07:06,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1532046.6666666667, ans=0.125
2023-11-21 14:07:11,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1532046.6666666667, ans=0.125
2023-11-21 14:07:27,752 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.91 vs. limit=15.0
2023-11-21 14:07:35,738 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:07:36,586 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.51 vs. limit=15.0
2023-11-21 14:07:49,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1532246.6666666667, ans=0.125
2023-11-21 14:07:55,124 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1400, loss[loss=0.08114, simple_loss=0.09788, pruned_loss=0.0218, audio_tagging_loss=0.0104, over 14953.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09755, pruned_loss=0.01705, audio_tagging_loss=0.009508, over 3043577.11 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:07:56,467 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229850
2023-11-21 14:08:00,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1532313.3333333333, ans=0.2
2023-11-21 14:08:01,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1532313.3333333333, ans=0.125
2023-11-21 14:08:07,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1532380.0, ans=0.1
2023-11-21 14:08:10,531 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.288e+01 8.939e+01 9.468e+01 2.052e+02, threshold=1.788e+02, percent-clipped=1.0
2023-11-21 14:08:25,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1532446.6666666667, ans=0.1
2023-11-21 14:08:26,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1532446.6666666667, ans=0.0
2023-11-21 14:08:39,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1532513.3333333333, ans=0.0
2023-11-21 14:08:52,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1532580.0, ans=0.0
2023-11-21 14:08:59,455 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1450, loss[loss=0.07321, simple_loss=0.09496, pruned_loss=0.015, audio_tagging_loss=0.01074, over 16759.00 frames. ], tot_loss[loss=0.07504, simple_loss=0.09718, pruned_loss=0.01687, audio_tagging_loss=0.009573, over 3051417.76 frames. ], batch size: 63, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:09:00,863 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229900
2023-11-21 14:09:11,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.whiten.whitening_limit, batch_count=1532713.3333333333, ans=12.0
2023-11-21 14:09:27,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1532780.0, ans=10.0
2023-11-21 14:09:37,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1532846.6666666667, ans=0.1
2023-11-21 14:09:37,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1532846.6666666667, ans=0.125
2023-11-21 14:09:41,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1532846.6666666667, ans=0.0
2023-11-21 14:09:51,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1532913.3333333333, ans=0.125
2023-11-21 14:10:03,594 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1500, loss[loss=0.05051, simple_loss=0.06184, pruned_loss=0.006964, audio_tagging_loss=0.01263, over 14888.00 frames. ], tot_loss[loss=0.07531, simple_loss=0.09768, pruned_loss=0.01689, audio_tagging_loss=0.009585, over 3054346.42 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:10:04,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 229950
2023-11-21 14:10:04,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1532980.0, ans=0.1
2023-11-21 14:10:06,779 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.54 vs. limit=15.0
2023-11-21 14:10:18,864 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.143e+01 8.696e+01 9.361e+01 1.676e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 14:10:24,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1533046.6666666667, ans=0.0
2023-11-21 14:10:25,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1533046.6666666667, ans=0.125
2023-11-21 14:10:58,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.71 vs. limit=15.0
2023-11-21 14:11:08,125 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1550, loss[loss=0.08582, simple_loss=0.1026, pruned_loss=0.02193, audio_tagging_loss=0.0126, over 15643.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.0963, pruned_loss=0.01658, audio_tagging_loss=0.009785, over 3056941.26 frames. ], batch size: 59, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:11:10,145 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230000
2023-11-21 14:11:28,209 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.21 vs. limit=22.5
2023-11-21 14:11:37,781 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:11:39,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1533446.6666666667, ans=0.125
2023-11-21 14:11:42,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1533446.6666666667, ans=0.1
2023-11-21 14:12:03,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1533580.0, ans=0.125
2023-11-21 14:12:14,365 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1600, loss[loss=0.07013, simple_loss=0.09587, pruned_loss=0.01328, audio_tagging_loss=0.008922, over 14491.00 frames. ], tot_loss[loss=0.075, simple_loss=0.09672, pruned_loss=0.01677, audio_tagging_loss=0.009867, over 3051330.70 frames. ], batch size: 54, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:12:16,320 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230050
2023-11-21 14:12:26,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1533713.3333333333, ans=0.125
2023-11-21 14:12:30,015 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.321e+01 8.893e+01 9.688e+01 1.461e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 14:12:32,056 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.57 vs. limit=10.0
2023-11-21 14:12:32,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1533713.3333333333, ans=0.1
2023-11-21 14:12:37,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1533713.3333333333, ans=0.125
2023-11-21 14:12:51,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1533780.0, ans=0.1
2023-11-21 14:12:54,080 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.92 vs. limit=15.0
2023-11-21 14:13:07,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1533913.3333333333, ans=0.125
2023-11-21 14:13:19,685 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1650, loss[loss=0.07629, simple_loss=0.09702, pruned_loss=0.01957, audio_tagging_loss=0.008212, over 15202.00 frames. ], tot_loss[loss=0.07504, simple_loss=0.09666, pruned_loss=0.01673, audio_tagging_loss=0.009986, over 3050794.36 frames. ], batch size: 61, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:13:21,092 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230100
2023-11-21 14:13:39,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1534046.6666666667, ans=0.2
2023-11-21 14:13:52,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1534113.3333333333, ans=0.0
2023-11-21 14:13:56,835 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.55 vs. limit=15.0
2023-11-21 14:14:02,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1534180.0, ans=0.125
2023-11-21 14:14:12,761 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.90 vs. limit=22.5
2023-11-21 14:14:17,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.90 vs. limit=15.0
2023-11-21 14:14:18,715 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.98 vs. limit=15.0
2023-11-21 14:14:24,714 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1700, loss[loss=0.07787, simple_loss=0.106, pruned_loss=0.01421, audio_tagging_loss=0.01064, over 16207.00 frames. ], tot_loss[loss=0.0754, simple_loss=0.09716, pruned_loss=0.01684, audio_tagging_loss=0.009979, over 3048721.93 frames. ], batch size: 57, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:14:26,034 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230150
2023-11-21 14:14:39,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1534380.0, ans=0.2
2023-11-21 14:14:40,772 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.297e+01 8.260e+01 8.837e+01 9.595e+01 1.453e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 14:14:46,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1534380.0, ans=0.0
2023-11-21 14:14:47,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1534380.0, ans=0.2
2023-11-21 14:14:52,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1534446.6666666667, ans=0.0
2023-11-21 14:14:54,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.00 vs. limit=6.0
2023-11-21 14:14:57,034 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=13.33 vs. limit=15.0
2023-11-21 14:15:08,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1534513.3333333333, ans=0.0
2023-11-21 14:15:22,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1534580.0, ans=0.125
2023-11-21 14:15:27,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1534580.0, ans=0.125
2023-11-21 14:15:30,390 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1750, loss[loss=0.07506, simple_loss=0.09073, pruned_loss=0.02005, audio_tagging_loss=0.009646, over 14531.00 frames. ], tot_loss[loss=0.07464, simple_loss=0.09617, pruned_loss=0.01665, audio_tagging_loss=0.009903, over 3035746.87 frames. ], batch size: 56, lr: 3.46e-03, grad_scale: 32.0
2023-11-21 14:15:31,710 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230200
2023-11-21 14:15:31,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1534646.6666666667, ans=0.125
2023-11-21 14:15:37,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1534646.6666666667, ans=0.0
2023-11-21 14:15:37,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1534646.6666666667, ans=0.125
2023-11-21 14:15:48,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1534713.3333333333, ans=0.0
2023-11-21 14:15:57,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1534780.0, ans=0.09899494936611666
2023-11-21 14:16:00,683 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.80 vs. limit=15.0
2023-11-21 14:16:12,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1534846.6666666667, ans=0.125
2023-11-21 14:16:32,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1534913.3333333333, ans=0.125
2023-11-21 14:16:34,749 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1800, loss[loss=0.06889, simple_loss=0.0899, pruned_loss=0.01335, audio_tagging_loss=0.01059, over 14085.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.09672, pruned_loss=0.01678, audio_tagging_loss=0.009676, over 3035651.56 frames. ], batch size: 53, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:16:36,046 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230250
2023-11-21 14:16:50,983 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.24 vs. limit=15.0
2023-11-21 14:16:51,331 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.667e+01 8.097e+01 8.882e+01 9.608e+01 1.325e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-21 14:16:57,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1535046.6666666667, ans=0.0
2023-11-21 14:17:02,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1535113.3333333333, ans=0.0
2023-11-21 14:17:02,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1535113.3333333333, ans=0.0
2023-11-21 14:17:06,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1535113.3333333333, ans=0.125
2023-11-21 14:17:16,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1535180.0, ans=0.125
2023-11-21 14:17:39,510 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1850, loss[loss=0.08861, simple_loss=0.1122, pruned_loss=0.02421, audio_tagging_loss=0.008296, over 15392.00 frames. ], tot_loss[loss=0.0751, simple_loss=0.09719, pruned_loss=0.01691, audio_tagging_loss=0.009592, over 3039559.28 frames. ], batch size: 58, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:17:40,787 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230300
2023-11-21 14:17:44,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1535313.3333333333, ans=0.125
2023-11-21 14:17:48,634 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.09 vs. limit=15.0
2023-11-21 14:17:54,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1535380.0, ans=0.1
2023-11-21 14:18:21,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1535513.3333333333, ans=0.125
2023-11-21 14:18:44,317 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:18:45,300 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1900, loss[loss=0.06019, simple_loss=0.07281, pruned_loss=0.01253, audio_tagging_loss=0.01126, over 13665.00 frames. ], tot_loss[loss=0.07418, simple_loss=0.09631, pruned_loss=0.01647, audio_tagging_loss=0.009557, over 3036193.16 frames. ], batch size: 55, lr: 3.46e-03, grad_scale: 16.0
2023-11-21 14:18:46,570 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230350
2023-11-21 14:18:47,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1535646.6666666667, ans=0.125
2023-11-21 14:18:48,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1535646.6666666667, ans=0.125
2023-11-21 14:19:00,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1535713.3333333333, ans=0.1
2023-11-21 14:19:01,199 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.472e+01 7.941e+01 8.654e+01 9.429e+01 1.278e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 14:19:09,829 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.01 vs. limit=15.0
2023-11-21 14:19:16,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1535780.0, ans=0.125
2023-11-21 14:19:32,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1535846.6666666667, ans=0.2
2023-11-21 14:19:33,731 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.36 vs. limit=12.0
2023-11-21 14:19:35,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1535913.3333333333, ans=0.0
2023-11-21 14:19:38,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1535913.3333333333, ans=0.125
2023-11-21 14:19:49,115 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 1950, loss[loss=0.05391, simple_loss=0.06502, pruned_loss=0.008292, audio_tagging_loss=0.01311, over 15726.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.0961, pruned_loss=0.01627, audio_tagging_loss=0.009485, over 3027269.16 frames. ], batch size: 63, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:19:50,472 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230400
2023-11-21 14:20:05,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1536046.6666666667, ans=0.1
2023-11-21 14:20:26,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1536113.3333333333, ans=0.0
2023-11-21 14:20:54,220 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2000, loss[loss=0.06644, simple_loss=0.08147, pruned_loss=0.01702, audio_tagging_loss=0.008683, over 14053.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.09529, pruned_loss=0.01619, audio_tagging_loss=0.009499, over 3032525.70 frames. ], batch size: 53, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:20:55,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230450
2023-11-21 14:20:56,086 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.25 vs. limit=22.5
2023-11-21 14:20:58,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1536313.3333333333, ans=0.125
2023-11-21 14:21:04,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1536313.3333333333, ans=0.125
2023-11-21 14:21:10,599 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.899e+01 7.989e+01 8.739e+01 9.353e+01 1.086e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 14:21:15,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1536380.0, ans=0.1
2023-11-21 14:21:18,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1536446.6666666667, ans=0.1
2023-11-21 14:21:34,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1536513.3333333333, ans=0.5
2023-11-21 14:21:39,286 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.64 vs. limit=22.5
2023-11-21 14:21:58,510 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2050, loss[loss=0.09247, simple_loss=0.1136, pruned_loss=0.02735, audio_tagging_loss=0.008326, over 15543.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09613, pruned_loss=0.01641, audio_tagging_loss=0.009318, over 3030929.71 frames. ], batch size: 58, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:21:59,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230500
2023-11-21 14:22:14,263 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.55 vs. limit=22.5
2023-11-21 14:22:21,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1536713.3333333333, ans=0.0
2023-11-21 14:22:29,636 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.11 vs. limit=15.0
2023-11-21 14:23:01,841 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2100, loss[loss=0.08884, simple_loss=0.1157, pruned_loss=0.02311, audio_tagging_loss=0.007897, over 15231.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09606, pruned_loss=0.01647, audio_tagging_loss=0.009354, over 3034979.78 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:23:03,176 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230550
2023-11-21 14:23:17,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1537046.6666666667, ans=0.1
2023-11-21 14:23:20,270 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.887e+01 8.079e+01 8.846e+01 9.402e+01 1.190e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 14:23:32,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1537113.3333333333, ans=0.125
2023-11-21 14:24:05,872 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2150, loss[loss=0.07796, simple_loss=0.09988, pruned_loss=0.01782, audio_tagging_loss=0.0102, over 13756.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09609, pruned_loss=0.01645, audio_tagging_loss=0.009337, over 3033425.62 frames. ], batch size: 52, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:24:07,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230600
2023-11-21 14:24:16,601 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.41 vs. limit=15.0
2023-11-21 14:24:45,332 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:25:01,812 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.08 vs. limit=12.0
2023-11-21 14:25:11,565 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2200, loss[loss=0.07457, simple_loss=0.09959, pruned_loss=0.0136, audio_tagging_loss=0.01117, over 15080.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09622, pruned_loss=0.01639, audio_tagging_loss=0.009325, over 3034814.60 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:25:12,933 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230650
2023-11-21 14:25:23,264 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.83 vs. limit=15.0
2023-11-21 14:25:28,621 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.037e+01 8.654e+01 9.441e+01 1.350e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 14:25:46,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1537780.0, ans=0.0
2023-11-21 14:25:47,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1537846.6666666667, ans=0.2
2023-11-21 14:25:51,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1537846.6666666667, ans=0.0
2023-11-21 14:25:57,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1537846.6666666667, ans=0.125
2023-11-21 14:26:00,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1537846.6666666667, ans=0.125
2023-11-21 14:26:08,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1537913.3333333333, ans=0.125
2023-11-21 14:26:14,308 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2250, loss[loss=0.08274, simple_loss=0.1, pruned_loss=0.02288, audio_tagging_loss=0.009843, over 15176.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09611, pruned_loss=0.01645, audio_tagging_loss=0.009444, over 3035931.62 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:26:15,607 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230700
2023-11-21 14:26:46,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1538113.3333333333, ans=0.1
2023-11-21 14:27:14,303 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.71 vs. limit=15.0
2023-11-21 14:27:17,393 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2300, loss[loss=0.05002, simple_loss=0.06664, pruned_loss=0.00743, audio_tagging_loss=0.009272, over 15284.00 frames. ], tot_loss[loss=0.07372, simple_loss=0.09575, pruned_loss=0.01627, audio_tagging_loss=0.00958, over 3036881.73 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:27:18,764 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230750
2023-11-21 14:27:21,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1538313.3333333333, ans=0.125
2023-11-21 14:27:22,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1538313.3333333333, ans=0.0
2023-11-21 14:27:32,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1538380.0, ans=0.95
2023-11-21 14:27:36,132 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.774e+01 7.885e+01 8.573e+01 9.171e+01 1.137e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 14:27:42,023 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.04 vs. limit=22.5
2023-11-21 14:28:02,131 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:28:13,479 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:28:16,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1538580.0, ans=0.0
2023-11-21 14:28:22,099 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2350, loss[loss=0.07504, simple_loss=0.08855, pruned_loss=0.02109, audio_tagging_loss=0.009678, over 14058.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09549, pruned_loss=0.01625, audio_tagging_loss=0.009598, over 3040075.12 frames. ], batch size: 53, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:28:23,391 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230800
2023-11-21 14:28:30,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1538646.6666666667, ans=0.1
2023-11-21 14:28:32,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1538646.6666666667, ans=0.0
2023-11-21 14:28:33,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1538713.3333333333, ans=0.0
2023-11-21 14:28:37,777 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.44 vs. limit=15.0
2023-11-21 14:28:44,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1538713.3333333333, ans=0.07
2023-11-21 14:28:54,229 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.08 vs. limit=15.0
2023-11-21 14:28:59,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1538846.6666666667, ans=0.0
2023-11-21 14:29:26,628 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2400, loss[loss=0.0873, simple_loss=0.124, pruned_loss=0.01918, audio_tagging_loss=0.006139, over 16158.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09469, pruned_loss=0.01622, audio_tagging_loss=0.009784, over 3045632.07 frames. ], batch size: 61, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:29:28,043 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230850
2023-11-21 14:29:43,570 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.100e+01 8.345e+01 8.985e+01 9.638e+01 1.227e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-21 14:29:48,551 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0
2023-11-21 14:29:50,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1539113.3333333333, ans=0.09899494936611666
2023-11-21 14:30:04,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1539180.0, ans=0.125
2023-11-21 14:30:22,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1539246.6666666667, ans=0.09899494936611666
2023-11-21 14:30:24,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1539246.6666666667, ans=0.125
2023-11-21 14:30:26,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1539246.6666666667, ans=0.125
2023-11-21 14:30:30,102 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2450, loss[loss=0.07141, simple_loss=0.08832, pruned_loss=0.01724, audio_tagging_loss=0.01, over 16168.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09537, pruned_loss=0.01651, audio_tagging_loss=0.009821, over 3045174.86 frames. ], batch size: 62, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:30:31,498 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230900
2023-11-21 14:30:37,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1539313.3333333333, ans=0.125
2023-11-21 14:30:46,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1539380.0, ans=0.125
2023-11-21 14:30:46,668 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.84 vs. limit=6.0
2023-11-21 14:30:56,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1539446.6666666667, ans=0.125
2023-11-21 14:30:59,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1539446.6666666667, ans=0.015
2023-11-21 14:31:09,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1539513.3333333333, ans=0.2
2023-11-21 14:31:31,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1539580.0, ans=0.2
2023-11-21 14:31:35,080 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2500, loss[loss=0.08926, simple_loss=0.1133, pruned_loss=0.02519, audio_tagging_loss=0.007421, over 15567.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.0963, pruned_loss=0.01677, audio_tagging_loss=0.009842, over 3046434.87 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:31:35,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1539646.6666666667, ans=0.125
2023-11-21 14:31:37,046 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 230950
2023-11-21 14:31:53,379 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.534e+01 7.942e+01 8.529e+01 9.362e+01 1.301e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-21 14:32:03,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1539780.0, ans=0.1
2023-11-21 14:32:17,616 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:32:21,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1539846.6666666667, ans=0.07
2023-11-21 14:32:24,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.07 vs. limit=10.0
2023-11-21 14:32:40,133 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2550, loss[loss=0.08499, simple_loss=0.1103, pruned_loss=0.01902, audio_tagging_loss=0.01082, over 15443.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09555, pruned_loss=0.01674, audio_tagging_loss=0.009728, over 3043979.05 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:32:41,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231000
2023-11-21 14:32:41,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1539980.0, ans=0.2
2023-11-21 14:32:46,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1539980.0, ans=0.0
2023-11-21 14:32:49,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1539980.0, ans=0.125
2023-11-21 14:32:52,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1540046.6666666667, ans=0.0
2023-11-21 14:32:56,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=1540046.6666666667, ans=10.0
2023-11-21 14:33:07,277 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=23.01 vs. limit=22.5
2023-11-21 14:33:16,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1540113.3333333333, ans=0.0
2023-11-21 14:33:18,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1540180.0, ans=0.125
2023-11-21 14:33:22,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1540180.0, ans=0.125
2023-11-21 14:33:29,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1540180.0, ans=0.125
2023-11-21 14:33:44,037 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2600, loss[loss=0.08726, simple_loss=0.1205, pruned_loss=0.01968, audio_tagging_loss=0.007312, over 14763.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09457, pruned_loss=0.01637, audio_tagging_loss=0.009604, over 3038895.55 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:33:45,318 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231050
2023-11-21 14:34:00,858 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.22 vs. limit=15.0
2023-11-21 14:34:03,318 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 7.933e+01 8.665e+01 9.563e+01 1.438e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 14:34:07,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1540380.0, ans=0.125
2023-11-21 14:34:10,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1540446.6666666667, ans=0.125
2023-11-21 14:34:27,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1540513.3333333333, ans=0.5
2023-11-21 14:34:43,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1540580.0, ans=0.0
2023-11-21 14:34:47,252 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2650, loss[loss=0.06747, simple_loss=0.0882, pruned_loss=0.01457, audio_tagging_loss=0.008798, over 13501.00 frames. ], tot_loss[loss=0.0743, simple_loss=0.0961, pruned_loss=0.01675, audio_tagging_loss=0.009494, over 3037202.53 frames. ], batch size: 55, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:34:48,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231100
2023-11-21 14:34:52,729 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.80 vs. limit=6.0
2023-11-21 14:35:19,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1540780.0, ans=0.125
2023-11-21 14:35:29,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1540846.6666666667, ans=0.0
2023-11-21 14:35:29,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1540846.6666666667, ans=0.1
2023-11-21 14:35:32,366 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:35:34,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1540846.6666666667, ans=0.1
2023-11-21 14:35:49,968 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.88 vs. limit=15.0
2023-11-21 14:35:51,743 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2700, loss[loss=0.09696, simple_loss=0.133, pruned_loss=0.02283, audio_tagging_loss=0.007655, over 14843.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09587, pruned_loss=0.01659, audio_tagging_loss=0.009448, over 3035278.84 frames. ], batch size: 52, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:35:53,044 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231150
2023-11-21 14:36:10,602 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.769e+01 8.073e+01 8.815e+01 9.662e+01 1.358e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 14:36:43,197 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.54 vs. limit=15.0
2023-11-21 14:36:55,377 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2750, loss[loss=0.06226, simple_loss=0.07915, pruned_loss=0.01184, audio_tagging_loss=0.01085, over 16191.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09615, pruned_loss=0.01659, audio_tagging_loss=0.00942, over 3036212.77 frames. ], batch size: 61, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:36:56,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231200
2023-11-21 14:37:04,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1541313.3333333333, ans=0.125
2023-11-21 14:37:06,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1541313.3333333333, ans=0.0
2023-11-21 14:37:06,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1541313.3333333333, ans=0.0
2023-11-21 14:37:06,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=1541313.3333333333, ans=15.0
2023-11-21 14:37:21,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1541446.6666666667, ans=0.1
2023-11-21 14:37:37,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1541513.3333333333, ans=0.125
2023-11-21 14:37:45,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1541580.0, ans=0.125
2023-11-21 14:37:50,357 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:37:58,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1541646.6666666667, ans=0.0
2023-11-21 14:37:59,538 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2800, loss[loss=0.06131, simple_loss=0.08023, pruned_loss=0.01156, audio_tagging_loss=0.009635, over 15711.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09541, pruned_loss=0.01626, audio_tagging_loss=0.009401, over 3035151.47 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:38:00,893 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231250
2023-11-21 14:38:05,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1541646.6666666667, ans=0.125
2023-11-21 14:38:07,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1541646.6666666667, ans=0.125
2023-11-21 14:38:12,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1541713.3333333333, ans=0.0
2023-11-21 14:38:19,390 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 8.133e+01 8.565e+01 9.239e+01 1.235e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 14:38:30,454 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.65 vs. limit=22.5
2023-11-21 14:38:54,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1541913.3333333333, ans=0.0
2023-11-21 14:39:04,744 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2850, loss[loss=0.06374, simple_loss=0.0781, pruned_loss=0.01382, audio_tagging_loss=0.01087, over 14878.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09568, pruned_loss=0.01637, audio_tagging_loss=0.009467, over 3040270.36 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:39:06,079 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231300
2023-11-21 14:39:24,005 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:39:28,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1542113.3333333333, ans=0.125
2023-11-21 14:39:35,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1542113.3333333333, ans=0.125
2023-11-21 14:39:56,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1542246.6666666667, ans=0.125
2023-11-21 14:40:08,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1542313.3333333333, ans=0.0
2023-11-21 14:40:09,152 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2900, loss[loss=0.07005, simple_loss=0.09112, pruned_loss=0.01619, audio_tagging_loss=0.008301, over 15283.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09563, pruned_loss=0.01631, audio_tagging_loss=0.009523, over 3038650.59 frames. ], batch size: 55, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:40:10,488 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231350
2023-11-21 14:40:29,512 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 8.134e+01 8.745e+01 9.473e+01 1.198e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 14:40:34,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1542446.6666666667, ans=0.95
2023-11-21 14:40:39,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1542446.6666666667, ans=0.05
2023-11-21 14:40:40,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1542446.6666666667, ans=0.0
2023-11-21 14:40:50,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1542513.3333333333, ans=0.0
2023-11-21 14:41:02,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1542580.0, ans=0.0
2023-11-21 14:41:06,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1542580.0, ans=0.2
2023-11-21 14:41:12,927 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 2950, loss[loss=0.06445, simple_loss=0.07624, pruned_loss=0.01449, audio_tagging_loss=0.01184, over 15890.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09591, pruned_loss=0.01649, audio_tagging_loss=0.009577, over 3039730.45 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:41:14,319 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231400
2023-11-21 14:41:18,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1542646.6666666667, ans=0.04949747468305833
2023-11-21 14:41:28,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1542713.3333333333, ans=0.125
2023-11-21 14:41:56,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1542846.6666666667, ans=0.125
2023-11-21 14:42:18,046 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3000, loss[loss=0.0597, simple_loss=0.08553, pruned_loss=0.01017, audio_tagging_loss=0.006768, over 15061.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.0949, pruned_loss=0.01626, audio_tagging_loss=0.009632, over 3033800.89 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:42:18,049 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 14:42:56,768 INFO [train_asr.py:1253] (0/4) Epoch 20, validation: loss=0.05942, simple_loss=0.05225, pruned_loss=0.00524, audio_tagging_loss=0.02805, over 4681554.00 frames. 
2023-11-21 14:42:56,769 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 14:42:57,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1542980.0, ans=0.125
2023-11-21 14:42:58,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231450
2023-11-21 14:43:17,879 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.247e+01 8.896e+01 9.640e+01 1.197e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 14:43:22,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1543113.3333333333, ans=0.1
2023-11-21 14:43:33,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=2.97 vs. limit=15.0
2023-11-21 14:43:54,212 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.22 vs. limit=15.0
2023-11-21 14:43:58,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.53 vs. limit=15.0
2023-11-21 14:44:00,887 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3050, loss[loss=0.09241, simple_loss=0.1185, pruned_loss=0.0227, audio_tagging_loss=0.01048, over 16156.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09611, pruned_loss=0.01657, audio_tagging_loss=0.009587, over 3038273.95 frames. ], batch size: 57, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:44:02,242 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231500
2023-11-21 14:44:06,011 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.53 vs. limit=12.0
2023-11-21 14:44:23,321 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:44:25,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1543446.6666666667, ans=0.1
2023-11-21 14:44:34,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1543446.6666666667, ans=0.1
2023-11-21 14:44:36,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1543446.6666666667, ans=0.125
2023-11-21 14:44:37,577 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:44:39,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1543513.3333333333, ans=0.0
2023-11-21 14:44:45,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1543513.3333333333, ans=0.2
2023-11-21 14:44:46,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=1543513.3333333333, ans=0.2
2023-11-21 14:44:48,689 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:44:58,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1543580.0, ans=0.125
2023-11-21 14:45:05,761 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3100, loss[loss=0.1136, simple_loss=0.1591, pruned_loss=0.02805, audio_tagging_loss=0.005996, over 16335.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09608, pruned_loss=0.01668, audio_tagging_loss=0.00964, over 3041973.32 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:45:07,043 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231550
2023-11-21 14:45:25,357 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.074e+01 8.626e+01 9.315e+01 1.250e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-21 14:45:31,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1543780.0, ans=0.125
2023-11-21 14:45:36,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1543780.0, ans=0.125
2023-11-21 14:45:37,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1543780.0, ans=0.125
2023-11-21 14:45:44,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1543846.6666666667, ans=0.125
2023-11-21 14:46:08,550 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3150, loss[loss=0.07897, simple_loss=0.1021, pruned_loss=0.01733, audio_tagging_loss=0.01058, over 15946.00 frames. ], tot_loss[loss=0.07457, simple_loss=0.0963, pruned_loss=0.01672, audio_tagging_loss=0.009694, over 3043707.50 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 16.0
2023-11-21 14:46:09,913 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231600
2023-11-21 14:46:14,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1543980.0, ans=0.1
2023-11-21 14:46:19,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1543980.0, ans=0.035
2023-11-21 14:46:34,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1544113.3333333333, ans=0.125
2023-11-21 14:46:34,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1544113.3333333333, ans=0.125
2023-11-21 14:46:35,422 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.17 vs. limit=6.0
2023-11-21 14:47:05,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1544246.6666666667, ans=0.0
2023-11-21 14:47:13,970 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3200, loss[loss=0.06529, simple_loss=0.08344, pruned_loss=0.0134, audio_tagging_loss=0.01017, over 15697.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09677, pruned_loss=0.01674, audio_tagging_loss=0.009787, over 3047691.25 frames. ], batch size: 59, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:47:14,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1544313.3333333333, ans=0.1
2023-11-21 14:47:15,334 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231650
2023-11-21 14:47:30,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1544380.0, ans=0.1
2023-11-21 14:47:34,893 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.766e+01 7.989e+01 8.860e+01 9.564e+01 1.510e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-21 14:47:45,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.55 vs. limit=15.0
2023-11-21 14:48:19,276 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3250, loss[loss=0.09169, simple_loss=0.1149, pruned_loss=0.0222, audio_tagging_loss=0.01202, over 14570.00 frames. ], tot_loss[loss=0.0745, simple_loss=0.09641, pruned_loss=0.01644, audio_tagging_loss=0.009861, over 3045814.28 frames. ], batch size: 56, lr: 3.45e-03, grad_scale: 32.0
2023-11-21 14:48:20,639 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231700
2023-11-21 14:49:00,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1544846.6666666667, ans=0.2
2023-11-21 14:49:08,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1544846.6666666667, ans=0.125
2023-11-21 14:49:22,565 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3300, loss[loss=0.06586, simple_loss=0.07612, pruned_loss=0.01533, audio_tagging_loss=0.01247, over 16371.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.09671, pruned_loss=0.01642, audio_tagging_loss=0.009951, over 3057950.18 frames. ], batch size: 65, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:49:23,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231750
2023-11-21 14:49:24,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1544980.0, ans=0.0
2023-11-21 14:49:26,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1544980.0, ans=0.1
2023-11-21 14:49:29,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1544980.0, ans=0.2
2023-11-21 14:49:39,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1545046.6666666667, ans=6.0
2023-11-21 14:49:43,229 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 8.162e+01 8.724e+01 9.354e+01 1.405e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 14:49:46,501 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.49 vs. limit=15.0
2023-11-21 14:49:59,477 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.76 vs. limit=12.0
2023-11-21 14:50:25,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1545313.3333333333, ans=0.0
2023-11-21 14:50:26,001 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3350, loss[loss=0.06967, simple_loss=0.0821, pruned_loss=0.0189, audio_tagging_loss=0.009718, over 15390.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09613, pruned_loss=0.01639, audio_tagging_loss=0.009908, over 3057601.94 frames. ], batch size: 59, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:50:27,357 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231800
2023-11-21 14:50:36,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1545313.3333333333, ans=0.125
2023-11-21 14:50:44,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1545380.0, ans=0.0
2023-11-21 14:51:05,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1545513.3333333333, ans=0.125
2023-11-21 14:51:14,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1545513.3333333333, ans=0.2
2023-11-21 14:51:24,009 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:51:24,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1545580.0, ans=0.025
2023-11-21 14:51:29,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1545646.6666666667, ans=0.2
2023-11-21 14:51:30,456 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3400, loss[loss=0.05778, simple_loss=0.08054, pruned_loss=0.008852, audio_tagging_loss=0.008663, over 15778.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09635, pruned_loss=0.01645, audio_tagging_loss=0.009743, over 3055744.67 frames. ], batch size: 58, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:51:31,716 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231850
2023-11-21 14:51:34,363 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:51:45,442 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.63 vs. limit=15.0
2023-11-21 14:51:49,682 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.184e+01 8.749e+01 9.223e+01 1.235e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 14:51:57,264 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:52:07,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1545846.6666666667, ans=0.125
2023-11-21 14:52:12,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1545846.6666666667, ans=0.125
2023-11-21 14:52:16,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1545846.6666666667, ans=0.0
2023-11-21 14:52:24,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1545913.3333333333, ans=0.125
2023-11-21 14:52:33,411 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3450, loss[loss=0.05493, simple_loss=0.07221, pruned_loss=0.00978, audio_tagging_loss=0.009045, over 17319.00 frames. ], tot_loss[loss=0.07375, simple_loss=0.09551, pruned_loss=0.01638, audio_tagging_loss=0.009608, over 3050713.59 frames. ], batch size: 66, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:52:34,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231900
2023-11-21 14:52:53,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1546046.6666666667, ans=0.125
2023-11-21 14:52:55,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1546046.6666666667, ans=0.125
2023-11-21 14:53:33,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1546246.6666666667, ans=0.125
2023-11-21 14:53:36,130 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.33 vs. limit=6.0
2023-11-21 14:53:36,604 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3500, loss[loss=0.05189, simple_loss=0.06315, pruned_loss=0.008978, audio_tagging_loss=0.01133, over 15675.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.09495, pruned_loss=0.01639, audio_tagging_loss=0.009657, over 3050137.41 frames. ], batch size: 59, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 14:53:38,519 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 231950
2023-11-21 14:53:41,711 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.54 vs. limit=15.0
2023-11-21 14:53:48,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1546380.0, ans=0.125
2023-11-21 14:53:59,557 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.228e+01 8.776e+01 9.674e+01 1.230e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 14:54:10,767 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:54:17,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1546513.3333333333, ans=0.125
2023-11-21 14:54:18,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1546513.3333333333, ans=0.2
2023-11-21 14:54:40,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1546646.6666666667, ans=0.125
2023-11-21 14:54:41,980 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3550, loss[loss=0.08405, simple_loss=0.1126, pruned_loss=0.02079, audio_tagging_loss=0.006986, over 14897.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.09536, pruned_loss=0.01655, audio_tagging_loss=0.009641, over 3046104.37 frames. ], batch size: 55, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 14:54:43,356 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232000
2023-11-21 14:54:45,417 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-232000.pt
2023-11-21 14:54:53,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1546646.6666666667, ans=0.125
2023-11-21 14:55:15,810 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.50 vs. limit=15.0
2023-11-21 14:55:16,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1546780.0, ans=0.2
2023-11-21 14:55:18,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1546780.0, ans=0.125
2023-11-21 14:55:48,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1546980.0, ans=0.125
2023-11-21 14:55:49,457 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3600, loss[loss=0.0792, simple_loss=0.1017, pruned_loss=0.01819, audio_tagging_loss=0.01016, over 15088.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09556, pruned_loss=0.01648, audio_tagging_loss=0.00961, over 3044694.71 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:55:50,734 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232050
2023-11-21 14:56:10,954 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 7.980e+01 8.692e+01 9.366e+01 1.171e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 14:56:11,288 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:56:47,978 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.97 vs. limit=15.0
2023-11-21 14:56:53,211 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3650, loss[loss=0.07172, simple_loss=0.09506, pruned_loss=0.01467, audio_tagging_loss=0.00951, over 15529.00 frames. ], tot_loss[loss=0.07393, simple_loss=0.0958, pruned_loss=0.01645, audio_tagging_loss=0.009573, over 3050441.90 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:56:54,519 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232100
2023-11-21 14:56:57,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=1547313.3333333333, ans=10.0
2023-11-21 14:57:24,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1547446.6666666667, ans=0.0
2023-11-21 14:57:42,082 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.72 vs. limit=10.0
2023-11-21 14:57:52,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.53 vs. limit=22.5
2023-11-21 14:57:55,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1547580.0, ans=0.0
2023-11-21 14:57:58,561 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3700, loss[loss=0.06931, simple_loss=0.09431, pruned_loss=0.0136, audio_tagging_loss=0.008561, over 15725.00 frames. ], tot_loss[loss=0.07425, simple_loss=0.09649, pruned_loss=0.01656, audio_tagging_loss=0.00944, over 3049620.60 frames. ], batch size: 59, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:57:59,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232150
2023-11-21 14:57:59,939 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 14:58:20,223 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.897e+01 8.102e+01 8.765e+01 9.489e+01 1.317e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-21 14:58:30,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1547780.0, ans=0.1
2023-11-21 14:58:35,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1547846.6666666667, ans=0.2
2023-11-21 14:59:00,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1547913.3333333333, ans=0.1
2023-11-21 14:59:02,972 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3750, loss[loss=0.05503, simple_loss=0.07007, pruned_loss=0.01013, audio_tagging_loss=0.009865, over 14944.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09675, pruned_loss=0.01653, audio_tagging_loss=0.009433, over 3046904.41 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 14:59:04,272 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232200
2023-11-21 14:59:34,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1548113.3333333333, ans=0.125
2023-11-21 14:59:45,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1548180.0, ans=0.0
2023-11-21 14:59:45,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1548180.0, ans=0.125
2023-11-21 14:59:46,990 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 14:59:52,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1548180.0, ans=0.09899494936611666
2023-11-21 15:00:06,902 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3800, loss[loss=0.07938, simple_loss=0.09502, pruned_loss=0.02106, audio_tagging_loss=0.01081, over 15398.00 frames. ], tot_loss[loss=0.07449, simple_loss=0.0968, pruned_loss=0.0166, audio_tagging_loss=0.009496, over 3049099.76 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:00:08,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232250
2023-11-21 15:00:28,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1548380.0, ans=0.0
2023-11-21 15:00:29,253 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.346e+01 8.333e+01 9.130e+01 9.759e+01 1.207e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-21 15:01:05,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1548580.0, ans=0.0
2023-11-21 15:01:11,478 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3850, loss[loss=0.06632, simple_loss=0.09154, pruned_loss=0.0119, audio_tagging_loss=0.008653, over 15670.00 frames. ], tot_loss[loss=0.0749, simple_loss=0.09724, pruned_loss=0.0167, audio_tagging_loss=0.009578, over 3047317.53 frames. ], batch size: 59, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:01:12,797 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232300
2023-11-21 15:01:24,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.84 vs. limit=10.0
2023-11-21 15:02:03,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1548913.3333333333, ans=0.125
2023-11-21 15:02:15,389 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3900, loss[loss=0.08641, simple_loss=0.1176, pruned_loss=0.01832, audio_tagging_loss=0.009288, over 15236.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09668, pruned_loss=0.01674, audio_tagging_loss=0.00968, over 3045347.50 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:02:16,694 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232350
2023-11-21 15:02:30,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1549046.6666666667, ans=0.1
2023-11-21 15:02:36,390 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.138e+01 8.613e+01 9.443e+01 1.181e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 15:02:51,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1549113.3333333333, ans=0.125
2023-11-21 15:02:56,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1549180.0, ans=0.1
2023-11-21 15:03:19,008 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 3950, loss[loss=0.07591, simple_loss=0.09992, pruned_loss=0.01428, audio_tagging_loss=0.01167, over 16058.00 frames. ], tot_loss[loss=0.07502, simple_loss=0.09692, pruned_loss=0.0168, audio_tagging_loss=0.009761, over 3050072.85 frames. ], batch size: 58, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:03:20,371 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232400
2023-11-21 15:03:22,543 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.35 vs. limit=22.5
2023-11-21 15:04:01,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1549513.3333333333, ans=0.125
2023-11-21 15:04:02,970 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.33 vs. limit=22.5
2023-11-21 15:04:03,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1549513.3333333333, ans=0.0
2023-11-21 15:04:23,709 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4000, loss[loss=0.1083, simple_loss=0.1364, pruned_loss=0.03242, audio_tagging_loss=0.007664, over 15663.00 frames. ], tot_loss[loss=0.07536, simple_loss=0.09729, pruned_loss=0.01691, audio_tagging_loss=0.009811, over 3043614.41 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:04:25,037 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232450
2023-11-21 15:04:26,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1549646.6666666667, ans=0.2
2023-11-21 15:04:33,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1549646.6666666667, ans=0.0
2023-11-21 15:04:45,498 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.715e+01 8.188e+01 8.912e+01 9.535e+01 1.152e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-21 15:05:10,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1549846.6666666667, ans=0.2
2023-11-21 15:05:28,173 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4050, loss[loss=0.08204, simple_loss=0.106, pruned_loss=0.01906, audio_tagging_loss=0.009967, over 15037.00 frames. ], tot_loss[loss=0.07563, simple_loss=0.09752, pruned_loss=0.01701, audio_tagging_loss=0.009853, over 3044253.52 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:05:28,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1549980.0, ans=0.1
2023-11-21 15:05:29,466 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232500
2023-11-21 15:05:30,629 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:05:56,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1550113.3333333333, ans=0.2
2023-11-21 15:06:01,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1550113.3333333333, ans=0.1
2023-11-21 15:06:06,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1550180.0, ans=0.1
2023-11-21 15:06:13,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1550180.0, ans=0.2
2023-11-21 15:06:15,302 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.44 vs. limit=12.0
2023-11-21 15:06:31,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1550313.3333333333, ans=0.125
2023-11-21 15:06:32,507 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4100, loss[loss=0.08045, simple_loss=0.1053, pruned_loss=0.01988, audio_tagging_loss=0.007916, over 15351.00 frames. ], tot_loss[loss=0.07519, simple_loss=0.09705, pruned_loss=0.01686, audio_tagging_loss=0.009809, over 3042727.95 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:06:33,867 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232550
2023-11-21 15:06:37,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1550313.3333333333, ans=0.125
2023-11-21 15:06:44,291 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.69 vs. limit=15.0
2023-11-21 15:06:55,662 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.215e+01 8.893e+01 9.542e+01 1.222e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-21 15:07:13,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1550513.3333333333, ans=0.0
2023-11-21 15:07:36,350 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4150, loss[loss=0.08307, simple_loss=0.1093, pruned_loss=0.02065, audio_tagging_loss=0.007748, over 14563.00 frames. ], tot_loss[loss=0.07426, simple_loss=0.09572, pruned_loss=0.01665, audio_tagging_loss=0.009747, over 3031572.36 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:07:37,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232600
2023-11-21 15:07:42,234 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0
2023-11-21 15:07:43,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1550646.6666666667, ans=6.0
2023-11-21 15:07:52,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1550713.3333333333, ans=0.125
2023-11-21 15:08:04,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1550780.0, ans=0.0
2023-11-21 15:08:18,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.99 vs. limit=15.0
2023-11-21 15:08:22,751 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:08:24,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1550846.6666666667, ans=0.1
2023-11-21 15:08:29,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1550913.3333333333, ans=0.1
2023-11-21 15:08:39,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1550913.3333333333, ans=0.125
2023-11-21 15:08:39,940 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.19 vs. limit=22.5
2023-11-21 15:08:41,876 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4200, loss[loss=0.0921, simple_loss=0.1202, pruned_loss=0.02528, audio_tagging_loss=0.006703, over 15580.00 frames. ], tot_loss[loss=0.07378, simple_loss=0.09514, pruned_loss=0.01659, audio_tagging_loss=0.009619, over 3035494.64 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:08:43,215 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232650
2023-11-21 15:09:03,662 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.544e+01 8.218e+01 8.944e+01 9.900e+01 1.172e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-21 15:09:07,186 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:09:27,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1551180.0, ans=0.125
2023-11-21 15:09:29,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1551180.0, ans=0.1
2023-11-21 15:09:45,348 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4250, loss[loss=0.09567, simple_loss=0.1281, pruned_loss=0.02586, audio_tagging_loss=0.005778, over 15344.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09498, pruned_loss=0.01646, audio_tagging_loss=0.009564, over 3044328.79 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:09:46,754 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232700
2023-11-21 15:10:07,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1551380.0, ans=0.0
2023-11-21 15:10:11,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1551446.6666666667, ans=0.125
2023-11-21 15:10:11,734 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.74 vs. limit=15.0
2023-11-21 15:10:13,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1551446.6666666667, ans=0.0
2023-11-21 15:10:49,937 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4300, loss[loss=0.04969, simple_loss=0.06045, pruned_loss=0.009605, audio_tagging_loss=0.009857, over 15057.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09612, pruned_loss=0.0167, audio_tagging_loss=0.009481, over 3045766.26 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:10:51,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232750
2023-11-21 15:10:55,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1551646.6666666667, ans=0.2
2023-11-21 15:11:10,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.34 vs. limit=15.0
2023-11-21 15:11:12,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1551713.3333333333, ans=0.1
2023-11-21 15:11:13,427 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.861e+01 8.351e+01 9.107e+01 1.007e+02 1.335e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-21 15:11:37,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1551846.6666666667, ans=0.04949747468305833
2023-11-21 15:11:38,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1551846.6666666667, ans=0.125
2023-11-21 15:11:54,997 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4350, loss[loss=0.07613, simple_loss=0.1056, pruned_loss=0.01689, audio_tagging_loss=0.00643, over 15680.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09589, pruned_loss=0.01672, audio_tagging_loss=0.009436, over 3048715.16 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:11:56,271 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232800
2023-11-21 15:11:58,398 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.55 vs. limit=12.0
2023-11-21 15:12:04,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.47 vs. limit=22.5
2023-11-21 15:12:07,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1552046.6666666667, ans=10.0
2023-11-21 15:12:13,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1552046.6666666667, ans=0.125
2023-11-21 15:12:20,194 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.29 vs. limit=15.0
2023-11-21 15:12:22,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1552113.3333333333, ans=0.1
2023-11-21 15:12:33,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.26 vs. limit=15.0
2023-11-21 15:12:48,183 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.60 vs. limit=10.0
2023-11-21 15:12:48,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1552246.6666666667, ans=0.125
2023-11-21 15:12:58,326 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4400, loss[loss=0.07798, simple_loss=0.1022, pruned_loss=0.01635, audio_tagging_loss=0.01053, over 15232.00 frames. ], tot_loss[loss=0.07448, simple_loss=0.09672, pruned_loss=0.01671, audio_tagging_loss=0.009414, over 3048147.38 frames. ], batch size: 57, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:12:59,774 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232850
2023-11-21 15:12:59,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=1552313.3333333333, ans=0.1
2023-11-21 15:13:02,552 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.16 vs. limit=15.0
2023-11-21 15:13:04,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1552313.3333333333, ans=0.07
2023-11-21 15:13:12,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1552380.0, ans=0.025
2023-11-21 15:13:20,842 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.302e+01 8.149e+01 8.654e+01 9.425e+01 1.207e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 15:13:47,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1552513.3333333333, ans=0.125
2023-11-21 15:13:55,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1552580.0, ans=0.0
2023-11-21 15:14:01,246 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4450, loss[loss=0.07757, simple_loss=0.1022, pruned_loss=0.01946, audio_tagging_loss=0.007022, over 15176.00 frames. ], tot_loss[loss=0.07474, simple_loss=0.09697, pruned_loss=0.01686, audio_tagging_loss=0.009393, over 3054698.81 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:14:02,526 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232900
2023-11-21 15:14:15,785 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.07 vs. limit=15.0
2023-11-21 15:14:17,841 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:14:33,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1552780.0, ans=0.04949747468305833
2023-11-21 15:14:34,872 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.90 vs. limit=22.5
2023-11-21 15:14:35,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1552780.0, ans=0.125
2023-11-21 15:14:50,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1552846.6666666667, ans=0.125
2023-11-21 15:14:54,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1552913.3333333333, ans=0.2
2023-11-21 15:14:56,938 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.88 vs. limit=10.0
2023-11-21 15:14:58,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1552913.3333333333, ans=0.0
2023-11-21 15:14:59,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1552913.3333333333, ans=15.0
2023-11-21 15:15:00,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1552913.3333333333, ans=0.0
2023-11-21 15:15:04,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1552913.3333333333, ans=0.125
2023-11-21 15:15:06,404 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4500, loss[loss=0.08022, simple_loss=0.1078, pruned_loss=0.01882, audio_tagging_loss=0.007503, over 14830.00 frames. ], tot_loss[loss=0.07453, simple_loss=0.0971, pruned_loss=0.01666, audio_tagging_loss=0.009321, over 3053340.83 frames. ], batch size: 55, lr: 3.44e-03, grad_scale: 32.0
2023-11-21 15:15:07,744 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 232950
2023-11-21 15:15:15,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1552980.0, ans=0.125
2023-11-21 15:15:17,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1552980.0, ans=15.0
2023-11-21 15:15:24,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1553046.6666666667, ans=0.1
2023-11-21 15:15:27,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1553046.6666666667, ans=0.125
2023-11-21 15:15:30,007 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.406e+01 8.048e+01 8.795e+01 9.628e+01 1.621e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 15:15:34,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1553113.3333333333, ans=0.1
2023-11-21 15:15:37,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1553113.3333333333, ans=0.2
2023-11-21 15:15:52,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1553180.0, ans=0.0
2023-11-21 15:15:53,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1553180.0, ans=0.0
2023-11-21 15:16:02,790 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.80 vs. limit=22.5
2023-11-21 15:16:06,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1553246.6666666667, ans=0.125
2023-11-21 15:16:10,805 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4550, loss[loss=0.08814, simple_loss=0.1176, pruned_loss=0.02319, audio_tagging_loss=0.00616, over 16373.00 frames. ], tot_loss[loss=0.07494, simple_loss=0.0977, pruned_loss=0.01669, audio_tagging_loss=0.009404, over 3049330.59 frames. ], batch size: 60, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:16:12,077 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233000
2023-11-21 15:16:21,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.99 vs. limit=12.0
2023-11-21 15:16:24,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1553380.0, ans=0.0
2023-11-21 15:16:34,152 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.56 vs. limit=22.5
2023-11-21 15:16:36,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1553446.6666666667, ans=0.125
2023-11-21 15:16:36,814 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.94 vs. limit=22.5
2023-11-21 15:16:38,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1553446.6666666667, ans=0.125
2023-11-21 15:16:48,105 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=12.0
2023-11-21 15:16:54,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1553513.3333333333, ans=0.2
2023-11-21 15:16:55,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1553513.3333333333, ans=0.125
2023-11-21 15:17:00,252 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:17:15,286 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4600, loss[loss=0.06105, simple_loss=0.07779, pruned_loss=0.01273, audio_tagging_loss=0.009429, over 15324.00 frames. ], tot_loss[loss=0.07535, simple_loss=0.098, pruned_loss=0.01694, audio_tagging_loss=0.00941, over 3053531.03 frames. ], batch size: 56, lr: 3.44e-03, grad_scale: 16.0
2023-11-21 15:17:16,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233050
2023-11-21 15:17:40,752 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.564e+01 8.142e+01 8.759e+01 9.508e+01 1.262e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 15:17:44,315 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.82 vs. limit=8.0
2023-11-21 15:17:54,116 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.53 vs. limit=15.0
2023-11-21 15:18:06,422 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.47 vs. limit=10.0
2023-11-21 15:18:21,100 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4650, loss[loss=0.06935, simple_loss=0.09253, pruned_loss=0.01401, audio_tagging_loss=0.009079, over 16141.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.0967, pruned_loss=0.01668, audio_tagging_loss=0.009625, over 3045458.81 frames. ], batch size: 59, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:18:23,000 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233100
2023-11-21 15:18:25,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1553980.0, ans=0.125
2023-11-21 15:18:39,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1554046.6666666667, ans=0.1
2023-11-21 15:19:05,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1554180.0, ans=0.0
2023-11-21 15:19:26,196 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4700, loss[loss=0.09084, simple_loss=0.1114, pruned_loss=0.02632, audio_tagging_loss=0.008811, over 14634.00 frames. ], tot_loss[loss=0.07498, simple_loss=0.09708, pruned_loss=0.01681, audio_tagging_loss=0.009636, over 3049618.28 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:19:27,484 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233150
2023-11-21 15:19:37,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1554380.0, ans=0.125
2023-11-21 15:19:44,023 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.90 vs. limit=15.0
2023-11-21 15:19:46,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1554380.0, ans=0.2
2023-11-21 15:19:47,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1554380.0, ans=0.125
2023-11-21 15:19:49,361 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.239e+01 8.978e+01 9.572e+01 1.151e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-21 15:20:18,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1554580.0, ans=0.125
2023-11-21 15:20:20,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1554580.0, ans=0.125
2023-11-21 15:20:20,698 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.51 vs. limit=15.0
2023-11-21 15:20:21,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1554580.0, ans=0.125
2023-11-21 15:20:29,869 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4750, loss[loss=0.07266, simple_loss=0.09407, pruned_loss=0.01366, audio_tagging_loss=0.01196, over 14675.00 frames. ], tot_loss[loss=0.07505, simple_loss=0.09717, pruned_loss=0.01676, audio_tagging_loss=0.009702, over 3049658.05 frames. ], batch size: 54, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:20:31,162 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233200
2023-11-21 15:20:36,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1554646.6666666667, ans=0.125
2023-11-21 15:20:44,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1554713.3333333333, ans=0.1
2023-11-21 15:20:51,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1554713.3333333333, ans=0.125
2023-11-21 15:20:56,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1554780.0, ans=0.125
2023-11-21 15:21:01,914 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.20 vs. limit=15.0
2023-11-21 15:21:02,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1554780.0, ans=0.2
2023-11-21 15:21:05,894 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.52 vs. limit=6.0
2023-11-21 15:21:31,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1554913.3333333333, ans=0.125
2023-11-21 15:21:34,022 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4800, loss[loss=0.08513, simple_loss=0.1107, pruned_loss=0.0198, audio_tagging_loss=0.009987, over 14934.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09675, pruned_loss=0.01667, audio_tagging_loss=0.00986, over 3050132.94 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:21:35,979 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233250
2023-11-21 15:21:54,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1555046.6666666667, ans=0.0
2023-11-21 15:21:59,669 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.057e+01 8.234e+01 8.749e+01 9.624e+01 1.145e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 15:22:02,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1555113.3333333333, ans=0.0
2023-11-21 15:22:06,337 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.28 vs. limit=22.5
2023-11-21 15:22:08,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1555113.3333333333, ans=0.125
2023-11-21 15:22:10,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1555113.3333333333, ans=0.1
2023-11-21 15:22:12,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1555180.0, ans=0.1
2023-11-21 15:22:14,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1555180.0, ans=0.0
2023-11-21 15:22:40,340 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4850, loss[loss=0.06376, simple_loss=0.08335, pruned_loss=0.01351, audio_tagging_loss=0.008581, over 15070.00 frames. ], tot_loss[loss=0.07515, simple_loss=0.09716, pruned_loss=0.0167, audio_tagging_loss=0.009866, over 3046114.71 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:22:41,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233300
2023-11-21 15:23:18,749 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.64 vs. limit=15.0
2023-11-21 15:23:21,374 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.95 vs. limit=22.5
2023-11-21 15:23:22,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1555513.3333333333, ans=0.0
2023-11-21 15:23:31,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1555580.0, ans=0.0
2023-11-21 15:23:42,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1555646.6666666667, ans=0.125
2023-11-21 15:23:43,869 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4900, loss[loss=0.07285, simple_loss=0.1037, pruned_loss=0.01384, audio_tagging_loss=0.007145, over 15858.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09742, pruned_loss=0.01679, audio_tagging_loss=0.009745, over 3039877.80 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:23:45,296 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233350
2023-11-21 15:23:50,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1555646.6666666667, ans=0.125
2023-11-21 15:24:04,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1555713.3333333333, ans=0.2
2023-11-21 15:24:08,607 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.805e+01 8.037e+01 8.614e+01 9.424e+01 1.262e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 15:24:10,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1555780.0, ans=0.125
2023-11-21 15:24:32,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1555846.6666666667, ans=0.1
2023-11-21 15:24:47,990 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 4950, loss[loss=0.07292, simple_loss=0.0879, pruned_loss=0.01949, audio_tagging_loss=0.00948, over 15001.00 frames. ], tot_loss[loss=0.07456, simple_loss=0.09655, pruned_loss=0.01665, audio_tagging_loss=0.009642, over 3040560.46 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:24:49,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233400
2023-11-21 15:24:51,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1555980.0, ans=0.0
2023-11-21 15:24:56,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1555980.0, ans=0.125
2023-11-21 15:24:58,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1555980.0, ans=0.0
2023-11-21 15:24:58,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=7.69 vs. limit=10.0
2023-11-21 15:25:04,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1556046.6666666667, ans=0.1
2023-11-21 15:25:18,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1556113.3333333333, ans=0.1
2023-11-21 15:25:35,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1556180.0, ans=0.0
2023-11-21 15:25:53,388 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5000, loss[loss=0.06164, simple_loss=0.07838, pruned_loss=0.01391, audio_tagging_loss=0.008541, over 15806.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.0962, pruned_loss=0.01645, audio_tagging_loss=0.009508, over 3040862.71 frames. ], batch size: 61, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:25:54,620 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233450
2023-11-21 15:26:05,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1556380.0, ans=10.0
2023-11-21 15:26:17,063 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.408e+01 8.018e+01 8.729e+01 9.400e+01 1.250e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 15:26:37,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.25 vs. limit=15.0
2023-11-21 15:26:46,118 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:26:47,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1556580.0, ans=0.125
2023-11-21 15:26:57,623 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5050, loss[loss=0.07606, simple_loss=0.1027, pruned_loss=0.0158, audio_tagging_loss=0.00889, over 16339.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.0955, pruned_loss=0.01641, audio_tagging_loss=0.009434, over 3038259.01 frames. ], batch size: 60, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:26:57,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1556646.6666666667, ans=10.0
2023-11-21 15:26:58,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233500
2023-11-21 15:27:10,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1556713.3333333333, ans=0.0
2023-11-21 15:27:10,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1556713.3333333333, ans=0.125
2023-11-21 15:27:23,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1556780.0, ans=0.0
2023-11-21 15:27:29,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1556780.0, ans=0.0
2023-11-21 15:27:36,800 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:27:58,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1556913.3333333333, ans=0.125
2023-11-21 15:27:59,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1556913.3333333333, ans=0.125
2023-11-21 15:28:01,660 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5100, loss[loss=0.07923, simple_loss=0.09565, pruned_loss=0.02161, audio_tagging_loss=0.009794, over 14858.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09532, pruned_loss=0.01644, audio_tagging_loss=0.009465, over 3037741.08 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:28:02,903 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233550
2023-11-21 15:28:11,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1556980.0, ans=0.1
2023-11-21 15:28:18,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1557046.6666666667, ans=0.0
2023-11-21 15:28:24,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1557046.6666666667, ans=0.07
2023-11-21 15:28:26,719 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 7.860e+01 8.610e+01 9.200e+01 1.229e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-21 15:28:39,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1557180.0, ans=0.125
2023-11-21 15:28:48,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1557180.0, ans=0.0
2023-11-21 15:29:02,303 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.26 vs. limit=12.0
2023-11-21 15:29:06,979 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5150, loss[loss=0.06088, simple_loss=0.07645, pruned_loss=0.01523, audio_tagging_loss=0.007422, over 14809.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09549, pruned_loss=0.01648, audio_tagging_loss=0.009413, over 3037108.54 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:29:08,279 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233600
2023-11-21 15:29:32,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1557446.6666666667, ans=0.0
2023-11-21 15:29:36,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1557446.6666666667, ans=0.125
2023-11-21 15:29:56,507 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:30:03,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1557580.0, ans=0.2
2023-11-21 15:30:11,331 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5200, loss[loss=0.07156, simple_loss=0.09688, pruned_loss=0.01396, audio_tagging_loss=0.009162, over 14096.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09568, pruned_loss=0.01643, audio_tagging_loss=0.009379, over 3040044.35 frames. ], batch size: 53, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:30:12,736 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233650
2023-11-21 15:30:23,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1557713.3333333333, ans=0.1
2023-11-21 15:30:30,593 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:30:35,766 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.165e+01 8.844e+01 9.636e+01 1.346e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 15:30:40,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1557780.0, ans=0.0
2023-11-21 15:30:41,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1557780.0, ans=0.125
2023-11-21 15:30:42,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.83 vs. limit=15.0
2023-11-21 15:30:51,252 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.33 vs. limit=15.0
2023-11-21 15:30:55,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1557846.6666666667, ans=0.0
2023-11-21 15:30:58,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1557846.6666666667, ans=0.125
2023-11-21 15:31:00,890 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.85 vs. limit=15.0
2023-11-21 15:31:14,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1557980.0, ans=0.1
2023-11-21 15:31:15,925 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5250, loss[loss=0.06871, simple_loss=0.08819, pruned_loss=0.01286, audio_tagging_loss=0.01175, over 15626.00 frames. ], tot_loss[loss=0.07467, simple_loss=0.09739, pruned_loss=0.01668, audio_tagging_loss=0.0093, over 3041682.46 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:31:17,264 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233700
2023-11-21 15:31:27,250 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:31:29,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1558046.6666666667, ans=0.125
2023-11-21 15:31:40,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1558046.6666666667, ans=0.0
2023-11-21 15:31:48,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1558113.3333333333, ans=0.2
2023-11-21 15:31:49,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1558113.3333333333, ans=0.05
2023-11-21 15:31:55,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1558180.0, ans=0.0
2023-11-21 15:32:11,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1558246.6666666667, ans=10.0
2023-11-21 15:32:13,643 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.44 vs. limit=15.0
2023-11-21 15:32:13,858 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.18 vs. limit=22.5
2023-11-21 15:32:21,264 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5300, loss[loss=0.07122, simple_loss=0.09042, pruned_loss=0.01876, audio_tagging_loss=0.007241, over 14207.00 frames. ], tot_loss[loss=0.07555, simple_loss=0.09865, pruned_loss=0.01698, audio_tagging_loss=0.009246, over 3039722.21 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:32:22,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233750
2023-11-21 15:32:45,208 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.789e+01 8.165e+01 8.716e+01 9.457e+01 1.108e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 15:32:53,779 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.49 vs. limit=15.0
2023-11-21 15:32:58,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1558513.3333333333, ans=0.0
2023-11-21 15:33:01,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.71 vs. limit=22.5
2023-11-21 15:33:02,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1558513.3333333333, ans=0.125
2023-11-21 15:33:25,901 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5350, loss[loss=0.0961, simple_loss=0.1347, pruned_loss=0.02178, audio_tagging_loss=0.006946, over 15978.00 frames. ], tot_loss[loss=0.07548, simple_loss=0.09856, pruned_loss=0.01687, audio_tagging_loss=0.00933, over 3043853.11 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:33:27,194 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233800
2023-11-21 15:33:46,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1558713.3333333333, ans=0.0
2023-11-21 15:34:30,876 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5400, loss[loss=0.0917, simple_loss=0.1229, pruned_loss=0.02012, audio_tagging_loss=0.01014, over 15532.00 frames. ], tot_loss[loss=0.07577, simple_loss=0.09882, pruned_loss=0.017, audio_tagging_loss=0.009354, over 3045039.82 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:34:31,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1558980.0, ans=0.0
2023-11-21 15:34:32,240 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233850
2023-11-21 15:34:39,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1558980.0, ans=0.0
2023-11-21 15:34:41,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1558980.0, ans=0.2
2023-11-21 15:34:47,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1559046.6666666667, ans=0.0
2023-11-21 15:34:57,300 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.221e+01 8.112e+01 8.686e+01 9.399e+01 1.125e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 15:34:59,182 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.73 vs. limit=15.0
2023-11-21 15:35:02,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1559113.3333333333, ans=0.2
2023-11-21 15:35:16,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1559180.0, ans=0.0
2023-11-21 15:35:25,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1559246.6666666667, ans=0.0
2023-11-21 15:35:32,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1559246.6666666667, ans=0.125
2023-11-21 15:35:32,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1559246.6666666667, ans=0.1
2023-11-21 15:35:34,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1559246.6666666667, ans=0.07
2023-11-21 15:35:36,469 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5450, loss[loss=0.06712, simple_loss=0.08281, pruned_loss=0.01579, audio_tagging_loss=0.009927, over 16157.00 frames. ], tot_loss[loss=0.07592, simple_loss=0.0993, pruned_loss=0.01701, audio_tagging_loss=0.009262, over 3056852.43 frames. ], batch size: 60, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:35:37,770 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233900
2023-11-21 15:35:44,066 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.45 vs. limit=15.0
2023-11-21 15:35:50,105 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.18 vs. limit=12.0
2023-11-21 15:35:50,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1559380.0, ans=0.125
2023-11-21 15:36:40,853 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5500, loss[loss=0.08155, simple_loss=0.1062, pruned_loss=0.01989, audio_tagging_loss=0.008589, over 15449.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.09861, pruned_loss=0.01683, audio_tagging_loss=0.009396, over 3049304.66 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:36:42,172 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 233950
2023-11-21 15:36:45,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1559646.6666666667, ans=0.0
2023-11-21 15:36:47,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1559646.6666666667, ans=0.125
2023-11-21 15:36:48,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1559646.6666666667, ans=0.125
2023-11-21 15:37:05,835 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.151e+01 8.405e+01 9.190e+01 9.926e+01 1.565e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-21 15:37:09,602 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.38 vs. limit=10.0
2023-11-21 15:37:17,884 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.47 vs. limit=15.0
2023-11-21 15:37:26,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1559846.6666666667, ans=0.0
2023-11-21 15:37:39,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=1559913.3333333333, ans=0.025
2023-11-21 15:37:41,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1559913.3333333333, ans=0.2
2023-11-21 15:37:44,553 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5550, loss[loss=0.0819, simple_loss=0.113, pruned_loss=0.01545, audio_tagging_loss=0.009956, over 15450.00 frames. ], tot_loss[loss=0.07523, simple_loss=0.09795, pruned_loss=0.01674, audio_tagging_loss=0.009515, over 3047456.58 frames. ], batch size: 56, lr: 3.43e-03, grad_scale: 16.0
2023-11-21 15:37:45,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234000
2023-11-21 15:37:50,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1559980.0, ans=0.125
2023-11-21 15:37:53,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1559980.0, ans=0.125
2023-11-21 15:37:56,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.02 vs. limit=12.0
2023-11-21 15:37:58,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1560046.6666666667, ans=0.0
2023-11-21 15:38:33,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1560180.0, ans=0.125
2023-11-21 15:38:48,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1560246.6666666667, ans=0.1
2023-11-21 15:38:51,272 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5600, loss[loss=0.05625, simple_loss=0.06863, pruned_loss=0.01307, audio_tagging_loss=0.008863, over 15618.00 frames. ], tot_loss[loss=0.07509, simple_loss=0.09761, pruned_loss=0.01668, audio_tagging_loss=0.009609, over 3053780.35 frames. ], batch size: 61, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:38:52,694 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234050
2023-11-21 15:39:14,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1560380.0, ans=0.0
2023-11-21 15:39:16,694 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.332e+01 7.943e+01 8.570e+01 9.599e+01 1.261e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 15:39:20,734 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=4.679e-02
2023-11-21 15:39:26,029 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.06 vs. limit=22.5
2023-11-21 15:39:26,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1560446.6666666667, ans=0.1
2023-11-21 15:39:36,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1560513.3333333333, ans=0.1
2023-11-21 15:39:37,655 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:39:56,436 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5650, loss[loss=0.05708, simple_loss=0.06876, pruned_loss=0.01199, audio_tagging_loss=0.0107, over 15646.00 frames. ], tot_loss[loss=0.07445, simple_loss=0.09657, pruned_loss=0.01646, audio_tagging_loss=0.009699, over 3057161.35 frames. ], batch size: 60, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:39:57,794 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234100
2023-11-21 15:40:09,131 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:40:13,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1560713.3333333333, ans=0.1
2023-11-21 15:40:34,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1560846.6666666667, ans=0.0
2023-11-21 15:40:40,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=1560846.6666666667, ans=6.0
2023-11-21 15:40:45,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1560846.6666666667, ans=0.125
2023-11-21 15:40:57,460 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:41:00,855 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5700, loss[loss=0.06402, simple_loss=0.07474, pruned_loss=0.01486, audio_tagging_loss=0.01178, over 14449.00 frames. ], tot_loss[loss=0.07437, simple_loss=0.09639, pruned_loss=0.01642, audio_tagging_loss=0.009745, over 3053648.55 frames. ], batch size: 57, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:41:02,208 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234150
2023-11-21 15:41:03,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1560980.0, ans=0.0
2023-11-21 15:41:03,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1560980.0, ans=0.125
2023-11-21 15:41:07,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1560980.0, ans=0.2
2023-11-21 15:41:15,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1561046.6666666667, ans=0.05
2023-11-21 15:41:22,927 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.10 vs. limit=10.0
2023-11-21 15:41:27,212 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.266e+01 8.051e+01 8.513e+01 9.127e+01 1.242e+02, threshold=1.703e+02, percent-clipped=0.0
2023-11-21 15:41:33,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1561113.3333333333, ans=0.0
2023-11-21 15:41:39,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1561180.0, ans=0.025
2023-11-21 15:41:46,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1561180.0, ans=0.125
2023-11-21 15:42:05,392 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5750, loss[loss=0.07331, simple_loss=0.09439, pruned_loss=0.01751, audio_tagging_loss=0.008608, over 15997.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09544, pruned_loss=0.01633, audio_tagging_loss=0.009657, over 3051544.19 frames. ], batch size: 62, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:42:06,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234200
2023-11-21 15:42:14,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1561313.3333333333, ans=0.125
2023-11-21 15:42:38,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1561446.6666666667, ans=0.125
2023-11-21 15:42:45,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1561513.3333333333, ans=0.04949747468305833
2023-11-21 15:43:11,844 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5800, loss[loss=0.06062, simple_loss=0.07736, pruned_loss=0.01113, audio_tagging_loss=0.01081, over 15940.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09595, pruned_loss=0.01651, audio_tagging_loss=0.00949, over 3051849.16 frames. ], batch size: 60, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:43:12,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1561646.6666666667, ans=0.0
2023-11-21 15:43:13,147 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234250
2023-11-21 15:43:15,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1561646.6666666667, ans=0.1
2023-11-21 15:43:21,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.43 vs. limit=8.0
2023-11-21 15:43:36,361 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.259e+01 8.121e+01 8.819e+01 9.541e+01 1.299e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 15:43:42,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.72 vs. limit=15.0
2023-11-21 15:43:47,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1561780.0, ans=0.125
2023-11-21 15:44:01,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1561846.6666666667, ans=0.125
2023-11-21 15:44:16,110 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5850, loss[loss=0.0708, simple_loss=0.08997, pruned_loss=0.01491, audio_tagging_loss=0.0109, over 13961.00 frames. ], tot_loss[loss=0.07411, simple_loss=0.09636, pruned_loss=0.01651, audio_tagging_loss=0.009417, over 3052229.17 frames. ], batch size: 54, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:44:17,464 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234300
2023-11-21 15:44:17,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1561980.0, ans=0.0
2023-11-21 15:44:33,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1562046.6666666667, ans=0.0
2023-11-21 15:44:35,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1562046.6666666667, ans=0.125
2023-11-21 15:44:50,017 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.63 vs. limit=15.0
2023-11-21 15:44:58,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1562180.0, ans=10.0
2023-11-21 15:45:18,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1562246.6666666667, ans=0.125
2023-11-21 15:45:20,800 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5900, loss[loss=0.06519, simple_loss=0.08244, pruned_loss=0.01347, audio_tagging_loss=0.0105, over 15356.00 frames. ], tot_loss[loss=0.07422, simple_loss=0.09707, pruned_loss=0.01642, audio_tagging_loss=0.009267, over 3050778.18 frames. ], batch size: 58, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:45:22,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234350
2023-11-21 15:45:26,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1562313.3333333333, ans=0.125
2023-11-21 15:45:47,511 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.953e+01 8.033e+01 8.577e+01 9.444e+01 1.216e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 15:46:26,930 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 5950, loss[loss=0.06767, simple_loss=0.08534, pruned_loss=0.01453, audio_tagging_loss=0.01047, over 14508.00 frames. ], tot_loss[loss=0.07418, simple_loss=0.09689, pruned_loss=0.01643, audio_tagging_loss=0.009304, over 3050501.97 frames. ], batch size: 55, lr: 3.43e-03, grad_scale: 32.0
2023-11-21 15:46:28,269 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234400
2023-11-21 15:46:28,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1562646.6666666667, ans=0.1
2023-11-21 15:46:45,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1562713.3333333333, ans=0.0
2023-11-21 15:46:49,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.22 vs. limit=15.0
2023-11-21 15:46:58,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1562780.0, ans=0.125
2023-11-21 15:47:31,480 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6000, loss[loss=0.06967, simple_loss=0.09656, pruned_loss=0.01212, audio_tagging_loss=0.009271, over 14465.00 frames. ], tot_loss[loss=0.07368, simple_loss=0.09595, pruned_loss=0.01629, audio_tagging_loss=0.009406, over 3051515.83 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:47:31,482 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 15:48:12,902 INFO [train_asr.py:1253] (0/4) Epoch 20, validation: loss=0.06068, simple_loss=0.0522, pruned_loss=0.005214, audio_tagging_loss=0.02937, over 4681554.00 frames. 
2023-11-21 15:48:12,903 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 15:48:14,244 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234450
2023-11-21 15:48:14,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1562980.0, ans=0.04949747468305833
2023-11-21 15:48:35,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1563046.6666666667, ans=0.0
2023-11-21 15:48:38,430 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 7.928e+01 8.696e+01 9.354e+01 1.094e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 15:48:43,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1563113.3333333333, ans=0.09899494936611666
2023-11-21 15:48:58,826 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 15:49:16,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1563313.3333333333, ans=0.2
2023-11-21 15:49:17,910 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6050, loss[loss=0.07383, simple_loss=0.101, pruned_loss=0.01335, audio_tagging_loss=0.01001, over 15275.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09607, pruned_loss=0.01657, audio_tagging_loss=0.00948, over 3052691.01 frames. ], batch size: 58, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:49:19,207 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234500
2023-11-21 15:49:29,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1563380.0, ans=0.2
2023-11-21 15:50:21,448 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6100, loss[loss=0.06671, simple_loss=0.0904, pruned_loss=0.0145, audio_tagging_loss=0.007012, over 14677.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09586, pruned_loss=0.0164, audio_tagging_loss=0.009501, over 3055588.84 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:50:22,754 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234550
2023-11-21 15:50:35,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1563713.3333333333, ans=0.2
2023-11-21 15:50:47,222 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.060e+01 8.562e+01 9.373e+01 1.321e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 15:50:47,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1563780.0, ans=0.125
2023-11-21 15:50:47,802 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.91 vs. limit=15.0
2023-11-21 15:51:08,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1563846.6666666667, ans=0.07
2023-11-21 15:51:09,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1563846.6666666667, ans=0.125
2023-11-21 15:51:11,412 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.37 vs. limit=22.5
2023-11-21 15:51:14,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1563913.3333333333, ans=0.1
2023-11-21 15:51:21,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1563913.3333333333, ans=0.125
2023-11-21 15:51:25,609 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6150, loss[loss=0.08651, simple_loss=0.1102, pruned_loss=0.01805, audio_tagging_loss=0.01336, over 15516.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09487, pruned_loss=0.01634, audio_tagging_loss=0.009598, over 3044613.36 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:51:27,904 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234600
2023-11-21 15:51:27,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1563980.0, ans=0.0
2023-11-21 15:51:38,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1564046.6666666667, ans=0.2
2023-11-21 15:51:54,746 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.91 vs. limit=12.0
2023-11-21 15:52:04,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1564180.0, ans=0.125
2023-11-21 15:52:13,227 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.63 vs. limit=10.0
2023-11-21 15:52:31,544 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6200, loss[loss=0.06151, simple_loss=0.08541, pruned_loss=0.009751, audio_tagging_loss=0.009055, over 15566.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09474, pruned_loss=0.01635, audio_tagging_loss=0.009632, over 3045621.34 frames. ], batch size: 59, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:52:32,794 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234650
2023-11-21 15:52:47,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-21 15:52:49,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1564380.0, ans=0.0
2023-11-21 15:52:56,120 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 7.947e+01 8.423e+01 9.173e+01 1.216e+02, threshold=1.685e+02, percent-clipped=0.0
2023-11-21 15:53:30,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1564580.0, ans=0.0
2023-11-21 15:53:35,403 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6250, loss[loss=0.06755, simple_loss=0.08609, pruned_loss=0.01214, audio_tagging_loss=0.01237, over 14572.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09575, pruned_loss=0.01652, audio_tagging_loss=0.009732, over 3048051.40 frames. ], batch size: 54, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:53:36,007 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.26 vs. limit=15.0
2023-11-21 15:53:36,771 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234700
2023-11-21 15:53:44,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1564646.6666666667, ans=0.0
2023-11-21 15:53:44,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1564646.6666666667, ans=0.125
2023-11-21 15:54:01,914 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.02 vs. limit=12.0
2023-11-21 15:54:03,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1564780.0, ans=0.125
2023-11-21 15:54:24,826 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.19 vs. limit=10.0
2023-11-21 15:54:35,938 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.06 vs. limit=15.0
2023-11-21 15:54:39,041 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6300, loss[loss=0.07943, simple_loss=0.1091, pruned_loss=0.017, audio_tagging_loss=0.007865, over 15813.00 frames. ], tot_loss[loss=0.07443, simple_loss=0.09632, pruned_loss=0.01652, audio_tagging_loss=0.009741, over 3052332.47 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:54:40,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234750
2023-11-21 15:54:40,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1564980.0, ans=0.1
2023-11-21 15:55:02,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1565046.6666666667, ans=0.0
2023-11-21 15:55:05,690 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.628e+01 8.110e+01 8.746e+01 9.579e+01 1.135e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 15:55:20,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1565180.0, ans=0.125
2023-11-21 15:55:36,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1565246.6666666667, ans=0.0
2023-11-21 15:55:36,343 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 15:55:45,081 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6350, loss[loss=0.08291, simple_loss=0.1024, pruned_loss=0.0183, audio_tagging_loss=0.01342, over 14668.00 frames. ], tot_loss[loss=0.0741, simple_loss=0.09594, pruned_loss=0.01635, audio_tagging_loss=0.009789, over 3051197.79 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:55:46,396 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234800
2023-11-21 15:55:50,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1565313.3333333333, ans=0.0
2023-11-21 15:55:58,431 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.61 vs. limit=15.0
2023-11-21 15:56:32,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1565513.3333333333, ans=0.0
2023-11-21 15:56:35,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1565513.3333333333, ans=0.0
2023-11-21 15:56:49,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1565646.6666666667, ans=0.0
2023-11-21 15:56:50,000 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6400, loss[loss=0.07848, simple_loss=0.108, pruned_loss=0.01501, audio_tagging_loss=0.009491, over 15354.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09562, pruned_loss=0.01627, audio_tagging_loss=0.009823, over 3053476.29 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:56:51,317 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234850
2023-11-21 15:57:13,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1565713.3333333333, ans=0.0
2023-11-21 15:57:16,950 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.183e+01 8.791e+01 9.502e+01 1.187e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 15:57:36,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1565846.6666666667, ans=0.0
2023-11-21 15:57:37,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1565846.6666666667, ans=0.0
2023-11-21 15:57:41,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1565913.3333333333, ans=0.2
2023-11-21 15:57:50,076 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.38 vs. limit=5.0
2023-11-21 15:57:54,416 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=4.160e-02
2023-11-21 15:57:55,256 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6450, loss[loss=0.06007, simple_loss=0.06721, pruned_loss=0.01243, audio_tagging_loss=0.01403, over 14598.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09535, pruned_loss=0.01633, audio_tagging_loss=0.009899, over 3047188.29 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:57:56,603 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234900
2023-11-21 15:58:18,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1566046.6666666667, ans=0.0
2023-11-21 15:58:23,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1566113.3333333333, ans=0.125
2023-11-21 15:58:59,478 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.37 vs. limit=6.0
2023-11-21 15:59:01,489 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6500, loss[loss=0.06311, simple_loss=0.07923, pruned_loss=0.0131, audio_tagging_loss=0.0104, over 15332.00 frames. ], tot_loss[loss=0.07351, simple_loss=0.09476, pruned_loss=0.01621, audio_tagging_loss=0.009925, over 3041840.77 frames. ], batch size: 58, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 15:59:02,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 234950
2023-11-21 15:59:19,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1566380.0, ans=0.125
2023-11-21 15:59:26,837 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.005e+01 8.613e+01 9.254e+01 1.234e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 16:00:06,207 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6550, loss[loss=0.05328, simple_loss=0.0733, pruned_loss=0.009483, audio_tagging_loss=0.007143, over 14349.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09616, pruned_loss=0.01642, audio_tagging_loss=0.009672, over 3042728.42 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:00:07,640 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235000
2023-11-21 16:00:10,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1566646.6666666667, ans=0.0
2023-11-21 16:00:14,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.63 vs. limit=22.5
2023-11-21 16:00:21,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1566713.3333333333, ans=0.125
2023-11-21 16:00:21,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1566713.3333333333, ans=0.125
2023-11-21 16:00:25,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1566713.3333333333, ans=0.5
2023-11-21 16:00:30,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.17 vs. limit=15.0
2023-11-21 16:00:44,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1566780.0, ans=0.1
2023-11-21 16:00:47,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1566846.6666666667, ans=0.125
2023-11-21 16:00:48,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1566846.6666666667, ans=0.0
2023-11-21 16:00:56,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1566846.6666666667, ans=0.07
2023-11-21 16:01:02,106 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=15.40 vs. limit=15.0
2023-11-21 16:01:11,120 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6600, loss[loss=0.06121, simple_loss=0.08088, pruned_loss=0.01592, audio_tagging_loss=0.004848, over 15511.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.09558, pruned_loss=0.01638, audio_tagging_loss=0.009495, over 3041977.12 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:01:13,040 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235050
2023-11-21 16:01:39,699 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.039e+01 8.543e+01 9.389e+01 1.376e+02, threshold=1.709e+02, percent-clipped=0.0
2023-11-21 16:01:52,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1567180.0, ans=0.09899494936611666
2023-11-21 16:02:02,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1567246.6666666667, ans=0.125
2023-11-21 16:02:05,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1567246.6666666667, ans=0.125
2023-11-21 16:02:17,518 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6650, loss[loss=0.07085, simple_loss=0.09177, pruned_loss=0.01502, audio_tagging_loss=0.009948, over 14793.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09667, pruned_loss=0.01645, audio_tagging_loss=0.009366, over 3035942.52 frames. ], batch size: 59, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:02:17,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1567313.3333333333, ans=0.125
2023-11-21 16:02:18,829 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235100
2023-11-21 16:02:20,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1567313.3333333333, ans=0.2
2023-11-21 16:02:28,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1567313.3333333333, ans=0.0
2023-11-21 16:02:34,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1567380.0, ans=0.1
2023-11-21 16:02:51,420 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.38 vs. limit=15.0
2023-11-21 16:03:19,643 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.20 vs. limit=15.0
2023-11-21 16:03:22,492 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6700, loss[loss=0.05153, simple_loss=0.05634, pruned_loss=0.01181, audio_tagging_loss=0.01155, over 14623.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09657, pruned_loss=0.01652, audio_tagging_loss=0.009431, over 3035934.91 frames. ], batch size: 59, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:03:23,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235150
2023-11-21 16:03:49,615 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.083e+01 8.674e+01 9.269e+01 1.242e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 16:04:12,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1567846.6666666667, ans=0.1
2023-11-21 16:04:25,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1567980.0, ans=0.0
2023-11-21 16:04:26,632 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6750, loss[loss=0.08035, simple_loss=0.1067, pruned_loss=0.0206, audio_tagging_loss=0.006375, over 15087.00 frames. ], tot_loss[loss=0.07458, simple_loss=0.09725, pruned_loss=0.0166, audio_tagging_loss=0.009354, over 3044989.35 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:04:27,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235200
2023-11-21 16:04:30,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1567980.0, ans=0.0
2023-11-21 16:04:57,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1568113.3333333333, ans=0.0
2023-11-21 16:04:58,137 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.92 vs. limit=22.5
2023-11-21 16:05:07,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1568180.0, ans=0.2
2023-11-21 16:05:18,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1568246.6666666667, ans=0.125
2023-11-21 16:05:20,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1568246.6666666667, ans=0.125
2023-11-21 16:05:22,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1568246.6666666667, ans=0.1
2023-11-21 16:05:31,770 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6800, loss[loss=0.07987, simple_loss=0.1105, pruned_loss=0.01515, audio_tagging_loss=0.009475, over 14877.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09707, pruned_loss=0.01646, audio_tagging_loss=0.009347, over 3033944.38 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:05:33,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235250
2023-11-21 16:05:57,886 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 8.054e+01 8.686e+01 9.528e+01 1.643e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 16:05:59,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1568446.6666666667, ans=0.125
2023-11-21 16:06:12,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1568513.3333333333, ans=0.0
2023-11-21 16:06:35,965 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6850, loss[loss=0.07276, simple_loss=0.0873, pruned_loss=0.01861, audio_tagging_loss=0.0105, over 15132.00 frames. ], tot_loss[loss=0.07404, simple_loss=0.09686, pruned_loss=0.01635, audio_tagging_loss=0.009261, over 3040294.56 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:06:37,350 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235300
2023-11-21 16:06:44,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1568646.6666666667, ans=0.125
2023-11-21 16:06:57,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1568713.3333333333, ans=0.125
2023-11-21 16:07:15,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1568846.6666666667, ans=0.125
2023-11-21 16:07:18,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1568846.6666666667, ans=0.125
2023-11-21 16:07:32,706 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.35 vs. limit=15.0
2023-11-21 16:07:39,658 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6900, loss[loss=0.07363, simple_loss=0.1017, pruned_loss=0.01622, audio_tagging_loss=0.006544, over 15562.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09628, pruned_loss=0.01622, audio_tagging_loss=0.009233, over 3044742.29 frames. ], batch size: 60, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:07:41,035 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235350
2023-11-21 16:07:45,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1568980.0, ans=0.0
2023-11-21 16:07:48,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1568980.0, ans=0.04949747468305833
2023-11-21 16:08:06,945 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.623e+01 7.983e+01 8.656e+01 9.439e+01 1.131e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 16:08:13,302 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.79 vs. limit=15.0
2023-11-21 16:08:17,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1569180.0, ans=0.125
2023-11-21 16:08:22,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1569180.0, ans=0.2
2023-11-21 16:08:23,246 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.18 vs. limit=12.0
2023-11-21 16:08:30,055 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 16:08:44,339 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 6950, loss[loss=0.08888, simple_loss=0.1128, pruned_loss=0.02233, audio_tagging_loss=0.01014, over 14649.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09707, pruned_loss=0.01626, audio_tagging_loss=0.00921, over 3045232.59 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:08:45,664 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235400
2023-11-21 16:08:45,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1569313.3333333333, ans=10.0
2023-11-21 16:09:34,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff3.min_abs, batch_count=1569513.3333333333, ans=0.2
2023-11-21 16:09:50,358 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7000, loss[loss=0.05668, simple_loss=0.05985, pruned_loss=0.0126, audio_tagging_loss=0.01415, over 14608.00 frames. ], tot_loss[loss=0.07416, simple_loss=0.09729, pruned_loss=0.01623, audio_tagging_loss=0.009286, over 3047927.38 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:09:51,697 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235450
2023-11-21 16:09:54,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1569646.6666666667, ans=0.125
2023-11-21 16:09:59,391 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.19 vs. limit=15.0
2023-11-21 16:10:15,858 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.127e+01 8.792e+01 9.421e+01 1.149e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 16:10:18,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1569780.0, ans=0.025
2023-11-21 16:10:19,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1569780.0, ans=0.2
2023-11-21 16:10:27,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1569846.6666666667, ans=0.0
2023-11-21 16:10:35,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1569846.6666666667, ans=0.125
2023-11-21 16:10:44,752 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.37 vs. limit=6.0
2023-11-21 16:10:46,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1569913.3333333333, ans=0.125
2023-11-21 16:10:48,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.78 vs. limit=15.0
2023-11-21 16:10:50,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1569913.3333333333, ans=0.125
2023-11-21 16:10:53,811 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7050, loss[loss=0.09246, simple_loss=0.1232, pruned_loss=0.02268, audio_tagging_loss=0.008182, over 16033.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09624, pruned_loss=0.01606, audio_tagging_loss=0.009426, over 3046114.51 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:10:54,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1569980.0, ans=0.125
2023-11-21 16:10:55,105 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235500
2023-11-21 16:11:12,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1570046.6666666667, ans=0.125
2023-11-21 16:11:47,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1570246.6666666667, ans=0.2
2023-11-21 16:11:56,999 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7100, loss[loss=0.07453, simple_loss=0.0999, pruned_loss=0.01332, audio_tagging_loss=0.01125, over 16115.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09564, pruned_loss=0.01609, audio_tagging_loss=0.009574, over 3045090.67 frames. ], batch size: 58, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:11:58,919 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235550
2023-11-21 16:11:59,699 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.97 vs. limit=15.0
2023-11-21 16:12:08,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1570313.3333333333, ans=0.1
2023-11-21 16:12:11,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1570380.0, ans=0.0
2023-11-21 16:12:19,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1570380.0, ans=0.2
2023-11-21 16:12:25,213 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.128e+01 8.670e+01 9.348e+01 1.083e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 16:12:59,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1570646.6666666667, ans=0.2
2023-11-21 16:13:01,313 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7150, loss[loss=0.09077, simple_loss=0.1172, pruned_loss=0.02392, audio_tagging_loss=0.008248, over 15207.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09593, pruned_loss=0.01611, audio_tagging_loss=0.009635, over 3044370.08 frames. ], batch size: 57, lr: 3.42e-03, grad_scale: 16.0
2023-11-21 16:13:03,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235600
2023-11-21 16:13:11,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1570646.6666666667, ans=0.1
2023-11-21 16:13:32,011 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:13:40,659 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.75 vs. limit=15.0
2023-11-21 16:13:43,263 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.44 vs. limit=10.0
2023-11-21 16:14:05,658 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7200, loss[loss=0.1046, simple_loss=0.1437, pruned_loss=0.02543, audio_tagging_loss=0.007323, over 15236.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.0965, pruned_loss=0.01617, audio_tagging_loss=0.009665, over 3045146.35 frames. ], batch size: 55, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:14:06,994 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235650
2023-11-21 16:14:12,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1570980.0, ans=0.125
2023-11-21 16:14:18,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1571046.6666666667, ans=0.125
2023-11-21 16:14:23,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1571046.6666666667, ans=0.125
2023-11-21 16:14:32,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1571113.3333333333, ans=0.1
2023-11-21 16:14:33,170 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.636e+01 8.075e+01 8.939e+01 9.765e+01 1.346e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-21 16:14:33,803 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.36 vs. limit=10.0
2023-11-21 16:14:53,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1571180.0, ans=0.2
2023-11-21 16:14:58,010 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:15:00,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1571246.6666666667, ans=0.0
2023-11-21 16:15:08,521 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7250, loss[loss=0.0662, simple_loss=0.09335, pruned_loss=0.01244, audio_tagging_loss=0.007085, over 15343.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09677, pruned_loss=0.01639, audio_tagging_loss=0.009685, over 3049254.15 frames. ], batch size: 56, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:15:09,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235700
2023-11-21 16:15:23,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1571380.0, ans=0.1
2023-11-21 16:15:26,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1571380.0, ans=0.1
2023-11-21 16:15:32,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.00 vs. limit=15.0
2023-11-21 16:15:48,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1571513.3333333333, ans=0.0
2023-11-21 16:16:13,773 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7300, loss[loss=0.09441, simple_loss=0.1278, pruned_loss=0.02572, audio_tagging_loss=0.00478, over 16296.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.09653, pruned_loss=0.01647, audio_tagging_loss=0.009613, over 3053396.97 frames. ], batch size: 61, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:16:15,130 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235750
2023-11-21 16:16:15,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1571646.6666666667, ans=0.125
2023-11-21 16:16:42,201 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.129e+01 8.221e+01 8.789e+01 9.369e+01 1.391e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 16:16:42,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1571780.0, ans=0.125
2023-11-21 16:16:42,851 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.25 vs. limit=15.0
2023-11-21 16:16:59,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1571846.6666666667, ans=0.125
2023-11-21 16:17:11,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1571913.3333333333, ans=0.0
2023-11-21 16:17:19,093 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7350, loss[loss=0.06773, simple_loss=0.08478, pruned_loss=0.01561, audio_tagging_loss=0.009728, over 15983.00 frames. ], tot_loss[loss=0.07436, simple_loss=0.09631, pruned_loss=0.0167, audio_tagging_loss=0.009505, over 3054857.85 frames. ], batch size: 62, lr: 3.42e-03, grad_scale: 32.0
2023-11-21 16:17:20,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235800
2023-11-21 16:17:52,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1572113.3333333333, ans=0.1
2023-11-21 16:17:54,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1572113.3333333333, ans=0.125
2023-11-21 16:18:22,175 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7400, loss[loss=0.06665, simple_loss=0.09201, pruned_loss=0.01102, audio_tagging_loss=0.009623, over 14450.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09636, pruned_loss=0.01651, audio_tagging_loss=0.009429, over 3050506.87 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:18:23,510 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235850
2023-11-21 16:18:27,931 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.92 vs. limit=15.0
2023-11-21 16:18:30,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1572313.3333333333, ans=0.125
2023-11-21 16:18:47,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1572446.6666666667, ans=0.0
2023-11-21 16:18:47,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1572446.6666666667, ans=0.125
2023-11-21 16:18:50,703 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.093e+01 8.770e+01 9.313e+01 1.126e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 16:19:24,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1572580.0, ans=0.125
2023-11-21 16:19:26,601 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7450, loss[loss=0.1052, simple_loss=0.1413, pruned_loss=0.02755, audio_tagging_loss=0.007031, over 14799.00 frames. ], tot_loss[loss=0.07496, simple_loss=0.09764, pruned_loss=0.01682, audio_tagging_loss=0.00932, over 3050120.41 frames. ], batch size: 52, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:19:27,883 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235900
2023-11-21 16:19:27,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1572646.6666666667, ans=0.2
2023-11-21 16:19:31,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1572646.6666666667, ans=0.125
2023-11-21 16:19:34,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1572646.6666666667, ans=0.0
2023-11-21 16:19:46,596 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.82 vs. limit=22.5
2023-11-21 16:19:48,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff3.min_abs, batch_count=1572713.3333333333, ans=0.2
2023-11-21 16:20:15,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1572846.6666666667, ans=0.125
2023-11-21 16:20:18,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1572913.3333333333, ans=0.0
2023-11-21 16:20:27,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.78 vs. limit=15.0
2023-11-21 16:20:30,981 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7500, loss[loss=0.08096, simple_loss=0.1075, pruned_loss=0.01964, audio_tagging_loss=0.007553, over 15578.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09873, pruned_loss=0.01697, audio_tagging_loss=0.009248, over 3053775.71 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:20:32,335 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 235950
2023-11-21 16:20:43,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1573046.6666666667, ans=0.125
2023-11-21 16:20:59,384 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.902e+01 8.351e+01 8.857e+01 9.417e+01 1.207e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 16:21:16,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1573180.0, ans=0.125
2023-11-21 16:21:34,636 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7550, loss[loss=0.07639, simple_loss=0.09203, pruned_loss=0.01832, audio_tagging_loss=0.01205, over 14036.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09809, pruned_loss=0.017, audio_tagging_loss=0.009283, over 3049464.46 frames. ], batch size: 53, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:21:35,895 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236000
2023-11-21 16:21:37,401 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-236000.pt
2023-11-21 16:21:45,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1573313.3333333333, ans=0.0
2023-11-21 16:21:54,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1573380.0, ans=0.125
2023-11-21 16:22:03,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1573446.6666666667, ans=0.07
2023-11-21 16:22:32,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1573580.0, ans=0.0
2023-11-21 16:22:34,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1573580.0, ans=0.125
2023-11-21 16:22:35,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1573580.0, ans=0.1
2023-11-21 16:22:41,876 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7600, loss[loss=0.0864, simple_loss=0.1059, pruned_loss=0.02619, audio_tagging_loss=0.007263, over 14291.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09647, pruned_loss=0.01674, audio_tagging_loss=0.009411, over 3046460.03 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 32.0
2023-11-21 16:22:43,189 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236050
2023-11-21 16:23:09,954 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.352e+01 8.154e+01 8.768e+01 9.478e+01 1.233e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 16:23:10,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1573780.0, ans=0.125
2023-11-21 16:23:11,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1573780.0, ans=0.125
2023-11-21 16:23:42,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1573913.3333333333, ans=0.0
2023-11-21 16:23:46,826 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7650, loss[loss=0.05619, simple_loss=0.07454, pruned_loss=0.00989, audio_tagging_loss=0.009035, over 15167.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09582, pruned_loss=0.01668, audio_tagging_loss=0.009423, over 3048377.00 frames. ], batch size: 59, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:23:48,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236100
2023-11-21 16:24:08,655 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:24:10,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1574113.3333333333, ans=0.125
2023-11-21 16:24:25,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1574180.0, ans=0.95
2023-11-21 16:24:31,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1574180.0, ans=0.125
2023-11-21 16:24:51,714 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.38 vs. limit=22.5
2023-11-21 16:24:52,299 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7700, loss[loss=0.06406, simple_loss=0.08564, pruned_loss=0.01379, audio_tagging_loss=0.007447, over 14993.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.09628, pruned_loss=0.01669, audio_tagging_loss=0.009415, over 3052144.93 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:24:53,592 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236150
2023-11-21 16:24:59,538 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.76 vs. limit=15.0
2023-11-21 16:25:11,864 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.34 vs. limit=10.0
2023-11-21 16:25:14,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1574380.0, ans=0.0
2023-11-21 16:25:16,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1574380.0, ans=0.125
2023-11-21 16:25:22,833 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 7.998e+01 8.716e+01 9.349e+01 1.459e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 16:25:25,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1574446.6666666667, ans=0.2
2023-11-21 16:25:37,146 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.20 vs. limit=10.0
2023-11-21 16:25:57,893 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7750, loss[loss=0.0858, simple_loss=0.1157, pruned_loss=0.01838, audio_tagging_loss=0.009552, over 15785.00 frames. ], tot_loss[loss=0.0747, simple_loss=0.09688, pruned_loss=0.01682, audio_tagging_loss=0.009443, over 3052734.38 frames. ], batch size: 59, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:25:59,178 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236200
2023-11-21 16:26:30,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1574780.0, ans=0.125
2023-11-21 16:26:36,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1574846.6666666667, ans=0.09899494936611666
2023-11-21 16:26:40,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1574846.6666666667, ans=0.2
2023-11-21 16:26:46,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1574846.6666666667, ans=0.125
2023-11-21 16:26:51,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1574913.3333333333, ans=0.09899494936611666
2023-11-21 16:26:53,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.71 vs. limit=22.5
2023-11-21 16:27:03,057 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7800, loss[loss=0.08157, simple_loss=0.1161, pruned_loss=0.01569, audio_tagging_loss=0.007842, over 14649.00 frames. ], tot_loss[loss=0.07462, simple_loss=0.09682, pruned_loss=0.01674, audio_tagging_loss=0.009466, over 3044513.94 frames. ], batch size: 54, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:27:03,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1574980.0, ans=0.125
2023-11-21 16:27:03,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1574980.0, ans=0.0
2023-11-21 16:27:04,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236250
2023-11-21 16:27:09,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1574980.0, ans=0.125
2023-11-21 16:27:09,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1574980.0, ans=0.0
2023-11-21 16:27:14,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.44 vs. limit=15.0
2023-11-21 16:27:29,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1575113.3333333333, ans=0.125
2023-11-21 16:27:32,171 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.88 vs. limit=15.0
2023-11-21 16:27:32,551 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.496e+01 8.260e+01 8.924e+01 9.487e+01 1.142e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-21 16:27:47,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1575180.0, ans=0.2
2023-11-21 16:27:57,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1575246.6666666667, ans=0.125
2023-11-21 16:28:06,796 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7850, loss[loss=0.08214, simple_loss=0.1148, pruned_loss=0.01849, audio_tagging_loss=0.006249, over 15408.00 frames. ], tot_loss[loss=0.07469, simple_loss=0.09704, pruned_loss=0.01675, audio_tagging_loss=0.009415, over 3046477.06 frames. ], batch size: 54, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:28:08,196 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236300
2023-11-21 16:28:36,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1575446.6666666667, ans=0.125
2023-11-21 16:28:43,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1575446.6666666667, ans=0.5
2023-11-21 16:28:44,511 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.26 vs. limit=22.5
2023-11-21 16:29:03,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1575580.0, ans=0.1
2023-11-21 16:29:12,433 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7900, loss[loss=0.08522, simple_loss=0.1089, pruned_loss=0.01958, audio_tagging_loss=0.0112, over 14419.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.09629, pruned_loss=0.01672, audio_tagging_loss=0.009648, over 3051240.34 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:29:13,082 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.62 vs. limit=15.0
2023-11-21 16:29:13,791 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236350
2023-11-21 16:29:21,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1575646.6666666667, ans=0.09899494936611666
2023-11-21 16:29:21,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1575646.6666666667, ans=0.2
2023-11-21 16:29:32,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.29 vs. limit=22.5
2023-11-21 16:29:36,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1575780.0, ans=0.125
2023-11-21 16:29:39,780 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.31 vs. limit=12.0
2023-11-21 16:29:42,402 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.695e+01 8.223e+01 8.741e+01 9.737e+01 2.619e+02, threshold=1.748e+02, percent-clipped=1.0
2023-11-21 16:29:51,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1575846.6666666667, ans=0.1
2023-11-21 16:29:55,089 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.25 vs. limit=15.0
2023-11-21 16:29:56,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1575846.6666666667, ans=0.0
2023-11-21 16:30:07,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1575913.3333333333, ans=0.0
2023-11-21 16:30:16,764 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 7950, loss[loss=0.07965, simple_loss=0.1021, pruned_loss=0.01903, audio_tagging_loss=0.009591, over 15426.00 frames. ], tot_loss[loss=0.07439, simple_loss=0.09593, pruned_loss=0.01663, audio_tagging_loss=0.009795, over 3053122.35 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:30:18,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236400
2023-11-21 16:30:26,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1575980.0, ans=0.125
2023-11-21 16:30:30,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1576046.6666666667, ans=0.0
2023-11-21 16:30:31,828 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 16:31:09,119 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.30 vs. limit=15.0
2023-11-21 16:31:17,375 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.23 vs. limit=10.0
2023-11-21 16:31:20,504 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8000, loss[loss=0.08043, simple_loss=0.1056, pruned_loss=0.01916, audio_tagging_loss=0.008464, over 14561.00 frames. ], tot_loss[loss=0.07383, simple_loss=0.09524, pruned_loss=0.01632, audio_tagging_loss=0.009895, over 3046423.88 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:31:21,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236450
2023-11-21 16:31:23,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1576313.3333333333, ans=0.0
2023-11-21 16:31:39,633 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.10 vs. limit=22.5
2023-11-21 16:31:42,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1576380.0, ans=0.1
2023-11-21 16:31:48,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1576446.6666666667, ans=0.0
2023-11-21 16:31:52,251 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.291e+01 7.990e+01 8.622e+01 9.652e+01 1.226e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-21 16:32:01,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1576513.3333333333, ans=0.125
2023-11-21 16:32:20,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1576580.0, ans=0.0
2023-11-21 16:32:23,867 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8050, loss[loss=0.07798, simple_loss=0.097, pruned_loss=0.01694, audio_tagging_loss=0.01254, over 15787.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09541, pruned_loss=0.01624, audio_tagging_loss=0.009911, over 3042010.71 frames. ], batch size: 59, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:32:25,827 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236500
2023-11-21 16:32:27,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1576646.6666666667, ans=0.125
2023-11-21 16:32:31,188 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.07 vs. limit=22.5
2023-11-21 16:32:31,376 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.72 vs. limit=15.0
2023-11-21 16:32:48,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1576713.3333333333, ans=0.0
2023-11-21 16:33:18,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1576913.3333333333, ans=22.5
2023-11-21 16:33:28,746 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8100, loss[loss=0.07882, simple_loss=0.09046, pruned_loss=0.02273, audio_tagging_loss=0.01086, over 14812.00 frames. ], tot_loss[loss=0.07403, simple_loss=0.0953, pruned_loss=0.01655, audio_tagging_loss=0.009832, over 3037510.99 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:33:30,054 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236550
2023-11-21 16:33:47,371 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:33:51,086 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:33:58,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1577113.3333333333, ans=0.2
2023-11-21 16:33:59,138 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.498e+01 8.221e+01 8.771e+01 9.447e+01 1.298e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 16:34:18,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1577246.6666666667, ans=0.1
2023-11-21 16:34:29,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1577246.6666666667, ans=0.125
2023-11-21 16:34:31,738 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8150, loss[loss=0.07909, simple_loss=0.1041, pruned_loss=0.01843, audio_tagging_loss=0.008589, over 15382.00 frames. ], tot_loss[loss=0.07424, simple_loss=0.0959, pruned_loss=0.01656, audio_tagging_loss=0.009729, over 3039489.28 frames. ], batch size: 56, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:34:33,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236600
2023-11-21 16:34:39,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1577313.3333333333, ans=0.1
2023-11-21 16:34:44,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1577380.0, ans=0.125
2023-11-21 16:34:44,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1577380.0, ans=0.125
2023-11-21 16:34:49,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1577380.0, ans=0.0
2023-11-21 16:34:55,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1577380.0, ans=0.125
2023-11-21 16:35:03,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1577446.6666666667, ans=0.0
2023-11-21 16:35:14,835 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:35:26,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1577580.0, ans=0.0
2023-11-21 16:35:34,886 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8200, loss[loss=0.07866, simple_loss=0.09092, pruned_loss=0.02295, audio_tagging_loss=0.01025, over 14284.00 frames. ], tot_loss[loss=0.07403, simple_loss=0.0958, pruned_loss=0.01645, audio_tagging_loss=0.009682, over 3039112.26 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:35:34,956 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 16:35:35,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1577646.6666666667, ans=0.125
2023-11-21 16:35:36,131 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236650
2023-11-21 16:36:07,540 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.532e+01 8.090e+01 8.732e+01 9.271e+01 1.131e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 16:36:10,741 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.42 vs. limit=15.0
2023-11-21 16:36:12,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1577846.6666666667, ans=0.125
2023-11-21 16:36:40,221 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8250, loss[loss=0.05057, simple_loss=0.06551, pruned_loss=0.006729, audio_tagging_loss=0.01109, over 15443.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09517, pruned_loss=0.01618, audio_tagging_loss=0.009625, over 3037892.39 frames. ], batch size: 60, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:36:41,528 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236700
2023-11-21 16:36:53,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1578046.6666666667, ans=0.1
2023-11-21 16:36:54,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1578046.6666666667, ans=0.0
2023-11-21 16:37:01,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1578046.6666666667, ans=0.125
2023-11-21 16:37:29,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1578246.6666666667, ans=0.125
2023-11-21 16:37:30,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1578246.6666666667, ans=0.1
2023-11-21 16:37:43,354 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8300, loss[loss=0.08496, simple_loss=0.1154, pruned_loss=0.0195, audio_tagging_loss=0.007769, over 16600.00 frames. ], tot_loss[loss=0.07296, simple_loss=0.09463, pruned_loss=0.01602, audio_tagging_loss=0.009623, over 3046924.13 frames. ], batch size: 60, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:37:44,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236750
2023-11-21 16:37:49,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1578313.3333333333, ans=0.0
2023-11-21 16:37:52,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1578313.3333333333, ans=0.0
2023-11-21 16:38:05,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1578380.0, ans=0.0
2023-11-21 16:38:14,686 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.835e+01 8.193e+01 8.671e+01 9.431e+01 1.215e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 16:38:44,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1578646.6666666667, ans=0.125
2023-11-21 16:38:45,472 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8350, loss[loss=0.08986, simple_loss=0.1252, pruned_loss=0.02038, audio_tagging_loss=0.006864, over 15309.00 frames. ], tot_loss[loss=0.07297, simple_loss=0.09482, pruned_loss=0.01605, audio_tagging_loss=0.009509, over 3045318.11 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 8.0
2023-11-21 16:38:46,744 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236800
2023-11-21 16:39:01,885 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.64 vs. limit=15.0
2023-11-21 16:39:24,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1578846.6666666667, ans=0.1
2023-11-21 16:39:35,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=1578913.3333333333, ans=15.0
2023-11-21 16:39:49,518 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8400, loss[loss=0.07196, simple_loss=0.08749, pruned_loss=0.017, audio_tagging_loss=0.01122, over 14406.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09387, pruned_loss=0.0158, audio_tagging_loss=0.009471, over 3042023.92 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:39:50,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236850
2023-11-21 16:39:53,186 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0
2023-11-21 16:40:13,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.17 vs. limit=6.0
2023-11-21 16:40:15,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1579113.3333333333, ans=0.1
2023-11-21 16:40:21,104 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 7.886e+01 8.656e+01 9.499e+01 1.187e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 16:40:28,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1579180.0, ans=0.05
2023-11-21 16:40:52,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1579313.3333333333, ans=0.1
2023-11-21 16:40:52,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.75 vs. limit=12.0
2023-11-21 16:40:53,314 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8450, loss[loss=0.08318, simple_loss=0.1216, pruned_loss=0.01266, audio_tagging_loss=0.009698, over 14599.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09517, pruned_loss=0.01602, audio_tagging_loss=0.009496, over 3054064.04 frames. ], batch size: 53, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:40:54,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236900
2023-11-21 16:40:56,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1579313.3333333333, ans=0.1
2023-11-21 16:41:20,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1579446.6666666667, ans=0.0
2023-11-21 16:41:28,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=1579446.6666666667, ans=15.0
2023-11-21 16:41:29,000 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:41:30,607 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.65 vs. limit=6.0
2023-11-21 16:41:31,946 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.16 vs. limit=22.5
2023-11-21 16:41:40,138 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.88 vs. limit=15.0
2023-11-21 16:41:49,651 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:41:56,721 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8500, loss[loss=0.07489, simple_loss=0.09563, pruned_loss=0.01868, audio_tagging_loss=0.0084, over 16291.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09426, pruned_loss=0.01593, audio_tagging_loss=0.009539, over 3042623.92 frames. ], batch size: 60, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:41:58,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 236950
2023-11-21 16:42:12,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1579713.3333333333, ans=0.5
2023-11-21 16:42:27,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1579780.0, ans=0.125
2023-11-21 16:42:29,762 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.566e+01 7.919e+01 8.659e+01 9.250e+01 1.200e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 16:42:50,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1579913.3333333333, ans=0.125
2023-11-21 16:43:01,483 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8550, loss[loss=0.05701, simple_loss=0.07399, pruned_loss=0.01066, audio_tagging_loss=0.009349, over 14305.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.0953, pruned_loss=0.01611, audio_tagging_loss=0.009562, over 3037769.92 frames. ], batch size: 58, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:43:02,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237000
2023-11-21 16:44:00,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1580246.6666666667, ans=0.125
2023-11-21 16:44:06,836 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8600, loss[loss=0.05608, simple_loss=0.07442, pruned_loss=0.01068, audio_tagging_loss=0.008194, over 15007.00 frames. ], tot_loss[loss=0.073, simple_loss=0.09487, pruned_loss=0.01601, audio_tagging_loss=0.009553, over 3035516.60 frames. ], batch size: 59, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:44:08,110 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237050
2023-11-21 16:44:26,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1580380.0, ans=0.125
2023-11-21 16:44:37,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1580446.6666666667, ans=0.125
2023-11-21 16:44:38,174 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 8.198e+01 8.655e+01 9.570e+01 1.230e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 16:45:09,710 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8650, loss[loss=0.07308, simple_loss=0.09774, pruned_loss=0.0151, audio_tagging_loss=0.009108, over 14870.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.0951, pruned_loss=0.01609, audio_tagging_loss=0.009716, over 3040714.96 frames. ], batch size: 55, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:45:10,994 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237100
2023-11-21 16:45:30,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1580713.3333333333, ans=0.0
2023-11-21 16:45:43,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.92 vs. limit=15.0
2023-11-21 16:45:47,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1580846.6666666667, ans=0.125
2023-11-21 16:45:48,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1580846.6666666667, ans=0.0
2023-11-21 16:45:48,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1580846.6666666667, ans=0.125
2023-11-21 16:45:51,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1580846.6666666667, ans=0.1
2023-11-21 16:46:14,127 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8700, loss[loss=0.08431, simple_loss=0.1069, pruned_loss=0.02112, audio_tagging_loss=0.009759, over 15332.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09465, pruned_loss=0.01612, audio_tagging_loss=0.009823, over 3051279.03 frames. ], batch size: 57, lr: 3.41e-03, grad_scale: 16.0
2023-11-21 16:46:15,402 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237150
2023-11-21 16:46:17,258 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.43 vs. limit=12.0
2023-11-21 16:46:23,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.75 vs. limit=15.0
2023-11-21 16:46:45,559 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.271e+01 8.853e+01 9.798e+01 1.350e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-21 16:46:53,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1581180.0, ans=0.1
2023-11-21 16:47:17,474 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8750, loss[loss=0.06253, simple_loss=0.08339, pruned_loss=0.01061, audio_tagging_loss=0.01023, over 15728.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09432, pruned_loss=0.01606, audio_tagging_loss=0.00988, over 3046350.43 frames. ], batch size: 59, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 16:47:18,132 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.67 vs. limit=15.0
2023-11-21 16:47:18,755 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237200
2023-11-21 16:47:20,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1581313.3333333333, ans=0.125
2023-11-21 16:47:28,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1581313.3333333333, ans=0.1
2023-11-21 16:47:49,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1581446.6666666667, ans=0.0
2023-11-21 16:48:03,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1581513.3333333333, ans=0.0
2023-11-21 16:48:05,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1581513.3333333333, ans=0.125
2023-11-21 16:48:17,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1581580.0, ans=0.125
2023-11-21 16:48:21,626 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8800, loss[loss=0.09308, simple_loss=0.1282, pruned_loss=0.02378, audio_tagging_loss=0.005201, over 15604.00 frames. ], tot_loss[loss=0.07313, simple_loss=0.09426, pruned_loss=0.01608, audio_tagging_loss=0.009926, over 3048938.05 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:48:22,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237250
2023-11-21 16:48:33,199 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.94 vs. limit=22.5
2023-11-21 16:48:35,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1581713.3333333333, ans=0.1
2023-11-21 16:48:47,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1581780.0, ans=0.2
2023-11-21 16:48:48,536 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:48:53,818 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.845e+01 8.270e+01 9.064e+01 1.008e+02 1.291e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-21 16:49:01,082 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.90 vs. limit=15.0
2023-11-21 16:49:25,646 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8850, loss[loss=0.06495, simple_loss=0.09331, pruned_loss=0.01057, audio_tagging_loss=0.007724, over 16947.00 frames. ], tot_loss[loss=0.07422, simple_loss=0.0959, pruned_loss=0.01641, audio_tagging_loss=0.009863, over 3052375.63 frames. ], batch size: 64, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:49:26,906 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237300
2023-11-21 16:49:30,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1581980.0, ans=0.0
2023-11-21 16:49:38,439 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 16:49:46,924 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.24 vs. limit=15.0
2023-11-21 16:49:53,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1582113.3333333333, ans=0.125
2023-11-21 16:50:00,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1582113.3333333333, ans=0.09899494936611666
2023-11-21 16:50:05,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1582180.0, ans=0.1
2023-11-21 16:50:18,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1582246.6666666667, ans=0.0
2023-11-21 16:50:30,407 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8900, loss[loss=0.07828, simple_loss=0.1054, pruned_loss=0.01662, audio_tagging_loss=0.008941, over 14242.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.0964, pruned_loss=0.01655, audio_tagging_loss=0.009722, over 3049552.79 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:50:31,686 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237350
2023-11-21 16:51:01,802 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 7.931e+01 8.617e+01 9.547e+01 1.329e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 16:51:07,696 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:51:33,480 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 8950, loss[loss=0.09278, simple_loss=0.1173, pruned_loss=0.0245, audio_tagging_loss=0.009622, over 14430.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.09602, pruned_loss=0.01636, audio_tagging_loss=0.009577, over 3053084.47 frames. ], batch size: 53, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:51:34,809 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237400
2023-11-21 16:51:43,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1582646.6666666667, ans=0.1
2023-11-21 16:52:01,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1582780.0, ans=0.125
2023-11-21 16:52:12,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1582846.6666666667, ans=0.125
2023-11-21 16:52:38,588 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9000, loss[loss=0.06369, simple_loss=0.0841, pruned_loss=0.01235, audio_tagging_loss=0.009295, over 15628.00 frames. ], tot_loss[loss=0.07483, simple_loss=0.09709, pruned_loss=0.0168, audio_tagging_loss=0.009489, over 3051997.44 frames. ], batch size: 60, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:52:38,592 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 16:53:17,962 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8536, 3.5544, 4.8607, 4.4277], device='cuda:0')
2023-11-21 16:53:19,764 INFO [train_asr.py:1253] (0/4) Epoch 20, validation: loss=0.06046, simple_loss=0.05217, pruned_loss=0.005278, audio_tagging_loss=0.0291, over 4681554.00 frames. 
2023-11-21 16:53:19,765 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 16:53:21,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237450
2023-11-21 16:53:23,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1582980.0, ans=0.1
2023-11-21 16:53:47,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1583113.3333333333, ans=0.125
2023-11-21 16:53:51,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1583113.3333333333, ans=0.125
2023-11-21 16:53:52,349 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.325e+01 8.299e+01 8.982e+01 9.463e+01 1.306e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-21 16:54:00,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1583180.0, ans=0.0
2023-11-21 16:54:02,035 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.49 vs. limit=15.0
2023-11-21 16:54:11,837 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:54:22,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.26 vs. limit=15.0
2023-11-21 16:54:22,597 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9050, loss[loss=0.08029, simple_loss=0.1054, pruned_loss=0.0188, audio_tagging_loss=0.008809, over 15743.00 frames. ], tot_loss[loss=0.07517, simple_loss=0.09783, pruned_loss=0.01688, audio_tagging_loss=0.009378, over 3053888.25 frames. ], batch size: 60, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 16:54:22,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1583313.3333333333, ans=0.1
2023-11-21 16:54:23,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237500
2023-11-21 16:54:54,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1583446.6666666667, ans=0.0
2023-11-21 16:55:00,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1583513.3333333333, ans=0.2
2023-11-21 16:55:05,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1583513.3333333333, ans=0.1
2023-11-21 16:55:17,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1583580.0, ans=0.125
2023-11-21 16:55:22,422 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.24 vs. limit=15.0
2023-11-21 16:55:24,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1583580.0, ans=0.0
2023-11-21 16:55:27,151 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9100, loss[loss=0.08544, simple_loss=0.1195, pruned_loss=0.01817, audio_tagging_loss=0.007541, over 16728.00 frames. ], tot_loss[loss=0.07505, simple_loss=0.09793, pruned_loss=0.01679, audio_tagging_loss=0.009299, over 3054060.49 frames. ], batch size: 61, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 16:55:28,495 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237550
2023-11-21 16:55:51,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1583780.0, ans=0.1
2023-11-21 16:55:54,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1583780.0, ans=0.09899494936611666
2023-11-21 16:56:00,161 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.100e+01 8.666e+01 9.594e+01 1.245e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 16:56:12,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1583846.6666666667, ans=0.125
2023-11-21 16:56:32,107 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9150, loss[loss=0.06187, simple_loss=0.08021, pruned_loss=0.01145, audio_tagging_loss=0.01032, over 14610.00 frames. ], tot_loss[loss=0.07427, simple_loss=0.09699, pruned_loss=0.0164, audio_tagging_loss=0.00938, over 3047001.46 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 16:56:33,382 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237600
2023-11-21 16:56:41,637 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.32 vs. limit=15.0
2023-11-21 16:56:49,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1584046.6666666667, ans=0.1
2023-11-21 16:56:55,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1584113.3333333333, ans=0.2
2023-11-21 16:57:06,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1584113.3333333333, ans=0.125
2023-11-21 16:57:22,834 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.24 vs. limit=22.5
2023-11-21 16:57:29,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1584246.6666666667, ans=10.0
2023-11-21 16:57:35,468 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9200, loss[loss=0.04495, simple_loss=0.05594, pruned_loss=0.009176, audio_tagging_loss=0.007809, over 15636.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09647, pruned_loss=0.01621, audio_tagging_loss=0.009411, over 3052509.32 frames. ], batch size: 60, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:57:36,748 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237650
2023-11-21 16:57:40,525 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:57:41,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1584313.3333333333, ans=0.0
2023-11-21 16:58:08,692 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.617e+01 8.012e+01 8.576e+01 9.231e+01 1.366e+02, threshold=1.715e+02, percent-clipped=0.0
2023-11-21 16:58:12,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1584513.3333333333, ans=0.0
2023-11-21 16:58:21,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff3.min_abs, batch_count=1584513.3333333333, ans=0.2
2023-11-21 16:58:37,786 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9250, loss[loss=0.05062, simple_loss=0.06386, pruned_loss=0.01012, audio_tagging_loss=0.008568, over 14510.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09541, pruned_loss=0.01622, audio_tagging_loss=0.009497, over 3051304.07 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:58:39,705 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237700
2023-11-21 16:58:41,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1584646.6666666667, ans=0.5
2023-11-21 16:58:56,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1584713.3333333333, ans=0.125
2023-11-21 16:58:59,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1584713.3333333333, ans=0.1
2023-11-21 16:59:09,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1584780.0, ans=0.2
2023-11-21 16:59:11,932 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 16:59:33,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1584913.3333333333, ans=0.125
2023-11-21 16:59:42,842 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9300, loss[loss=0.08339, simple_loss=0.1096, pruned_loss=0.0192, audio_tagging_loss=0.009397, over 15992.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09508, pruned_loss=0.01611, audio_tagging_loss=0.009558, over 3051195.15 frames. ], batch size: 59, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 16:59:44,220 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237750
2023-11-21 17:00:10,706 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.30 vs. limit=8.0
2023-11-21 17:00:14,594 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.079e+01 8.667e+01 9.296e+01 1.099e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 17:00:25,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-21 17:00:37,911 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.33 vs. limit=22.5
2023-11-21 17:00:45,979 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9350, loss[loss=0.06955, simple_loss=0.08659, pruned_loss=0.01758, audio_tagging_loss=0.008672, over 14623.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09476, pruned_loss=0.01603, audio_tagging_loss=0.009673, over 3051242.36 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:00:47,287 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237800
2023-11-21 17:00:52,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1585313.3333333333, ans=0.125
2023-11-21 17:01:33,587 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:01:47,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1585580.0, ans=0.0
2023-11-21 17:01:48,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1585646.6666666667, ans=0.025
2023-11-21 17:01:49,454 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9400, loss[loss=0.06589, simple_loss=0.07421, pruned_loss=0.01409, audio_tagging_loss=0.0147, over 15841.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09467, pruned_loss=0.01588, audio_tagging_loss=0.009842, over 3053097.74 frames. ], batch size: 61, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:01:50,862 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237850
2023-11-21 17:01:58,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1585646.6666666667, ans=0.5
2023-11-21 17:02:04,655 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.19 vs. limit=22.5
2023-11-21 17:02:20,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1585780.0, ans=0.125
2023-11-21 17:02:24,231 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.888e+01 8.317e+01 9.011e+01 9.971e+01 1.419e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-21 17:02:40,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1585913.3333333333, ans=0.125
2023-11-21 17:02:53,513 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:02:55,306 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9450, loss[loss=0.07198, simple_loss=0.08141, pruned_loss=0.02029, audio_tagging_loss=0.01098, over 13848.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09553, pruned_loss=0.0161, audio_tagging_loss=0.009724, over 3053503.19 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:02:56,571 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237900
2023-11-21 17:03:04,055 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.00 vs. limit=15.0
2023-11-21 17:03:19,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1586113.3333333333, ans=0.1
2023-11-21 17:03:29,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1586113.3333333333, ans=0.05
2023-11-21 17:03:59,168 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9500, loss[loss=0.07883, simple_loss=0.1112, pruned_loss=0.0153, audio_tagging_loss=0.007945, over 14458.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09421, pruned_loss=0.01595, audio_tagging_loss=0.009848, over 3056798.36 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:04:00,478 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 237950
2023-11-21 17:04:05,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1586313.3333333333, ans=0.125
2023-11-21 17:04:16,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1586380.0, ans=0.1
2023-11-21 17:04:31,965 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.163e+01 8.915e+01 9.675e+01 1.127e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-21 17:04:32,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1586446.6666666667, ans=0.125
2023-11-21 17:04:34,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1586446.6666666667, ans=0.125
2023-11-21 17:04:38,390 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.57 vs. limit=15.0
2023-11-21 17:04:41,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1586513.3333333333, ans=0.125
2023-11-21 17:04:41,821 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.80 vs. limit=15.0
2023-11-21 17:04:43,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1586513.3333333333, ans=0.125
2023-11-21 17:04:52,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1586580.0, ans=0.0
2023-11-21 17:04:54,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1586580.0, ans=0.125
2023-11-21 17:05:01,660 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9550, loss[loss=0.08812, simple_loss=0.1292, pruned_loss=0.01796, audio_tagging_loss=0.005569, over 15041.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09529, pruned_loss=0.01625, audio_tagging_loss=0.009865, over 3060117.74 frames. ], batch size: 56, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:05:03,001 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238000
2023-11-21 17:05:06,572 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.94 vs. limit=15.0
2023-11-21 17:05:13,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1586646.6666666667, ans=0.1
2023-11-21 17:05:20,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1586713.3333333333, ans=0.1
2023-11-21 17:05:24,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1586713.3333333333, ans=0.2
2023-11-21 17:05:37,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1586780.0, ans=0.2
2023-11-21 17:05:49,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1586846.6666666667, ans=0.1
2023-11-21 17:05:54,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1586913.3333333333, ans=0.0
2023-11-21 17:06:06,212 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9600, loss[loss=0.06965, simple_loss=0.09001, pruned_loss=0.01425, audio_tagging_loss=0.0104, over 15054.00 frames. ], tot_loss[loss=0.07398, simple_loss=0.09552, pruned_loss=0.01631, audio_tagging_loss=0.009913, over 3056346.90 frames. ], batch size: 58, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:06:07,548 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238050
2023-11-21 17:06:38,577 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.933e+01 8.410e+01 9.133e+01 9.513e+01 1.325e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-21 17:06:44,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1587180.0, ans=0.2
2023-11-21 17:06:56,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1587246.6666666667, ans=0.0
2023-11-21 17:07:05,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1587246.6666666667, ans=0.1
2023-11-21 17:07:10,206 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9650, loss[loss=0.06713, simple_loss=0.08887, pruned_loss=0.0128, audio_tagging_loss=0.009895, over 14443.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09446, pruned_loss=0.01618, audio_tagging_loss=0.009944, over 3042532.43 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:07:11,571 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238100
2023-11-21 17:07:11,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1587313.3333333333, ans=0.5
2023-11-21 17:07:13,360 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=15.0
2023-11-21 17:07:16,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1587313.3333333333, ans=0.0
2023-11-21 17:07:17,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1587313.3333333333, ans=0.1
2023-11-21 17:07:25,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1587380.0, ans=0.1
2023-11-21 17:08:01,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1587580.0, ans=0.025
2023-11-21 17:08:07,963 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.71 vs. limit=15.0
2023-11-21 17:08:11,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.whiten.whitening_limit, batch_count=1587580.0, ans=12.0
2023-11-21 17:08:13,230 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9700, loss[loss=0.06919, simple_loss=0.08666, pruned_loss=0.0144, audio_tagging_loss=0.01145, over 14936.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09437, pruned_loss=0.01609, audio_tagging_loss=0.009771, over 3044385.04 frames. ], batch size: 59, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:08:14,577 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238150
2023-11-21 17:08:28,008 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:08:45,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1587780.0, ans=0.015
2023-11-21 17:08:46,892 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.277e+01 8.905e+01 9.605e+01 1.183e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 17:09:02,948 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:09:08,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1587913.3333333333, ans=0.125
2023-11-21 17:09:15,527 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.01 vs. limit=15.0
2023-11-21 17:09:17,301 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9750, loss[loss=0.06385, simple_loss=0.08833, pruned_loss=0.01103, audio_tagging_loss=0.008649, over 16386.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.09439, pruned_loss=0.01622, audio_tagging_loss=0.00961, over 3043215.97 frames. ], batch size: 61, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:09:18,584 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238200
2023-11-21 17:09:45,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1588113.3333333333, ans=0.125
2023-11-21 17:09:48,422 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.68 vs. limit=12.0
2023-11-21 17:09:54,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1588180.0, ans=0.125
2023-11-21 17:10:02,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1588180.0, ans=0.125
2023-11-21 17:10:06,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1588180.0, ans=0.125
2023-11-21 17:10:08,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1588246.6666666667, ans=0.125
2023-11-21 17:10:08,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1588246.6666666667, ans=0.04949747468305833
2023-11-21 17:10:21,272 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9800, loss[loss=0.07219, simple_loss=0.09153, pruned_loss=0.0153, audio_tagging_loss=0.01113, over 15814.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.0951, pruned_loss=0.01651, audio_tagging_loss=0.009555, over 3042265.17 frames. ], batch size: 59, lr: 3.40e-03, grad_scale: 32.0
2023-11-21 17:10:22,672 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238250
2023-11-21 17:10:53,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1588446.6666666667, ans=0.2
2023-11-21 17:10:54,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1588446.6666666667, ans=0.0
2023-11-21 17:10:55,715 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.038e+01 8.658e+01 9.399e+01 1.112e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 17:11:13,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1588580.0, ans=0.0
2023-11-21 17:11:16,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1588580.0, ans=0.0
2023-11-21 17:11:17,622 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:11:18,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1588580.0, ans=0.125
2023-11-21 17:11:24,847 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9850, loss[loss=0.07731, simple_loss=0.1008, pruned_loss=0.01813, audio_tagging_loss=0.008803, over 15052.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.09619, pruned_loss=0.01668, audio_tagging_loss=0.009416, over 3047352.38 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 17:11:26,113 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238300
2023-11-21 17:11:43,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1588713.3333333333, ans=0.125
2023-11-21 17:11:53,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1588780.0, ans=0.125
2023-11-21 17:12:23,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1588913.3333333333, ans=0.1
2023-11-21 17:12:29,613 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9900, loss[loss=0.08605, simple_loss=0.1039, pruned_loss=0.02137, audio_tagging_loss=0.01272, over 15497.00 frames. ], tot_loss[loss=0.07458, simple_loss=0.09657, pruned_loss=0.01679, audio_tagging_loss=0.009501, over 3045795.28 frames. ], batch size: 57, lr: 3.40e-03, grad_scale: 8.0
2023-11-21 17:12:29,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1588980.0, ans=0.0
2023-11-21 17:12:30,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238350
2023-11-21 17:12:52,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1589046.6666666667, ans=0.125
2023-11-21 17:12:57,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1589113.3333333333, ans=0.125
2023-11-21 17:13:04,958 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.984e+01 7.926e+01 8.564e+01 9.409e+01 1.276e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 17:13:30,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1589246.6666666667, ans=0.125
2023-11-21 17:13:33,216 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 9950, loss[loss=0.0557, simple_loss=0.07676, pruned_loss=0.01076, audio_tagging_loss=0.006565, over 14205.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09621, pruned_loss=0.01654, audio_tagging_loss=0.009473, over 3046756.60 frames. ], batch size: 54, lr: 3.40e-03, grad_scale: 8.0
2023-11-21 17:13:34,511 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238400
2023-11-21 17:13:49,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1589380.0, ans=0.125
2023-11-21 17:14:04,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1589446.6666666667, ans=0.2
2023-11-21 17:14:17,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1589513.3333333333, ans=0.1
2023-11-21 17:14:25,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1589580.0, ans=0.125
2023-11-21 17:14:36,969 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10000, loss[loss=0.06701, simple_loss=0.09421, pruned_loss=0.01369, audio_tagging_loss=0.006219, over 14964.00 frames. ], tot_loss[loss=0.07383, simple_loss=0.09591, pruned_loss=0.01643, audio_tagging_loss=0.009441, over 3047132.03 frames. ], batch size: 55, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 17:14:38,308 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238450
2023-11-21 17:14:49,927 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=23.65 vs. limit=22.5
2023-11-21 17:15:03,128 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:15:04,642 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.06 vs. limit=12.0
2023-11-21 17:15:08,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1589780.0, ans=0.125
2023-11-21 17:15:13,230 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.031e+01 7.913e+01 8.716e+01 9.510e+01 1.205e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 17:15:23,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1589846.6666666667, ans=0.1
2023-11-21 17:15:31,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1589913.3333333333, ans=0.1
2023-11-21 17:15:31,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1589913.3333333333, ans=0.0
2023-11-21 17:15:37,989 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.59 vs. limit=12.0
2023-11-21 17:15:41,451 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10050, loss[loss=0.06179, simple_loss=0.07587, pruned_loss=0.01454, audio_tagging_loss=0.009311, over 15186.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09524, pruned_loss=0.01638, audio_tagging_loss=0.009469, over 3048048.48 frames. ], batch size: 60, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 17:15:42,746 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238500
2023-11-21 17:15:45,924 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:16:07,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1590113.3333333333, ans=0.2
2023-11-21 17:16:20,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1590180.0, ans=0.125
2023-11-21 17:16:30,910 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.43 vs. limit=15.0
2023-11-21 17:16:46,166 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10100, loss[loss=0.09321, simple_loss=0.1237, pruned_loss=0.02597, audio_tagging_loss=0.005394, over 16469.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09594, pruned_loss=0.01649, audio_tagging_loss=0.009544, over 3054514.41 frames. ], batch size: 59, lr: 3.40e-03, grad_scale: 16.0
2023-11-21 17:16:47,395 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238550
2023-11-21 17:16:57,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1590380.0, ans=0.2
2023-11-21 17:17:06,596 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.82 vs. limit=22.5
2023-11-21 17:17:22,051 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.174e+01 9.110e+01 9.882e+01 1.190e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-21 17:17:22,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1590446.6666666667, ans=0.125
2023-11-21 17:17:36,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1590580.0, ans=0.0
2023-11-21 17:17:37,873 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:17:38,499 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.61 vs. limit=15.0
2023-11-21 17:17:49,964 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10150, loss[loss=0.09105, simple_loss=0.1072, pruned_loss=0.02448, audio_tagging_loss=0.01295, over 15577.00 frames. ], tot_loss[loss=0.07471, simple_loss=0.09683, pruned_loss=0.01673, audio_tagging_loss=0.009563, over 3059276.49 frames. ], batch size: 60, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:17:51,331 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238600
2023-11-21 17:18:20,040 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:18:21,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1590780.0, ans=0.1
2023-11-21 17:18:28,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1590846.6666666667, ans=0.0
2023-11-21 17:18:35,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1590846.6666666667, ans=0.0
2023-11-21 17:18:36,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1590846.6666666667, ans=0.125
2023-11-21 17:18:41,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1590913.3333333333, ans=0.0
2023-11-21 17:18:50,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1590913.3333333333, ans=0.125
2023-11-21 17:18:54,015 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10200, loss[loss=0.09552, simple_loss=0.1163, pruned_loss=0.03039, audio_tagging_loss=0.007006, over 14807.00 frames. ], tot_loss[loss=0.07458, simple_loss=0.09666, pruned_loss=0.01673, audio_tagging_loss=0.009521, over 3060428.89 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:18:55,297 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238650
2023-11-21 17:19:01,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1590980.0, ans=0.1
2023-11-21 17:19:05,651 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.24 vs. limit=15.0
2023-11-21 17:19:13,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1591046.6666666667, ans=0.125
2023-11-21 17:19:17,857 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:19:18,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1591046.6666666667, ans=0.1
2023-11-21 17:19:22,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1591113.3333333333, ans=0.0
2023-11-21 17:19:24,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1591113.3333333333, ans=10.0
2023-11-21 17:19:24,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1591113.3333333333, ans=0.125
2023-11-21 17:19:29,933 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.636e+01 9.230e+01 9.917e+01 1.306e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-21 17:19:30,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1591113.3333333333, ans=0.2
2023-11-21 17:19:56,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1591246.6666666667, ans=0.2
2023-11-21 17:19:58,402 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10250, loss[loss=0.07007, simple_loss=0.08067, pruned_loss=0.01704, audio_tagging_loss=0.01269, over 15389.00 frames. ], tot_loss[loss=0.07464, simple_loss=0.09692, pruned_loss=0.0166, audio_tagging_loss=0.009583, over 3062904.32 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:19:59,716 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238700
2023-11-21 17:20:07,553 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:20:15,483 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.88 vs. limit=15.0
2023-11-21 17:20:19,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1591380.0, ans=0.05
2023-11-21 17:20:30,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1591446.6666666667, ans=0.125
2023-11-21 17:20:54,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1591580.0, ans=0.1
2023-11-21 17:21:01,965 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10300, loss[loss=0.06791, simple_loss=0.08707, pruned_loss=0.01516, audio_tagging_loss=0.009217, over 15058.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.09616, pruned_loss=0.01645, audio_tagging_loss=0.00975, over 3056489.24 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:21:03,217 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238750
2023-11-21 17:21:13,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.74 vs. limit=15.0
2023-11-21 17:21:38,273 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.809e+01 7.986e+01 8.562e+01 9.250e+01 1.387e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 17:21:41,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1591846.6666666667, ans=0.0
2023-11-21 17:21:41,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1591846.6666666667, ans=0.0
2023-11-21 17:21:49,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1591846.6666666667, ans=0.0
2023-11-21 17:22:05,335 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10350, loss[loss=0.07554, simple_loss=0.1023, pruned_loss=0.01553, audio_tagging_loss=0.008844, over 15123.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09571, pruned_loss=0.01618, audio_tagging_loss=0.009763, over 3057879.17 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:22:06,711 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238800
2023-11-21 17:22:11,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1591980.0, ans=0.2
2023-11-21 17:22:39,589 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.10 vs. limit=10.0
2023-11-21 17:23:07,333 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.16 vs. limit=15.0
2023-11-21 17:23:10,386 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10400, loss[loss=0.06211, simple_loss=0.07791, pruned_loss=0.0139, audio_tagging_loss=0.009264, over 14106.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09491, pruned_loss=0.01618, audio_tagging_loss=0.009861, over 3049963.63 frames. ], batch size: 54, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:23:11,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238850
2023-11-21 17:23:26,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1592380.0, ans=0.125
2023-11-21 17:23:37,129 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:23:46,371 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 7.924e+01 8.815e+01 9.464e+01 2.008e+02, threshold=1.763e+02, percent-clipped=1.0
2023-11-21 17:23:59,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1592513.3333333333, ans=0.125
2023-11-21 17:23:59,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1592513.3333333333, ans=0.125
2023-11-21 17:24:14,051 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10450, loss[loss=0.07275, simple_loss=0.0975, pruned_loss=0.01376, audio_tagging_loss=0.01024, over 15575.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09534, pruned_loss=0.01618, audio_tagging_loss=0.009795, over 3052996.62 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:24:15,342 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238900
2023-11-21 17:24:17,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=14.27 vs. limit=15.0
2023-11-21 17:24:31,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1592713.3333333333, ans=0.2
2023-11-21 17:25:16,690 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10500, loss[loss=0.08314, simple_loss=0.1066, pruned_loss=0.02182, audio_tagging_loss=0.008002, over 13635.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09418, pruned_loss=0.01597, audio_tagging_loss=0.009713, over 3050206.40 frames. ], batch size: 54, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:25:17,997 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 238950
2023-11-21 17:25:32,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1593046.6666666667, ans=0.125
2023-11-21 17:25:33,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1593046.6666666667, ans=0.0
2023-11-21 17:25:35,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1593046.6666666667, ans=0.125
2023-11-21 17:25:37,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1593046.6666666667, ans=0.125
2023-11-21 17:25:48,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1593113.3333333333, ans=0.2
2023-11-21 17:25:54,793 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 7.981e+01 8.666e+01 9.446e+01 1.198e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 17:25:57,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1593180.0, ans=0.125
2023-11-21 17:26:04,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1593180.0, ans=0.125
2023-11-21 17:26:12,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1593246.6666666667, ans=0.0
2023-11-21 17:26:18,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1593246.6666666667, ans=0.125
2023-11-21 17:26:22,630 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10550, loss[loss=0.07277, simple_loss=0.09609, pruned_loss=0.01415, audio_tagging_loss=0.01058, over 15420.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.0942, pruned_loss=0.01614, audio_tagging_loss=0.009501, over 3048185.52 frames. ], batch size: 61, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:26:23,904 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239000
2023-11-21 17:26:39,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1593380.0, ans=0.2
2023-11-21 17:27:00,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1593513.3333333333, ans=0.0
2023-11-21 17:27:19,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1593580.0, ans=0.125
2023-11-21 17:27:26,938 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10600, loss[loss=0.08003, simple_loss=0.1112, pruned_loss=0.01636, audio_tagging_loss=0.00805, over 16040.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09451, pruned_loss=0.0161, audio_tagging_loss=0.009457, over 3041451.46 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:27:28,360 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239050
2023-11-21 17:27:32,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1593646.6666666667, ans=0.125
2023-11-21 17:27:42,676 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.04 vs. limit=15.0
2023-11-21 17:28:04,042 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.078e+01 8.229e+01 8.607e+01 9.453e+01 1.327e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 17:28:30,409 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10650, loss[loss=0.06873, simple_loss=0.09168, pruned_loss=0.01403, audio_tagging_loss=0.008862, over 16760.00 frames. ], tot_loss[loss=0.07276, simple_loss=0.09439, pruned_loss=0.01604, audio_tagging_loss=0.009523, over 3047976.63 frames. ], batch size: 65, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:28:30,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1593980.0, ans=0.125
2023-11-21 17:28:31,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239100
2023-11-21 17:28:37,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1593980.0, ans=0.125
2023-11-21 17:28:39,512 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.56 vs. limit=15.0
2023-11-21 17:28:52,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1594046.6666666667, ans=0.1
2023-11-21 17:29:05,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1594113.3333333333, ans=0.07
2023-11-21 17:29:14,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1594180.0, ans=0.125
2023-11-21 17:29:16,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1594180.0, ans=0.1
2023-11-21 17:29:25,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1594246.6666666667, ans=0.0
2023-11-21 17:29:34,680 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10700, loss[loss=0.07687, simple_loss=0.1052, pruned_loss=0.01755, audio_tagging_loss=0.006751, over 14848.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09448, pruned_loss=0.01607, audio_tagging_loss=0.009498, over 3044986.03 frames. ], batch size: 55, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:29:35,991 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239150
2023-11-21 17:29:42,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.89 vs. limit=15.0
2023-11-21 17:29:48,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1594380.0, ans=0.125
2023-11-21 17:29:56,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1594380.0, ans=0.0
2023-11-21 17:30:01,835 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:30:11,209 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.705e+01 8.191e+01 8.750e+01 9.714e+01 1.316e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 17:30:29,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1594580.0, ans=0.0
2023-11-21 17:30:31,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1594580.0, ans=0.125
2023-11-21 17:30:33,487 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:30:39,190 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10750, loss[loss=0.06411, simple_loss=0.08211, pruned_loss=0.01212, audio_tagging_loss=0.01093, over 15393.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.0953, pruned_loss=0.01623, audio_tagging_loss=0.009397, over 3042043.11 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:30:40,530 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239200
2023-11-21 17:30:54,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1594713.3333333333, ans=0.125
2023-11-21 17:31:03,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1594780.0, ans=15.0
2023-11-21 17:31:43,359 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10800, loss[loss=0.08501, simple_loss=0.1073, pruned_loss=0.02362, audio_tagging_loss=0.007735, over 15139.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09541, pruned_loss=0.0162, audio_tagging_loss=0.009462, over 3044105.43 frames. ], batch size: 59, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:31:44,697 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239250
2023-11-21 17:31:47,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1594980.0, ans=0.125
2023-11-21 17:32:01,273 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=7.68 vs. limit=12.0
2023-11-21 17:32:14,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1595113.3333333333, ans=0.0
2023-11-21 17:32:21,237 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.353e+01 8.126e+01 8.721e+01 9.296e+01 1.155e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 17:32:21,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1595180.0, ans=0.05
2023-11-21 17:32:27,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1595180.0, ans=0.125
2023-11-21 17:32:36,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1595246.6666666667, ans=0.2
2023-11-21 17:32:48,047 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10850, loss[loss=0.07635, simple_loss=0.1014, pruned_loss=0.01666, audio_tagging_loss=0.009015, over 15088.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09569, pruned_loss=0.01616, audio_tagging_loss=0.009516, over 3048921.89 frames. ], batch size: 57, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:32:49,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239300
2023-11-21 17:32:50,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1595313.3333333333, ans=0.1
2023-11-21 17:33:02,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1595380.0, ans=0.0
2023-11-21 17:33:09,333 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.67 vs. limit=15.0
2023-11-21 17:33:21,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1595446.6666666667, ans=0.125
2023-11-21 17:33:26,305 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.47 vs. limit=15.0
2023-11-21 17:33:33,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1595513.3333333333, ans=0.1
2023-11-21 17:33:39,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1595580.0, ans=0.125
2023-11-21 17:33:45,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1595580.0, ans=0.0
2023-11-21 17:33:46,328 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:33:51,757 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10900, loss[loss=0.06929, simple_loss=0.08616, pruned_loss=0.01502, audio_tagging_loss=0.01119, over 15446.00 frames. ], tot_loss[loss=0.07379, simple_loss=0.09582, pruned_loss=0.0163, audio_tagging_loss=0.00958, over 3049384.54 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:33:53,724 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239350
2023-11-21 17:33:54,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1595646.6666666667, ans=0.0
2023-11-21 17:34:04,312 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.45 vs. limit=15.0
2023-11-21 17:34:08,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.15 vs. limit=15.0
2023-11-21 17:34:11,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.34 vs. limit=10.0
2023-11-21 17:34:21,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.10 vs. limit=15.0
2023-11-21 17:34:28,929 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.501e+01 8.126e+01 8.588e+01 9.275e+01 1.184e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-21 17:34:34,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1595846.6666666667, ans=0.2
2023-11-21 17:34:46,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1595913.3333333333, ans=0.125
2023-11-21 17:34:49,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1595913.3333333333, ans=0.1
2023-11-21 17:34:50,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1595913.3333333333, ans=0.1
2023-11-21 17:34:52,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1595913.3333333333, ans=0.125
2023-11-21 17:34:56,180 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 10950, loss[loss=0.0792, simple_loss=0.1068, pruned_loss=0.01793, audio_tagging_loss=0.007865, over 15009.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09552, pruned_loss=0.01619, audio_tagging_loss=0.009676, over 3045007.39 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:34:57,556 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239400
2023-11-21 17:35:03,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1595980.0, ans=0.07
2023-11-21 17:35:04,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1595980.0, ans=0.2
2023-11-21 17:35:19,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1596046.6666666667, ans=0.125
2023-11-21 17:35:24,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1596113.3333333333, ans=0.2
2023-11-21 17:35:31,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1596113.3333333333, ans=0.125
2023-11-21 17:35:34,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1596180.0, ans=0.125
2023-11-21 17:35:37,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1596180.0, ans=0.125
2023-11-21 17:35:42,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1596180.0, ans=0.125
2023-11-21 17:36:00,730 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11000, loss[loss=0.06459, simple_loss=0.07775, pruned_loss=0.01205, audio_tagging_loss=0.01367, over 15672.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09549, pruned_loss=0.01609, audio_tagging_loss=0.00975, over 3052724.15 frames. ], batch size: 60, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:36:01,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1596313.3333333333, ans=0.05
2023-11-21 17:36:02,725 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239450
2023-11-21 17:36:05,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1596313.3333333333, ans=0.2
2023-11-21 17:36:11,851 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:36:30,225 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.49 vs. limit=10.0
2023-11-21 17:36:37,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1596446.6666666667, ans=0.125
2023-11-21 17:36:38,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1596446.6666666667, ans=0.125
2023-11-21 17:36:40,588 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.031e+01 8.533e+01 9.347e+01 2.206e+02, threshold=1.707e+02, percent-clipped=1.0
2023-11-21 17:36:46,368 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.11 vs. limit=15.0
2023-11-21 17:36:49,761 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.49 vs. limit=15.0
2023-11-21 17:36:55,867 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=20.65 vs. limit=15.0
2023-11-21 17:37:06,719 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11050, loss[loss=0.08837, simple_loss=0.1101, pruned_loss=0.02462, audio_tagging_loss=0.008724, over 13790.00 frames. ], tot_loss[loss=0.07398, simple_loss=0.09598, pruned_loss=0.01624, audio_tagging_loss=0.00975, over 3052045.31 frames. ], batch size: 52, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:37:08,074 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239500
2023-11-21 17:37:15,636 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:37:25,207 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.19 vs. limit=15.0
2023-11-21 17:37:34,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1596780.0, ans=0.125
2023-11-21 17:37:51,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1596846.6666666667, ans=0.0
2023-11-21 17:38:11,230 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11100, loss[loss=0.06011, simple_loss=0.08198, pruned_loss=0.009132, audio_tagging_loss=0.009988, over 14086.00 frames. ], tot_loss[loss=0.07398, simple_loss=0.09589, pruned_loss=0.01619, audio_tagging_loss=0.009847, over 3060334.80 frames. ], batch size: 53, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:38:12,536 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239550
2023-11-21 17:38:24,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1597046.6666666667, ans=0.125
2023-11-21 17:38:41,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1597113.3333333333, ans=0.125
2023-11-21 17:38:46,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1597113.3333333333, ans=0.1
2023-11-21 17:38:49,482 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.177e+01 8.722e+01 9.312e+01 1.121e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 17:38:50,056 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.14 vs. limit=22.5
2023-11-21 17:38:53,787 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.16 vs. limit=15.0
2023-11-21 17:39:04,428 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.37 vs. limit=15.0
2023-11-21 17:39:08,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1597246.6666666667, ans=0.04949747468305833
2023-11-21 17:39:14,660 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11150, loss[loss=0.06008, simple_loss=0.07572, pruned_loss=0.01063, audio_tagging_loss=0.0116, over 14911.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09558, pruned_loss=0.01619, audio_tagging_loss=0.009924, over 3061954.46 frames. ], batch size: 55, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:39:15,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239600
2023-11-21 17:39:18,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1597313.3333333333, ans=0.0
2023-11-21 17:39:21,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1597313.3333333333, ans=0.0
2023-11-21 17:39:43,776 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:39:50,767 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.49 vs. limit=15.0
2023-11-21 17:39:54,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1597513.3333333333, ans=0.125
2023-11-21 17:40:05,385 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.73 vs. limit=15.0
2023-11-21 17:40:19,210 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11200, loss[loss=0.07465, simple_loss=0.093, pruned_loss=0.01795, audio_tagging_loss=0.0102, over 14555.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09431, pruned_loss=0.01612, audio_tagging_loss=0.009991, over 3052328.86 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:40:20,510 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239650
2023-11-21 17:40:24,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff2.min_abs, batch_count=1597646.6666666667, ans=0.1
2023-11-21 17:40:58,062 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.528e+01 8.115e+01 8.678e+01 9.817e+01 1.232e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 17:41:22,808 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.39 vs. limit=22.5
2023-11-21 17:41:23,466 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11250, loss[loss=0.07795, simple_loss=0.1079, pruned_loss=0.01498, audio_tagging_loss=0.009024, over 16268.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09503, pruned_loss=0.01624, audio_tagging_loss=0.009952, over 3050684.98 frames. ], batch size: 62, lr: 3.39e-03, grad_scale: 32.0
2023-11-21 17:41:24,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239700
2023-11-21 17:41:29,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1597980.0, ans=0.0
2023-11-21 17:41:32,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1597980.0, ans=0.1
2023-11-21 17:42:18,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1598246.6666666667, ans=0.125
2023-11-21 17:42:27,193 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11300, loss[loss=0.06979, simple_loss=0.08751, pruned_loss=0.01538, audio_tagging_loss=0.01066, over 14606.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09548, pruned_loss=0.0163, audio_tagging_loss=0.009768, over 3052945.23 frames. ], batch size: 56, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:42:28,508 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239750
2023-11-21 17:43:03,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1598513.3333333333, ans=0.125
2023-11-21 17:43:06,115 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.360e+01 8.263e+01 8.887e+01 9.511e+01 1.252e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 17:43:27,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1598580.0, ans=0.1
2023-11-21 17:43:30,682 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11350, loss[loss=0.06901, simple_loss=0.08972, pruned_loss=0.01568, audio_tagging_loss=0.008466, over 14563.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09543, pruned_loss=0.01615, audio_tagging_loss=0.009641, over 3042238.51 frames. ], batch size: 55, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:43:31,983 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239800
2023-11-21 17:43:32,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1598646.6666666667, ans=0.125
2023-11-21 17:43:57,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1598780.0, ans=0.125
2023-11-21 17:43:57,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1598780.0, ans=0.125
2023-11-21 17:44:33,777 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11400, loss[loss=0.06017, simple_loss=0.07279, pruned_loss=0.01355, audio_tagging_loss=0.01023, over 14148.00 frames. ], tot_loss[loss=0.07303, simple_loss=0.09476, pruned_loss=0.01607, audio_tagging_loss=0.009572, over 3041051.72 frames. ], batch size: 54, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:44:35,095 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239850
2023-11-21 17:44:35,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=1598980.0, ans=15.0
2023-11-21 17:45:12,697 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.120e+01 8.500e+01 9.528e+01 1.320e+02, threshold=1.700e+02, percent-clipped=0.0
2023-11-21 17:45:30,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1599246.6666666667, ans=0.0
2023-11-21 17:45:37,186 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11450, loss[loss=0.1053, simple_loss=0.1511, pruned_loss=0.02605, audio_tagging_loss=0.003729, over 14244.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09461, pruned_loss=0.01615, audio_tagging_loss=0.00954, over 3042245.79 frames. ], batch size: 53, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:45:38,536 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239900
2023-11-21 17:45:59,054 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.05 vs. limit=15.0
2023-11-21 17:46:08,206 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.03 vs. limit=15.0
2023-11-21 17:46:16,666 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.20 vs. limit=15.0
2023-11-21 17:46:18,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1599513.3333333333, ans=0.125
2023-11-21 17:46:21,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1599513.3333333333, ans=0.0
2023-11-21 17:46:39,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1599646.6666666667, ans=0.1
2023-11-21 17:46:40,480 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11500, loss[loss=0.07712, simple_loss=0.09908, pruned_loss=0.01401, audio_tagging_loss=0.01357, over 15091.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09515, pruned_loss=0.01627, audio_tagging_loss=0.00954, over 3051250.11 frames. ], batch size: 58, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:46:40,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1599646.6666666667, ans=0.125
2023-11-21 17:46:41,770 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 239950
2023-11-21 17:46:52,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1599713.3333333333, ans=0.0
2023-11-21 17:47:07,289 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.35 vs. limit=6.0
2023-11-21 17:47:13,228 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.76 vs. limit=15.0
2023-11-21 17:47:19,135 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 7.879e+01 8.671e+01 9.205e+01 1.101e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 17:47:38,341 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.71 vs. limit=22.5
2023-11-21 17:47:41,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1599913.3333333333, ans=0.0
2023-11-21 17:47:43,799 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11550, loss[loss=0.07875, simple_loss=0.09576, pruned_loss=0.02032, audio_tagging_loss=0.01055, over 16194.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09503, pruned_loss=0.01642, audio_tagging_loss=0.009519, over 3053649.97 frames. ], batch size: 62, lr: 3.39e-03, grad_scale: 16.0
2023-11-21 17:47:45,081 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240000
2023-11-21 17:47:46,614 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-240000.pt
2023-11-21 17:47:50,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1599980.0, ans=0.07
2023-11-21 17:48:10,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1600046.6666666667, ans=0.0
2023-11-21 17:48:14,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1600113.3333333333, ans=0.125
2023-11-21 17:48:24,272 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 17:48:26,187 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.48 vs. limit=6.0
2023-11-21 17:48:31,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1600180.0, ans=0.125
2023-11-21 17:48:38,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1600246.6666666667, ans=0.015
2023-11-21 17:48:48,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1600246.6666666667, ans=0.0
2023-11-21 17:48:50,298 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11600, loss[loss=0.04836, simple_loss=0.05909, pruned_loss=0.00938, audio_tagging_loss=0.009435, over 15733.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09524, pruned_loss=0.01627, audio_tagging_loss=0.009564, over 3053929.63 frames. ], batch size: 62, lr: 3.38e-03, grad_scale: 32.0
2023-11-21 17:48:50,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1600313.3333333333, ans=0.2
2023-11-21 17:48:51,692 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240050
2023-11-21 17:48:51,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1600313.3333333333, ans=0.125
2023-11-21 17:48:54,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1600313.3333333333, ans=0.0
2023-11-21 17:49:05,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1600380.0, ans=0.125
2023-11-21 17:49:11,690 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.66 vs. limit=12.0
2023-11-21 17:49:11,762 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.02 vs. limit=10.0
2023-11-21 17:49:21,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1600446.6666666667, ans=0.125
2023-11-21 17:49:29,604 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.655e+01 8.173e+01 8.910e+01 9.625e+01 1.220e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-21 17:49:54,107 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11650, loss[loss=0.07559, simple_loss=0.09082, pruned_loss=0.02018, audio_tagging_loss=0.009999, over 14672.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09542, pruned_loss=0.01633, audio_tagging_loss=0.009507, over 3053811.41 frames. ], batch size: 57, lr: 3.38e-03, grad_scale: 32.0
2023-11-21 17:49:55,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240100
2023-11-21 17:49:56,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1600646.6666666667, ans=0.125
2023-11-21 17:50:23,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1600780.0, ans=0.125
2023-11-21 17:50:27,120 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.23 vs. limit=15.0
2023-11-21 17:50:28,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.00 vs. limit=12.0
2023-11-21 17:50:32,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1600846.6666666667, ans=0.2
2023-11-21 17:50:44,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1600913.3333333333, ans=0.125
2023-11-21 17:50:49,418 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.91 vs. limit=22.5
2023-11-21 17:50:50,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1600913.3333333333, ans=0.125
2023-11-21 17:50:54,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1600913.3333333333, ans=0.125
2023-11-21 17:50:57,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1600980.0, ans=22.5
2023-11-21 17:50:58,002 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11700, loss[loss=0.08759, simple_loss=0.1212, pruned_loss=0.02082, audio_tagging_loss=0.006181, over 14509.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09445, pruned_loss=0.0161, audio_tagging_loss=0.009567, over 3041100.54 frames. ], batch size: 53, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:50:59,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240150
2023-11-21 17:51:09,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=1601046.6666666667, ans=15.0
2023-11-21 17:51:35,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1601180.0, ans=0.125
2023-11-21 17:51:38,208 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.589e+01 8.116e+01 8.836e+01 9.594e+01 1.189e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 17:51:46,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1601180.0, ans=0.2
2023-11-21 17:51:50,508 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.91 vs. limit=15.0
2023-11-21 17:51:51,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1601246.6666666667, ans=0.125
2023-11-21 17:52:00,600 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11750, loss[loss=0.05959, simple_loss=0.06973, pruned_loss=0.01407, audio_tagging_loss=0.01066, over 14775.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09436, pruned_loss=0.01603, audio_tagging_loss=0.009567, over 3041578.10 frames. ], batch size: 58, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:52:01,903 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240200
2023-11-21 17:52:02,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1601313.3333333333, ans=0.125
2023-11-21 17:52:06,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1601313.3333333333, ans=0.0
2023-11-21 17:52:17,673 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.70 vs. limit=22.5
2023-11-21 17:52:19,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1601380.0, ans=0.125
2023-11-21 17:52:37,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1601446.6666666667, ans=0.1
2023-11-21 17:52:37,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1601446.6666666667, ans=0.125
2023-11-21 17:52:37,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.05 vs. limit=15.0
2023-11-21 17:52:41,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1601513.3333333333, ans=0.0
2023-11-21 17:53:03,894 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11800, loss[loss=0.09677, simple_loss=0.1213, pruned_loss=0.0282, audio_tagging_loss=0.007932, over 15020.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09385, pruned_loss=0.01593, audio_tagging_loss=0.009622, over 3042484.88 frames. ], batch size: 57, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:53:05,189 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240250
2023-11-21 17:53:14,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1601646.6666666667, ans=0.05
2023-11-21 17:53:16,440 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-21 17:53:28,515 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.534e-03
2023-11-21 17:53:44,076 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.430e+01 8.055e+01 8.902e+01 9.602e+01 1.321e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 17:54:07,648 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11850, loss[loss=0.09483, simple_loss=0.1188, pruned_loss=0.02668, audio_tagging_loss=0.008768, over 15679.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09479, pruned_loss=0.01605, audio_tagging_loss=0.009623, over 3044311.92 frames. ], batch size: 56, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:54:07,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1601980.0, ans=0.125
2023-11-21 17:54:08,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240300
2023-11-21 17:54:15,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1601980.0, ans=0.125
2023-11-21 17:54:25,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1602046.6666666667, ans=0.125
2023-11-21 17:54:27,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1602046.6666666667, ans=0.125
2023-11-21 17:54:32,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1602113.3333333333, ans=0.125
2023-11-21 17:54:39,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1602113.3333333333, ans=0.1
2023-11-21 17:54:59,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1602246.6666666667, ans=0.125
2023-11-21 17:54:59,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1602246.6666666667, ans=0.0
2023-11-21 17:55:08,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1602246.6666666667, ans=0.0
2023-11-21 17:55:08,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1602246.6666666667, ans=0.1
2023-11-21 17:55:11,010 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11900, loss[loss=0.08334, simple_loss=0.1178, pruned_loss=0.01741, audio_tagging_loss=0.007028, over 16125.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09547, pruned_loss=0.01608, audio_tagging_loss=0.009683, over 3043484.41 frames. ], batch size: 58, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:55:12,292 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240350
2023-11-21 17:55:14,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1602313.3333333333, ans=0.125
2023-11-21 17:55:16,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1602313.3333333333, ans=0.125
2023-11-21 17:55:30,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1602380.0, ans=0.125
2023-11-21 17:55:30,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1602380.0, ans=0.125
2023-11-21 17:55:52,197 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.678e+01 7.826e+01 8.536e+01 9.227e+01 1.162e+02, threshold=1.707e+02, percent-clipped=0.0
2023-11-21 17:55:58,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1602513.3333333333, ans=0.125
2023-11-21 17:56:05,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1602580.0, ans=0.0
2023-11-21 17:56:09,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1602580.0, ans=0.1
2023-11-21 17:56:14,818 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 11950, loss[loss=0.05947, simple_loss=0.07002, pruned_loss=0.01314, audio_tagging_loss=0.01131, over 14947.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09563, pruned_loss=0.01606, audio_tagging_loss=0.009711, over 3054032.21 frames. ], batch size: 59, lr: 3.38e-03, grad_scale: 16.0
2023-11-21 17:56:16,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240400
2023-11-21 17:56:48,863 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 17:56:58,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1602846.6666666667, ans=0.0
2023-11-21 17:57:16,248 INFO [train_asr.py:1221] (0/4) Epoch 20, batch 12000, loss[loss=0.05796, simple_loss=0.07905, pruned_loss=0.009836, audio_tagging_loss=0.008602, over 17247.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09564, pruned_loss=0.01615, audio_tagging_loss=0.009803, over 3051145.07 frames. ], batch size: 66, lr: 3.38e-03, grad_scale: 32.0
2023-11-21 17:57:16,252 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 17:57:57,979 INFO [train_asr.py:1253] (0/4) Epoch 20, validation: loss=0.06015, simple_loss=0.05214, pruned_loss=0.005246, audio_tagging_loss=0.02884, over 4681554.00 frames. 
2023-11-21 17:57:57,980 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 17:57:59,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240450
2023-11-21 17:58:01,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1602980.0, ans=0.125
2023-11-21 17:58:02,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1602980.0, ans=0.2
2023-11-21 17:58:18,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1603046.6666666667, ans=0.0
2023-11-21 17:58:22,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.31 vs. limit=15.0
2023-11-21 17:58:25,226 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-20.pt
2023-11-21 17:58:58,667 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 0, loss[loss=0.07087, simple_loss=0.08131, pruned_loss=0.009064, audio_tagging_loss=0.02115, over 15242.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.08131, pruned_loss=0.009064, audio_tagging_loss=0.02115, over 15242.00 frames. ], batch size: 58, lr: 3.30e-03, grad_scale: 32.0
2023-11-21 17:58:58,670 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 17:59:34,264 INFO [train_asr.py:1253] (0/4) Epoch 21, validation: loss=0.05942, simple_loss=0.05208, pruned_loss=0.00519, audio_tagging_loss=0.02819, over 4681554.00 frames. 
2023-11-21 17:59:34,264 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 17:59:46,184 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.163e+01 8.736e+01 1.018e+02 1.443e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 18:00:11,023 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240500
2023-11-21 18:00:16,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1603333.3333333333, ans=0.0
2023-11-21 18:00:25,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1603400.0, ans=0.2
2023-11-21 18:00:28,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1603400.0, ans=0.0
2023-11-21 18:00:33,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1603400.0, ans=0.0
2023-11-21 18:00:33,894 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.92 vs. limit=22.5
2023-11-21 18:00:39,134 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 50, loss[loss=0.08681, simple_loss=0.1045, pruned_loss=0.0186, audio_tagging_loss=0.01596, over 15795.00 frames. ], tot_loss[loss=0.08055, simple_loss=0.09314, pruned_loss=0.0156, audio_tagging_loss=0.01838, over 692702.57 frames. ], batch size: 58, lr: 3.30e-03, grad_scale: 32.0
2023-11-21 18:00:49,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1603466.6666666667, ans=0.125
2023-11-21 18:00:54,709 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.04 vs. limit=15.0
2023-11-21 18:00:56,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1603533.3333333333, ans=0.125
2023-11-21 18:01:03,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1603533.3333333333, ans=0.1
2023-11-21 18:01:15,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240550
2023-11-21 18:01:43,929 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 100, loss[loss=0.09916, simple_loss=0.1229, pruned_loss=0.02399, audio_tagging_loss=0.01371, over 14710.00 frames. ], tot_loss[loss=0.08076, simple_loss=0.09378, pruned_loss=0.01605, audio_tagging_loss=0.01782, over 1211101.95 frames. ], batch size: 56, lr: 3.30e-03, grad_scale: 32.0
2023-11-21 18:01:54,764 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.709e+01 8.663e+01 9.262e+01 1.008e+02 1.245e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-21 18:02:14,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1603933.3333333333, ans=0.1
2023-11-21 18:02:19,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240600
2023-11-21 18:02:26,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1604000.0, ans=0.07
2023-11-21 18:02:30,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1604000.0, ans=0.125
2023-11-21 18:02:47,888 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 150, loss[loss=0.07581, simple_loss=0.0977, pruned_loss=0.01446, audio_tagging_loss=0.0125, over 15308.00 frames. ], tot_loss[loss=0.0801, simple_loss=0.09547, pruned_loss=0.01642, audio_tagging_loss=0.01594, over 1619070.88 frames. ], batch size: 56, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:02:55,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1604133.3333333333, ans=0.125
2023-11-21 18:02:55,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1604133.3333333333, ans=0.2
2023-11-21 18:03:04,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1604200.0, ans=0.0
2023-11-21 18:03:14,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1604266.6666666667, ans=0.035
2023-11-21 18:03:21,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1604266.6666666667, ans=0.125
2023-11-21 18:03:24,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240650
2023-11-21 18:03:32,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1604333.3333333333, ans=0.0
2023-11-21 18:03:36,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1604333.3333333333, ans=0.0
2023-11-21 18:03:42,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1604400.0, ans=0.1
2023-11-21 18:03:52,002 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 200, loss[loss=0.08079, simple_loss=0.1017, pruned_loss=0.01736, audio_tagging_loss=0.01258, over 15067.00 frames. ], tot_loss[loss=0.07897, simple_loss=0.09669, pruned_loss=0.01663, audio_tagging_loss=0.01399, over 1933281.83 frames. ], batch size: 55, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:03:54,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1604466.6666666667, ans=0.125
2023-11-21 18:03:59,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1604466.6666666667, ans=0.05
2023-11-21 18:04:03,960 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.168e+01 8.831e+01 9.644e+01 1.295e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-21 18:04:27,167 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240700
2023-11-21 18:04:31,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1604666.6666666667, ans=0.125
2023-11-21 18:04:44,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1604733.3333333333, ans=0.125
2023-11-21 18:04:50,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1604733.3333333333, ans=0.5
2023-11-21 18:04:52,220 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.52 vs. limit=15.0
2023-11-21 18:04:55,611 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 250, loss[loss=0.05877, simple_loss=0.07047, pruned_loss=0.01411, audio_tagging_loss=0.009431, over 15265.00 frames. ], tot_loss[loss=0.07758, simple_loss=0.09682, pruned_loss=0.01651, audio_tagging_loss=0.01265, over 2181717.13 frames. ], batch size: 58, lr: 3.30e-03, grad_scale: 8.0
2023-11-21 18:05:00,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1604800.0, ans=0.125
2023-11-21 18:05:06,231 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:05:16,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.00 vs. limit=10.0
2023-11-21 18:05:30,860 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.96 vs. limit=15.0
2023-11-21 18:05:31,482 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240750
2023-11-21 18:05:33,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.29 vs. limit=22.5
2023-11-21 18:05:39,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.02 vs. limit=22.5
2023-11-21 18:05:59,427 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 300, loss[loss=0.05813, simple_loss=0.07464, pruned_loss=0.01206, audio_tagging_loss=0.008747, over 14917.00 frames. ], tot_loss[loss=0.07681, simple_loss=0.09701, pruned_loss=0.01658, audio_tagging_loss=0.01173, over 2379834.73 frames. ], batch size: 56, lr: 3.30e-03, grad_scale: 8.0
2023-11-21 18:06:10,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1605133.3333333333, ans=0.07
2023-11-21 18:06:14,089 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.194e+01 8.148e+01 8.758e+01 9.398e+01 1.354e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 18:06:35,642 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240800
2023-11-21 18:06:35,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1605266.6666666667, ans=0.125
2023-11-21 18:06:45,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1605333.3333333333, ans=0.0
2023-11-21 18:06:47,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1605333.3333333333, ans=0.2
2023-11-21 18:07:04,339 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 350, loss[loss=0.08527, simple_loss=0.1262, pruned_loss=0.0161, audio_tagging_loss=0.006084, over 14634.00 frames. ], tot_loss[loss=0.07648, simple_loss=0.09741, pruned_loss=0.01666, audio_tagging_loss=0.01112, over 2528761.20 frames. ], batch size: 53, lr: 3.30e-03, grad_scale: 8.0
2023-11-21 18:07:05,243 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.01 vs. limit=15.0
2023-11-21 18:07:11,337 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.10 vs. limit=22.5
2023-11-21 18:07:27,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1605533.3333333333, ans=0.125
2023-11-21 18:07:29,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1605600.0, ans=0.125
2023-11-21 18:07:31,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1605600.0, ans=0.125
2023-11-21 18:07:39,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240850
2023-11-21 18:07:46,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1605666.6666666667, ans=0.0
2023-11-21 18:07:55,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1605733.3333333333, ans=0.125
2023-11-21 18:07:56,415 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.39 vs. limit=15.0
2023-11-21 18:07:58,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1605733.3333333333, ans=0.1
2023-11-21 18:08:08,119 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 400, loss[loss=0.08875, simple_loss=0.122, pruned_loss=0.02021, audio_tagging_loss=0.007558, over 14959.00 frames. ], tot_loss[loss=0.07591, simple_loss=0.09745, pruned_loss=0.0165, audio_tagging_loss=0.01068, over 2636519.84 frames. ], batch size: 54, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:08:10,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1605800.0, ans=0.0
2023-11-21 18:08:22,787 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.012e+01 8.744e+01 9.405e+01 1.302e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-21 18:08:29,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1605866.6666666667, ans=0.5
2023-11-21 18:08:44,701 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240900
2023-11-21 18:08:48,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1606000.0, ans=0.125
2023-11-21 18:09:02,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1606066.6666666667, ans=0.125
2023-11-21 18:09:07,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1606066.6666666667, ans=0.125
2023-11-21 18:09:08,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1606066.6666666667, ans=0.2
2023-11-21 18:09:12,850 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 450, loss[loss=0.08079, simple_loss=0.1098, pruned_loss=0.02019, audio_tagging_loss=0.005694, over 15552.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09587, pruned_loss=0.01629, audio_tagging_loss=0.01043, over 2726863.12 frames. ], batch size: 57, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:09:27,459 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.93 vs. limit=15.0
2023-11-21 18:09:34,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1606200.0, ans=0.0
2023-11-21 18:09:40,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1606266.6666666667, ans=0.125
2023-11-21 18:09:49,210 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 240950
2023-11-21 18:09:51,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1606333.3333333333, ans=0.125
2023-11-21 18:10:01,513 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.99 vs. limit=22.5
2023-11-21 18:10:07,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1606400.0, ans=0.1
2023-11-21 18:10:17,057 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 500, loss[loss=0.06843, simple_loss=0.08749, pruned_loss=0.0139, audio_tagging_loss=0.01078, over 15646.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09518, pruned_loss=0.01624, audio_tagging_loss=0.01016, over 2793689.36 frames. ], batch size: 61, lr: 3.30e-03, grad_scale: 16.0
2023-11-21 18:10:17,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1606466.6666666667, ans=0.1
2023-11-21 18:10:31,502 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.140e+01 8.109e+01 8.628e+01 9.380e+01 1.335e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-21 18:10:43,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1606600.0, ans=0.0
2023-11-21 18:10:45,211 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.06 vs. limit=22.5
2023-11-21 18:10:53,703 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241000
2023-11-21 18:11:04,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1606666.6666666667, ans=0.0
2023-11-21 18:11:12,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1606733.3333333333, ans=0.125
2023-11-21 18:11:12,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1606733.3333333333, ans=0.125
2023-11-21 18:11:20,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1606733.3333333333, ans=0.0
2023-11-21 18:11:22,075 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 550, loss[loss=0.07587, simple_loss=0.1063, pruned_loss=0.01509, audio_tagging_loss=0.007637, over 16055.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.09595, pruned_loss=0.01639, audio_tagging_loss=0.01003, over 2848482.40 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:11:53,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1606933.3333333333, ans=0.1
2023-11-21 18:11:58,352 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241050
2023-11-21 18:12:16,710 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.96 vs. limit=15.0
2023-11-21 18:12:19,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1607066.6666666667, ans=0.0
2023-11-21 18:12:22,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1607066.6666666667, ans=0.1
2023-11-21 18:12:25,554 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 600, loss[loss=0.05687, simple_loss=0.06058, pruned_loss=0.01333, audio_tagging_loss=0.01326, over 14715.00 frames. ], tot_loss[loss=0.07329, simple_loss=0.09436, pruned_loss=0.01613, audio_tagging_loss=0.00998, over 2881414.34 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:12:39,622 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.804e+01 8.139e+01 8.699e+01 9.445e+01 1.159e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 18:13:01,883 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241100
2023-11-21 18:13:04,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1607333.3333333333, ans=0.07
2023-11-21 18:13:30,507 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 650, loss[loss=0.08045, simple_loss=0.1077, pruned_loss=0.01817, audio_tagging_loss=0.008433, over 16547.00 frames. ], tot_loss[loss=0.07383, simple_loss=0.09519, pruned_loss=0.01625, audio_tagging_loss=0.009979, over 2922741.19 frames. ], batch size: 59, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:13:48,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1607533.3333333333, ans=0.125
2023-11-21 18:13:49,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1607533.3333333333, ans=0.125
2023-11-21 18:13:58,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1607600.0, ans=0.125
2023-11-21 18:14:05,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241150
2023-11-21 18:14:05,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1607600.0, ans=0.0
2023-11-21 18:14:10,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1607666.6666666667, ans=0.0
2023-11-21 18:14:21,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1607733.3333333333, ans=0.1
2023-11-21 18:14:21,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1607733.3333333333, ans=0.125
2023-11-21 18:14:34,264 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 700, loss[loss=0.08057, simple_loss=0.1012, pruned_loss=0.02147, audio_tagging_loss=0.00849, over 15154.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09618, pruned_loss=0.01626, audio_tagging_loss=0.009771, over 2958705.44 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:14:38,443 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.42 vs. limit=15.0
2023-11-21 18:14:39,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1607800.0, ans=0.05
2023-11-21 18:14:47,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.94 vs. limit=12.0
2023-11-21 18:14:48,427 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.012e+01 8.519e+01 9.265e+01 1.175e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-21 18:15:04,519 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.76 vs. limit=15.0
2023-11-21 18:15:10,846 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241200
2023-11-21 18:15:11,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1607933.3333333333, ans=0.125
2023-11-21 18:15:22,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1608000.0, ans=0.125
2023-11-21 18:15:39,229 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 750, loss[loss=0.06372, simple_loss=0.07288, pruned_loss=0.01172, audio_tagging_loss=0.01555, over 14583.00 frames. ], tot_loss[loss=0.0742, simple_loss=0.09625, pruned_loss=0.01635, audio_tagging_loss=0.009731, over 2971585.68 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:15:49,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1608133.3333333333, ans=0.125
2023-11-21 18:16:06,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1608266.6666666667, ans=0.125
2023-11-21 18:16:14,938 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241250
2023-11-21 18:16:42,245 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 800, loss[loss=0.0788, simple_loss=0.1072, pruned_loss=0.01577, audio_tagging_loss=0.009441, over 14019.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09651, pruned_loss=0.01643, audio_tagging_loss=0.009851, over 2991909.25 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:16:55,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.36 vs. limit=15.0
2023-11-21 18:16:57,482 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.159e+01 8.981e+01 9.751e+01 1.353e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-21 18:17:18,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241300
2023-11-21 18:17:21,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1608666.6666666667, ans=0.125
2023-11-21 18:17:23,347 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.21 vs. limit=15.0
2023-11-21 18:17:26,669 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:17:28,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1608666.6666666667, ans=0.125
2023-11-21 18:17:31,932 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.27 vs. limit=10.0
2023-11-21 18:17:36,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1608733.3333333333, ans=0.125
2023-11-21 18:17:47,499 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 850, loss[loss=0.07765, simple_loss=0.106, pruned_loss=0.0171, audio_tagging_loss=0.007554, over 15552.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09573, pruned_loss=0.01632, audio_tagging_loss=0.009944, over 3001268.70 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:17:51,768 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.83 vs. limit=15.0
2023-11-21 18:18:05,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1608866.6666666667, ans=0.035
2023-11-21 18:18:18,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1608933.3333333333, ans=0.1
2023-11-21 18:18:23,365 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241350
2023-11-21 18:18:24,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1609000.0, ans=0.1
2023-11-21 18:18:31,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1609000.0, ans=0.125
2023-11-21 18:18:48,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1609066.6666666667, ans=0.05
2023-11-21 18:18:52,225 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 900, loss[loss=0.07096, simple_loss=0.08542, pruned_loss=0.01732, audio_tagging_loss=0.01093, over 15923.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09528, pruned_loss=0.01632, audio_tagging_loss=0.009918, over 3011238.93 frames. ], batch size: 60, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:18:58,930 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.57 vs. limit=22.5
2023-11-21 18:19:04,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1609200.0, ans=0.5
2023-11-21 18:19:05,367 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.985e+01 8.217e+01 8.787e+01 9.484e+01 1.241e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 18:19:08,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.69 vs. limit=15.0
2023-11-21 18:19:27,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241400
2023-11-21 18:19:52,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1609400.0, ans=0.125
2023-11-21 18:19:55,359 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 950, loss[loss=0.04876, simple_loss=0.06223, pruned_loss=0.007922, audio_tagging_loss=0.009726, over 13371.00 frames. ], tot_loss[loss=0.07455, simple_loss=0.0967, pruned_loss=0.01649, audio_tagging_loss=0.00972, over 3025594.03 frames. ], batch size: 53, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:20:23,746 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.13 vs. limit=22.5
2023-11-21 18:20:24,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1609600.0, ans=0.0
2023-11-21 18:20:31,534 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241450
2023-11-21 18:20:35,475 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:20:49,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1609733.3333333333, ans=0.0
2023-11-21 18:20:53,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1609733.3333333333, ans=0.1
2023-11-21 18:21:00,405 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1000, loss[loss=0.0813, simple_loss=0.1012, pruned_loss=0.02047, audio_tagging_loss=0.01024, over 15647.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09578, pruned_loss=0.01637, audio_tagging_loss=0.009612, over 3027493.81 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:21:10,676 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.42 vs. limit=22.5
2023-11-21 18:21:15,641 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.205e+01 8.818e+01 9.430e+01 1.136e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 18:21:26,815 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:21:32,601 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.49 vs. limit=15.0
2023-11-21 18:21:35,613 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241500
2023-11-21 18:22:01,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1610066.6666666667, ans=0.2
2023-11-21 18:22:04,694 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1050, loss[loss=0.0659, simple_loss=0.09288, pruned_loss=0.01137, audio_tagging_loss=0.008097, over 16172.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09525, pruned_loss=0.01623, audio_tagging_loss=0.009606, over 3027841.01 frames. ], batch size: 62, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:22:10,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.39 vs. limit=15.0
2023-11-21 18:22:36,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1610266.6666666667, ans=0.0
2023-11-21 18:22:40,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241550
2023-11-21 18:22:46,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1610333.3333333333, ans=0.125
2023-11-21 18:22:53,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1610333.3333333333, ans=0.2
2023-11-21 18:23:08,013 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1100, loss[loss=0.03837, simple_loss=0.04877, pruned_loss=0.005832, audio_tagging_loss=0.008156, over 14226.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09414, pruned_loss=0.01594, audio_tagging_loss=0.009566, over 3033035.05 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:23:10,540 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:23:13,867 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.49 vs. limit=15.0
2023-11-21 18:23:15,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1610466.6666666667, ans=0.1
2023-11-21 18:23:23,976 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.562e+01 7.887e+01 8.672e+01 9.418e+01 1.186e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-21 18:23:33,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1610600.0, ans=0.04949747468305833
2023-11-21 18:23:38,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1610600.0, ans=0.0
2023-11-21 18:23:44,664 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241600
2023-11-21 18:23:48,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1610666.6666666667, ans=0.125
2023-11-21 18:24:12,793 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1150, loss[loss=0.08402, simple_loss=0.1162, pruned_loss=0.01903, audio_tagging_loss=0.006896, over 16495.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09522, pruned_loss=0.01605, audio_tagging_loss=0.009501, over 3035143.77 frames. ], batch size: 61, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:24:22,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=1610800.0, ans=15.0
2023-11-21 18:24:31,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1610866.6666666667, ans=0.09899494936611666
2023-11-21 18:24:42,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1610933.3333333333, ans=0.125
2023-11-21 18:24:48,190 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241650
2023-11-21 18:25:04,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1611066.6666666667, ans=0.1
2023-11-21 18:25:17,315 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1200, loss[loss=0.07686, simple_loss=0.09038, pruned_loss=0.02176, audio_tagging_loss=0.009903, over 13892.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.09532, pruned_loss=0.01624, audio_tagging_loss=0.009444, over 3038874.07 frames. ], batch size: 53, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:25:33,027 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.048e+01 8.609e+01 9.236e+01 1.274e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-21 18:25:45,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.16 vs. limit=15.0
2023-11-21 18:25:52,514 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241700
2023-11-21 18:26:20,654 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1250, loss[loss=0.08085, simple_loss=0.1138, pruned_loss=0.01676, audio_tagging_loss=0.007182, over 15775.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09548, pruned_loss=0.01622, audio_tagging_loss=0.009396, over 3042947.75 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:26:45,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1611600.0, ans=0.1
2023-11-21 18:26:50,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1611600.0, ans=0.125
2023-11-21 18:26:56,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff3.min_abs, batch_count=1611600.0, ans=0.2
2023-11-21 18:26:56,426 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=10.61 vs. limit=12.0
2023-11-21 18:26:57,040 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241750
2023-11-21 18:27:08,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1611666.6666666667, ans=0.125
2023-11-21 18:27:15,919 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.60 vs. limit=15.0
2023-11-21 18:27:25,089 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1300, loss[loss=0.06563, simple_loss=0.08223, pruned_loss=0.01566, audio_tagging_loss=0.008849, over 14301.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09481, pruned_loss=0.01603, audio_tagging_loss=0.009508, over 3037550.73 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:27:29,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.55 vs. limit=22.5
2023-11-21 18:27:41,454 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.105e+01 8.749e+01 9.378e+01 1.268e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 18:28:00,365 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241800
2023-11-21 18:28:07,412 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:28:14,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1612000.0, ans=0.1
2023-11-21 18:28:26,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1612066.6666666667, ans=0.0
2023-11-21 18:28:28,416 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1350, loss[loss=0.09752, simple_loss=0.1266, pruned_loss=0.02543, audio_tagging_loss=0.008813, over 16387.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09548, pruned_loss=0.01618, audio_tagging_loss=0.009401, over 3041109.93 frames. ], batch size: 59, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:28:35,321 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.16 vs. limit=5.0
2023-11-21 18:28:38,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1612133.3333333333, ans=0.125
2023-11-21 18:28:44,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1612200.0, ans=0.125
2023-11-21 18:29:05,068 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241850
2023-11-21 18:29:14,746 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:29:17,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1612333.3333333333, ans=10.0
2023-11-21 18:29:32,997 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1400, loss[loss=0.0862, simple_loss=0.1156, pruned_loss=0.01885, audio_tagging_loss=0.009548, over 15461.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09577, pruned_loss=0.01621, audio_tagging_loss=0.009426, over 3043340.98 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:29:33,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1612466.6666666667, ans=10.0
2023-11-21 18:29:50,232 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.505e+01 9.180e+01 9.942e+01 1.333e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-21 18:29:56,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.whiten.whitening_limit, batch_count=1612533.3333333333, ans=12.0
2023-11-21 18:30:09,483 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241900
2023-11-21 18:30:37,671 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1450, loss[loss=0.06676, simple_loss=0.08977, pruned_loss=0.01399, audio_tagging_loss=0.00788, over 14160.00 frames. ], tot_loss[loss=0.07407, simple_loss=0.09653, pruned_loss=0.01635, audio_tagging_loss=0.009452, over 3048652.36 frames. ], batch size: 53, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:30:57,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1612866.6666666667, ans=0.125
2023-11-21 18:30:59,512 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:31:02,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1612933.3333333333, ans=0.125
2023-11-21 18:31:13,401 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 241950
2023-11-21 18:31:14,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1613000.0, ans=0.125
2023-11-21 18:31:41,557 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1500, loss[loss=0.09497, simple_loss=0.1173, pruned_loss=0.02226, audio_tagging_loss=0.01408, over 16151.00 frames. ], tot_loss[loss=0.07452, simple_loss=0.09717, pruned_loss=0.0164, audio_tagging_loss=0.009529, over 3051893.53 frames. ], batch size: 60, lr: 3.29e-03, grad_scale: 8.0
2023-11-21 18:31:42,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1613133.3333333333, ans=0.0
2023-11-21 18:31:43,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1613133.3333333333, ans=0.125
2023-11-21 18:31:44,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1613133.3333333333, ans=0.125
2023-11-21 18:31:58,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1613200.0, ans=0.125
2023-11-21 18:31:59,585 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.238e+01 7.980e+01 8.647e+01 9.591e+01 1.272e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-21 18:32:00,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1613200.0, ans=0.0
2023-11-21 18:32:17,998 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242000
2023-11-21 18:32:46,303 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1550, loss[loss=0.05525, simple_loss=0.06818, pruned_loss=0.008937, audio_tagging_loss=0.01222, over 14186.00 frames. ], tot_loss[loss=0.07462, simple_loss=0.09708, pruned_loss=0.01643, audio_tagging_loss=0.009644, over 3053221.59 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 8.0
2023-11-21 18:32:48,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1613466.6666666667, ans=10.0
2023-11-21 18:32:54,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1613466.6666666667, ans=0.125
2023-11-21 18:32:57,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1613533.3333333333, ans=0.125
2023-11-21 18:32:58,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1613533.3333333333, ans=0.1
2023-11-21 18:33:02,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1613533.3333333333, ans=0.0
2023-11-21 18:33:06,198 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.90 vs. limit=15.0
2023-11-21 18:33:09,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1613533.3333333333, ans=0.0
2023-11-21 18:33:17,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1613600.0, ans=0.125
2023-11-21 18:33:22,005 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242050
2023-11-21 18:33:50,193 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1600, loss[loss=0.09062, simple_loss=0.1214, pruned_loss=0.02059, audio_tagging_loss=0.009332, over 15769.00 frames. ], tot_loss[loss=0.0744, simple_loss=0.09674, pruned_loss=0.01639, audio_tagging_loss=0.009637, over 3048663.89 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:34:07,794 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.028e+01 8.717e+01 9.586e+01 1.995e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 18:34:26,157 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242100
2023-11-21 18:34:39,531 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.33 vs. limit=15.0
2023-11-21 18:34:54,121 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1650, loss[loss=0.06625, simple_loss=0.07822, pruned_loss=0.01566, audio_tagging_loss=0.01148, over 16541.00 frames. ], tot_loss[loss=0.07402, simple_loss=0.09605, pruned_loss=0.01634, audio_tagging_loss=0.009656, over 3055031.69 frames. ], batch size: 64, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:35:31,028 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242150
2023-11-21 18:35:46,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1614400.0, ans=0.1
2023-11-21 18:35:58,539 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1700, loss[loss=0.0829, simple_loss=0.1184, pruned_loss=0.01747, audio_tagging_loss=0.006219, over 16711.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.09634, pruned_loss=0.0164, audio_tagging_loss=0.009614, over 3051629.49 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:36:00,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1614466.6666666667, ans=0.125
2023-11-21 18:36:10,568 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.58 vs. limit=15.0
2023-11-21 18:36:12,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1614533.3333333333, ans=0.1
2023-11-21 18:36:13,043 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.70 vs. limit=15.0
2023-11-21 18:36:16,624 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.384e+01 8.955e+01 9.684e+01 1.332e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-21 18:36:16,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1614533.3333333333, ans=0.09899494936611666
2023-11-21 18:36:34,505 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242200
2023-11-21 18:36:37,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1614666.6666666667, ans=0.125
2023-11-21 18:36:45,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1614666.6666666667, ans=0.125
2023-11-21 18:36:48,205 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.25 vs. limit=15.0
2023-11-21 18:36:55,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1614733.3333333333, ans=0.95
2023-11-21 18:37:02,899 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1750, loss[loss=0.05133, simple_loss=0.06467, pruned_loss=0.01076, audio_tagging_loss=0.008242, over 14302.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09513, pruned_loss=0.01599, audio_tagging_loss=0.009613, over 3048996.40 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:37:04,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1614800.0, ans=0.125
2023-11-21 18:37:06,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.95 vs. limit=10.0
2023-11-21 18:37:13,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.38 vs. limit=12.0
2023-11-21 18:37:26,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1614866.6666666667, ans=0.125
2023-11-21 18:37:38,585 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242250
2023-11-21 18:37:46,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1615000.0, ans=0.05
2023-11-21 18:38:07,258 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1800, loss[loss=0.09243, simple_loss=0.1291, pruned_loss=0.01942, audio_tagging_loss=0.008483, over 16053.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.0946, pruned_loss=0.01588, audio_tagging_loss=0.009593, over 3044380.13 frames. ], batch size: 57, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:38:18,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1615133.3333333333, ans=0.125
2023-11-21 18:38:24,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1615200.0, ans=0.125
2023-11-21 18:38:25,233 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.425e+01 8.166e+01 8.773e+01 9.532e+01 1.071e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 18:38:37,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1615266.6666666667, ans=0.0
2023-11-21 18:38:39,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1615266.6666666667, ans=0.125
2023-11-21 18:38:39,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1615266.6666666667, ans=0.1
2023-11-21 18:38:43,713 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242300
2023-11-21 18:38:48,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1615333.3333333333, ans=0.125
2023-11-21 18:38:49,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1615333.3333333333, ans=0.0
2023-11-21 18:38:56,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1615333.3333333333, ans=0.125
2023-11-21 18:39:11,201 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1850, loss[loss=0.06959, simple_loss=0.09418, pruned_loss=0.0125, audio_tagging_loss=0.01, over 15290.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09564, pruned_loss=0.01609, audio_tagging_loss=0.009477, over 3040423.49 frames. ], batch size: 55, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:39:37,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1615600.0, ans=0.125
2023-11-21 18:39:44,109 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.82 vs. limit=15.0
2023-11-21 18:39:47,078 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242350
2023-11-21 18:40:03,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1615733.3333333333, ans=0.0
2023-11-21 18:40:14,383 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1900, loss[loss=0.08294, simple_loss=0.108, pruned_loss=0.01901, audio_tagging_loss=0.009934, over 15569.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.0953, pruned_loss=0.01599, audio_tagging_loss=0.009451, over 3044617.09 frames. ], batch size: 58, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:40:33,150 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.159e+01 8.678e+01 9.357e+01 1.521e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 18:40:45,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1615933.3333333333, ans=0.125
2023-11-21 18:40:50,847 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242400
2023-11-21 18:41:19,186 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 1950, loss[loss=0.06527, simple_loss=0.08143, pruned_loss=0.01331, audio_tagging_loss=0.01124, over 14655.00 frames. ], tot_loss[loss=0.07265, simple_loss=0.0946, pruned_loss=0.01588, audio_tagging_loss=0.009474, over 3040543.93 frames. ], batch size: 56, lr: 3.29e-03, grad_scale: 16.0
2023-11-21 18:41:22,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.46 vs. limit=12.0
2023-11-21 18:41:31,568 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.25 vs. limit=15.0
2023-11-21 18:41:47,588 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.70 vs. limit=15.0
2023-11-21 18:41:54,103 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242450
2023-11-21 18:42:14,717 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.59 vs. limit=15.0
2023-11-21 18:42:18,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1616400.0, ans=0.1
2023-11-21 18:42:23,237 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2000, loss[loss=0.07702, simple_loss=0.09927, pruned_loss=0.01738, audio_tagging_loss=0.01, over 14619.00 frames. ], tot_loss[loss=0.07366, simple_loss=0.0959, pruned_loss=0.01625, audio_tagging_loss=0.009458, over 3039014.11 frames. ], batch size: 54, lr: 3.29e-03, grad_scale: 32.0
2023-11-21 18:42:30,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1616466.6666666667, ans=0.125
2023-11-21 18:42:40,262 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.542e+01 7.872e+01 8.647e+01 9.317e+01 2.008e+02, threshold=1.729e+02, percent-clipped=1.0
2023-11-21 18:42:46,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1616533.3333333333, ans=0.125
2023-11-21 18:42:58,694 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242500
2023-11-21 18:42:59,208 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.26 vs. limit=15.0
2023-11-21 18:43:02,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1616666.6666666667, ans=0.2
2023-11-21 18:43:26,338 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2050, loss[loss=0.05322, simple_loss=0.06436, pruned_loss=0.01069, audio_tagging_loss=0.01035, over 14887.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09529, pruned_loss=0.01617, audio_tagging_loss=0.009536, over 3033937.53 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:43:45,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1616866.6666666667, ans=0.125
2023-11-21 18:43:46,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1616866.6666666667, ans=0.0
2023-11-21 18:44:03,199 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242550
2023-11-21 18:44:31,777 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2100, loss[loss=0.06828, simple_loss=0.08259, pruned_loss=0.0169, audio_tagging_loss=0.01008, over 15247.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09587, pruned_loss=0.01643, audio_tagging_loss=0.009485, over 3034495.67 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:44:37,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.51 vs. limit=12.0
2023-11-21 18:44:43,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1617200.0, ans=0.125
2023-11-21 18:44:50,752 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.884e+01 8.232e+01 8.682e+01 9.296e+01 1.860e+02, threshold=1.736e+02, percent-clipped=1.0
2023-11-21 18:44:51,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1617200.0, ans=0.125
2023-11-21 18:45:02,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.30 vs. limit=10.0
2023-11-21 18:45:05,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1617266.6666666667, ans=0.05
2023-11-21 18:45:06,936 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242600
2023-11-21 18:45:14,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1617333.3333333333, ans=0.125
2023-11-21 18:45:20,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1617333.3333333333, ans=0.2
2023-11-21 18:45:36,238 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2150, loss[loss=0.0695, simple_loss=0.08308, pruned_loss=0.01485, audio_tagging_loss=0.01311, over 14956.00 frames. ], tot_loss[loss=0.07448, simple_loss=0.09653, pruned_loss=0.01671, audio_tagging_loss=0.009508, over 3032730.51 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:45:45,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1617466.6666666667, ans=0.0
2023-11-21 18:45:45,488 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.50 vs. limit=10.0
2023-11-21 18:45:55,432 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.54 vs. limit=22.5
2023-11-21 18:46:12,460 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242650
2023-11-21 18:46:14,895 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:46:39,508 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2200, loss[loss=0.0894, simple_loss=0.1191, pruned_loss=0.02267, audio_tagging_loss=0.007186, over 15513.00 frames. ], tot_loss[loss=0.07559, simple_loss=0.09809, pruned_loss=0.01707, audio_tagging_loss=0.009474, over 3036145.76 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:46:43,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1617800.0, ans=0.2
2023-11-21 18:46:58,820 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.497e+01 8.003e+01 8.660e+01 9.579e+01 1.643e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 18:47:08,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1617933.3333333333, ans=0.125
2023-11-21 18:47:09,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1617933.3333333333, ans=0.125
2023-11-21 18:47:15,849 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242700
2023-11-21 18:47:43,664 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2250, loss[loss=0.07312, simple_loss=0.08574, pruned_loss=0.01897, audio_tagging_loss=0.01129, over 15478.00 frames. ], tot_loss[loss=0.07514, simple_loss=0.0977, pruned_loss=0.01676, audio_tagging_loss=0.009532, over 3030859.61 frames. ], batch size: 62, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:47:46,473 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:47:58,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1618200.0, ans=0.2
2023-11-21 18:48:03,195 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:48:06,820 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.652e-03
2023-11-21 18:48:09,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1618266.6666666667, ans=0.2
2023-11-21 18:48:12,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1618266.6666666667, ans=0.125
2023-11-21 18:48:18,726 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242750
2023-11-21 18:48:33,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1618400.0, ans=0.125
2023-11-21 18:48:40,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1618400.0, ans=0.015
2023-11-21 18:48:46,653 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0
2023-11-21 18:48:47,307 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2300, loss[loss=0.1049, simple_loss=0.1392, pruned_loss=0.02785, audio_tagging_loss=0.007454, over 15605.00 frames. ], tot_loss[loss=0.07459, simple_loss=0.09684, pruned_loss=0.01659, audio_tagging_loss=0.009579, over 3036177.85 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:48:54,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1618466.6666666667, ans=0.0
2023-11-21 18:48:59,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1618533.3333333333, ans=0.2
2023-11-21 18:49:06,239 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.387e+01 7.925e+01 8.512e+01 9.086e+01 1.194e+02, threshold=1.702e+02, percent-clipped=0.0
2023-11-21 18:49:23,573 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242800
2023-11-21 18:49:44,461 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:49:46,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1618733.3333333333, ans=0.0
2023-11-21 18:49:51,759 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2350, loss[loss=0.0708, simple_loss=0.0796, pruned_loss=0.01697, audio_tagging_loss=0.01402, over 15316.00 frames. ], tot_loss[loss=0.07474, simple_loss=0.09689, pruned_loss=0.01665, audio_tagging_loss=0.009649, over 3045806.85 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:49:56,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1618800.0, ans=0.04949747468305833
2023-11-21 18:50:11,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1618866.6666666667, ans=0.0
2023-11-21 18:50:21,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1618933.3333333333, ans=0.125
2023-11-21 18:50:27,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1618933.3333333333, ans=0.125
2023-11-21 18:50:28,247 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242850
2023-11-21 18:50:33,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1619000.0, ans=0.125
2023-11-21 18:50:44,490 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.18 vs. limit=15.0
2023-11-21 18:50:46,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1619066.6666666667, ans=0.2
2023-11-21 18:50:48,027 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:50:56,095 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2400, loss[loss=0.07511, simple_loss=0.09887, pruned_loss=0.0152, audio_tagging_loss=0.01048, over 15047.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09624, pruned_loss=0.01646, audio_tagging_loss=0.009733, over 3040052.05 frames. ], batch size: 55, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:50:57,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1619133.3333333333, ans=0.0
2023-11-21 18:51:10,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.70 vs. limit=15.0
2023-11-21 18:51:14,806 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.18 vs. limit=15.0
2023-11-21 18:51:15,446 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.343e+01 8.783e+01 9.778e+01 1.689e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 18:51:25,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1619266.6666666667, ans=0.125
2023-11-21 18:51:30,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1619266.6666666667, ans=0.125
2023-11-21 18:51:31,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242900
2023-11-21 18:51:36,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1619333.3333333333, ans=0.0
2023-11-21 18:51:46,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1619400.0, ans=0.125
2023-11-21 18:51:48,964 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.31 vs. limit=15.0
2023-11-21 18:51:58,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1619466.6666666667, ans=0.0
2023-11-21 18:51:59,522 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2450, loss[loss=0.06754, simple_loss=0.08694, pruned_loss=0.01493, audio_tagging_loss=0.009146, over 15597.00 frames. ], tot_loss[loss=0.07413, simple_loss=0.09586, pruned_loss=0.01636, audio_tagging_loss=0.009831, over 3046071.86 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:52:14,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1619533.3333333333, ans=0.125
2023-11-21 18:52:25,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1619600.0, ans=0.125
2023-11-21 18:52:36,218 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 242950
2023-11-21 18:52:55,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1619733.3333333333, ans=0.04949747468305833
2023-11-21 18:53:00,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1619733.3333333333, ans=0.0
2023-11-21 18:53:04,007 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2500, loss[loss=0.06071, simple_loss=0.07464, pruned_loss=0.01232, audio_tagging_loss=0.01107, over 15073.00 frames. ], tot_loss[loss=0.07391, simple_loss=0.09548, pruned_loss=0.01633, audio_tagging_loss=0.009844, over 3051769.71 frames. ], batch size: 58, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:53:23,259 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.882e+01 8.197e+01 8.814e+01 9.365e+01 1.246e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-21 18:53:27,610 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=11.02 vs. limit=12.0
2023-11-21 18:53:39,827 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243000
2023-11-21 18:53:48,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1620000.0, ans=0.125
2023-11-21 18:54:08,144 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2550, loss[loss=0.05431, simple_loss=0.06382, pruned_loss=0.01211, audio_tagging_loss=0.01029, over 14745.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09576, pruned_loss=0.01624, audio_tagging_loss=0.009718, over 3050166.32 frames. ], batch size: 55, lr: 3.28e-03, grad_scale: 32.0
2023-11-21 18:54:11,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1620133.3333333333, ans=0.0
2023-11-21 18:54:44,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243050
2023-11-21 18:55:07,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1620400.0, ans=0.2
2023-11-21 18:55:12,133 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2600, loss[loss=0.05565, simple_loss=0.07103, pruned_loss=0.01123, audio_tagging_loss=0.008898, over 15062.00 frames. ], tot_loss[loss=0.07394, simple_loss=0.09625, pruned_loss=0.0162, audio_tagging_loss=0.009617, over 3050792.91 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:55:27,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1620533.3333333333, ans=0.0
2023-11-21 18:55:32,968 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.728e+01 8.396e+01 8.905e+01 9.438e+01 1.443e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-21 18:55:44,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1620600.0, ans=0.125
2023-11-21 18:55:48,681 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243100
2023-11-21 18:56:07,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1620733.3333333333, ans=0.025
2023-11-21 18:56:16,578 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2650, loss[loss=0.06422, simple_loss=0.08568, pruned_loss=0.01095, audio_tagging_loss=0.01043, over 14784.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09602, pruned_loss=0.0162, audio_tagging_loss=0.009559, over 3047251.42 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 18:56:18,581 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0
2023-11-21 18:56:23,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1620800.0, ans=0.125
2023-11-21 18:56:29,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1620866.6666666667, ans=15.0
2023-11-21 18:56:51,978 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243150
2023-11-21 18:56:55,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1621000.0, ans=0.0
2023-11-21 18:56:57,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1621000.0, ans=0.0
2023-11-21 18:57:03,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1621000.0, ans=0.125
2023-11-21 18:57:04,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1621000.0, ans=0.125
2023-11-21 18:57:07,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1621066.6666666667, ans=0.125
2023-11-21 18:57:19,994 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2700, loss[loss=0.0977, simple_loss=0.1253, pruned_loss=0.0228, audio_tagging_loss=0.01223, over 15632.00 frames. ], tot_loss[loss=0.07409, simple_loss=0.09625, pruned_loss=0.01643, audio_tagging_loss=0.009534, over 3041221.75 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 18:57:27,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.60 vs. limit=6.0
2023-11-21 18:57:37,478 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.12 vs. limit=15.0
2023-11-21 18:57:41,531 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 7.974e+01 8.822e+01 9.240e+01 1.303e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-21 18:57:56,193 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243200
2023-11-21 18:58:00,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1621333.3333333333, ans=0.0
2023-11-21 18:58:09,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1621400.0, ans=0.2
2023-11-21 18:58:12,896 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 18:58:12,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1621400.0, ans=0.0
2023-11-21 18:58:14,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1621400.0, ans=0.2
2023-11-21 18:58:22,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.88 vs. limit=22.5
2023-11-21 18:58:24,264 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2750, loss[loss=0.05734, simple_loss=0.07315, pruned_loss=0.01173, audio_tagging_loss=0.009039, over 16786.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09565, pruned_loss=0.01644, audio_tagging_loss=0.009562, over 3046942.38 frames. ], batch size: 64, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 18:58:30,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1621466.6666666667, ans=0.125
2023-11-21 18:58:32,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1621466.6666666667, ans=0.125
2023-11-21 18:58:50,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1621600.0, ans=0.0
2023-11-21 18:59:00,240 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243250
2023-11-21 18:59:17,895 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 18:59:28,136 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2800, loss[loss=0.07994, simple_loss=0.09957, pruned_loss=0.02101, audio_tagging_loss=0.009144, over 15444.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09553, pruned_loss=0.0163, audio_tagging_loss=0.009588, over 3047197.37 frames. ], batch size: 60, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 18:59:46,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1621866.6666666667, ans=0.2
2023-11-21 18:59:50,356 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.468e+01 7.876e+01 8.639e+01 9.327e+01 1.140e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 18:59:54,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1621933.3333333333, ans=0.2
2023-11-21 19:00:03,995 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243300
2023-11-21 19:00:04,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1621933.3333333333, ans=0.1
2023-11-21 19:00:09,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1622000.0, ans=0.0
2023-11-21 19:00:14,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1622000.0, ans=0.2
2023-11-21 19:00:31,563 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2850, loss[loss=0.06794, simple_loss=0.0872, pruned_loss=0.01467, audio_tagging_loss=0.009674, over 15289.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09533, pruned_loss=0.01627, audio_tagging_loss=0.009643, over 3044116.23 frames. ], batch size: 59, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:00:33,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1622133.3333333333, ans=0.125
2023-11-21 19:01:06,478 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243350
2023-11-21 19:01:09,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1622333.3333333333, ans=0.125
2023-11-21 19:01:22,127 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=15.0
2023-11-21 19:01:22,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1622400.0, ans=0.125
2023-11-21 19:01:34,251 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2900, loss[loss=0.08169, simple_loss=0.1056, pruned_loss=0.01978, audio_tagging_loss=0.009121, over 15958.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09541, pruned_loss=0.01639, audio_tagging_loss=0.009542, over 3032694.21 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:01:55,209 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.905e+01 8.105e+01 8.663e+01 9.336e+01 1.251e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-21 19:02:09,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243400
2023-11-21 19:02:17,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1622666.6666666667, ans=0.07
2023-11-21 19:02:24,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1622733.3333333333, ans=0.125
2023-11-21 19:02:37,974 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 2950, loss[loss=0.09223, simple_loss=0.1245, pruned_loss=0.02119, audio_tagging_loss=0.008809, over 16418.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09566, pruned_loss=0.01642, audio_tagging_loss=0.009599, over 3037947.24 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:02:40,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1622800.0, ans=0.05
2023-11-21 19:02:42,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1622800.0, ans=0.5
2023-11-21 19:02:58,599 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.25 vs. limit=15.0
2023-11-21 19:03:02,787 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.72 vs. limit=10.0
2023-11-21 19:03:09,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1622933.3333333333, ans=0.0
2023-11-21 19:03:13,813 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243450
2023-11-21 19:03:39,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1623066.6666666667, ans=0.0
2023-11-21 19:03:41,271 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3000, loss[loss=0.07018, simple_loss=0.08549, pruned_loss=0.02018, audio_tagging_loss=0.007257, over 14049.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09633, pruned_loss=0.01651, audio_tagging_loss=0.00973, over 3036550.27 frames. ], batch size: 55, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:03:41,274 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 19:04:22,577 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4631, 3.0419, 3.7725, 3.5127], device='cuda:0')
2023-11-21 19:04:24,766 INFO [train_asr.py:1253] (0/4) Epoch 21, validation: loss=0.0594, simple_loss=0.05205, pruned_loss=0.005197, audio_tagging_loss=0.02817, over 4681554.00 frames. 
2023-11-21 19:04:24,766 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 19:04:47,417 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.111e+01 8.701e+01 9.542e+01 1.276e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 19:05:00,364 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243500
2023-11-21 19:05:19,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1623400.0, ans=0.2
2023-11-21 19:05:28,939 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3050, loss[loss=0.09995, simple_loss=0.131, pruned_loss=0.02499, audio_tagging_loss=0.009454, over 16284.00 frames. ], tot_loss[loss=0.07451, simple_loss=0.09653, pruned_loss=0.01651, audio_tagging_loss=0.009741, over 3040588.86 frames. ], batch size: 62, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:06:05,530 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:06:05,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243550
2023-11-21 19:06:20,338 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.60 vs. limit=22.5
2023-11-21 19:06:32,834 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3100, loss[loss=0.08205, simple_loss=0.1176, pruned_loss=0.01468, audio_tagging_loss=0.008551, over 15076.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09706, pruned_loss=0.01659, audio_tagging_loss=0.009731, over 3043428.99 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:06:49,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1623866.6666666667, ans=0.0
2023-11-21 19:06:56,498 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.034e+01 8.558e+01 9.488e+01 1.458e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 19:07:01,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1623933.3333333333, ans=10.0
2023-11-21 19:07:09,223 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243600
2023-11-21 19:07:17,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1624000.0, ans=0.125
2023-11-21 19:07:18,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1624000.0, ans=0.125
2023-11-21 19:07:19,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1624000.0, ans=0.0
2023-11-21 19:07:30,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1624066.6666666667, ans=0.2
2023-11-21 19:07:38,440 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3150, loss[loss=0.08059, simple_loss=0.115, pruned_loss=0.01751, audio_tagging_loss=0.0056, over 15844.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09658, pruned_loss=0.01636, audio_tagging_loss=0.009793, over 3042454.72 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 8.0
2023-11-21 19:08:13,667 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243650
2023-11-21 19:08:14,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.69 vs. limit=6.0
2023-11-21 19:08:28,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1624333.3333333333, ans=0.125
2023-11-21 19:08:29,701 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-21 19:08:42,946 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3200, loss[loss=0.07152, simple_loss=0.09872, pruned_loss=0.01257, audio_tagging_loss=0.009592, over 14426.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09588, pruned_loss=0.01627, audio_tagging_loss=0.009854, over 3045952.82 frames. ], batch size: 56, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:08:43,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1624466.6666666667, ans=0.0
2023-11-21 19:09:04,559 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.386e+01 8.002e+01 8.673e+01 9.467e+01 1.702e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 19:09:07,723 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.52 vs. limit=22.5
2023-11-21 19:09:09,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1624600.0, ans=0.125
2023-11-21 19:09:18,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243700
2023-11-21 19:09:18,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1624600.0, ans=0.125
2023-11-21 19:09:28,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1624666.6666666667, ans=0.125
2023-11-21 19:09:39,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1624733.3333333333, ans=0.04949747468305833
2023-11-21 19:09:45,218 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3250, loss[loss=0.07797, simple_loss=0.0961, pruned_loss=0.01514, audio_tagging_loss=0.01478, over 15141.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09511, pruned_loss=0.0161, audio_tagging_loss=0.00994, over 3042684.53 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:09:46,694 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:10:15,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1624933.3333333333, ans=0.1
2023-11-21 19:10:16,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1624933.3333333333, ans=0.0
2023-11-21 19:10:17,889 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0
2023-11-21 19:10:18,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1624933.3333333333, ans=0.1
2023-11-21 19:10:20,903 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243750
2023-11-21 19:10:32,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1625000.0, ans=0.125
2023-11-21 19:10:44,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1625066.6666666667, ans=0.125
2023-11-21 19:10:48,922 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3300, loss[loss=0.06529, simple_loss=0.08195, pruned_loss=0.01541, audio_tagging_loss=0.008905, over 15171.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09449, pruned_loss=0.01605, audio_tagging_loss=0.009991, over 3042718.49 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:10:53,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1625133.3333333333, ans=0.1
2023-11-21 19:11:03,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1625200.0, ans=0.1
2023-11-21 19:11:11,998 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.784e+01 8.264e+01 8.839e+01 9.527e+01 1.267e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 19:11:24,388 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243800
2023-11-21 19:11:26,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.14 vs. limit=10.0
2023-11-21 19:11:29,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.49 vs. limit=22.5
2023-11-21 19:11:45,537 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=12.76 vs. limit=15.0
2023-11-21 19:11:53,899 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3350, loss[loss=0.0705, simple_loss=0.08023, pruned_loss=0.01895, audio_tagging_loss=0.01144, over 14280.00 frames. ], tot_loss[loss=0.07369, simple_loss=0.09517, pruned_loss=0.01621, audio_tagging_loss=0.009894, over 3044890.63 frames. ], batch size: 54, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:12:15,586 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.58 vs. limit=15.0
2023-11-21 19:12:19,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1625600.0, ans=0.125
2023-11-21 19:12:29,504 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243850
2023-11-21 19:12:55,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1625733.3333333333, ans=0.0
2023-11-21 19:12:57,944 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3400, loss[loss=0.06187, simple_loss=0.07623, pruned_loss=0.01222, audio_tagging_loss=0.01154, over 14964.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09501, pruned_loss=0.01616, audio_tagging_loss=0.009693, over 3039188.08 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:13:04,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1625800.0, ans=0.125
2023-11-21 19:13:12,261 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=15.0
2023-11-21 19:13:12,395 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.29 vs. limit=22.5
2023-11-21 19:13:13,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1625866.6666666667, ans=0.125
2023-11-21 19:13:19,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1625866.6666666667, ans=0.5
2023-11-21 19:13:20,639 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.826e+01 8.123e+01 8.713e+01 9.440e+01 3.289e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 19:13:22,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1625933.3333333333, ans=0.125
2023-11-21 19:13:33,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1625933.3333333333, ans=0.125
2023-11-21 19:13:34,178 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243900
2023-11-21 19:13:36,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.24 vs. limit=10.0
2023-11-21 19:13:49,680 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.57 vs. limit=22.5
2023-11-21 19:13:56,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1626066.6666666667, ans=0.125
2023-11-21 19:13:59,336 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.74 vs. limit=12.0
2023-11-21 19:14:01,809 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3450, loss[loss=0.1036, simple_loss=0.128, pruned_loss=0.02956, audio_tagging_loss=0.0101, over 16113.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09471, pruned_loss=0.01616, audio_tagging_loss=0.009654, over 3048565.69 frames. ], batch size: 57, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:14:33,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1626266.6666666667, ans=0.125
2023-11-21 19:14:37,944 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 243950
2023-11-21 19:14:53,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1626400.0, ans=0.1
2023-11-21 19:14:59,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1626400.0, ans=0.125
2023-11-21 19:15:03,257 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.88 vs. limit=15.0
2023-11-21 19:15:06,970 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3500, loss[loss=0.07219, simple_loss=0.09377, pruned_loss=0.01235, audio_tagging_loss=0.01296, over 14654.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09386, pruned_loss=0.01596, audio_tagging_loss=0.00964, over 3050925.27 frames. ], batch size: 55, lr: 3.28e-03, grad_scale: 16.0
2023-11-21 19:15:19,342 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.56 vs. limit=5.0
2023-11-21 19:15:29,260 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.309e+01 8.089e+01 8.726e+01 9.622e+01 1.634e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 19:15:35,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1626600.0, ans=0.125
2023-11-21 19:15:37,850 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:15:42,663 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244000
2023-11-21 19:15:44,200 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-244000.pt
2023-11-21 19:15:54,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1626666.6666666667, ans=0.0
2023-11-21 19:15:58,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1626666.6666666667, ans=0.125
2023-11-21 19:16:03,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1626733.3333333333, ans=0.0
2023-11-21 19:16:04,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1626733.3333333333, ans=0.125
2023-11-21 19:16:10,703 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.049e-02
2023-11-21 19:16:13,958 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3550, loss[loss=0.07661, simple_loss=0.09514, pruned_loss=0.01776, audio_tagging_loss=0.01128, over 15214.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09448, pruned_loss=0.01608, audio_tagging_loss=0.009625, over 3048643.98 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:16:51,063 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244050
2023-11-21 19:17:04,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1627066.6666666667, ans=0.125
2023-11-21 19:17:18,064 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3600, loss[loss=0.05615, simple_loss=0.07278, pruned_loss=0.009365, audio_tagging_loss=0.0104, over 15814.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09393, pruned_loss=0.01585, audio_tagging_loss=0.009594, over 3051369.14 frames. ], batch size: 60, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:17:42,707 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.963e+01 8.176e+01 8.799e+01 9.565e+01 1.406e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 19:17:53,942 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244100
2023-11-21 19:17:54,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1627266.6666666667, ans=0.1
2023-11-21 19:18:21,787 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3650, loss[loss=0.05833, simple_loss=0.0651, pruned_loss=0.01186, audio_tagging_loss=0.01392, over 14312.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09481, pruned_loss=0.01611, audio_tagging_loss=0.009546, over 3048843.75 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:18:30,746 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.71 vs. limit=22.5
2023-11-21 19:18:48,033 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.24 vs. limit=22.5
2023-11-21 19:18:57,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244150
2023-11-21 19:18:59,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1627666.6666666667, ans=0.125
2023-11-21 19:19:04,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1627666.6666666667, ans=0.125
2023-11-21 19:19:19,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1627733.3333333333, ans=0.0
2023-11-21 19:19:26,337 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3700, loss[loss=0.06947, simple_loss=0.08964, pruned_loss=0.01226, audio_tagging_loss=0.0124, over 14817.00 frames. ], tot_loss[loss=0.07375, simple_loss=0.09617, pruned_loss=0.01634, audio_tagging_loss=0.009326, over 3050245.94 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:19:26,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1627800.0, ans=0.1
2023-11-21 19:19:38,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1627866.6666666667, ans=0.1
2023-11-21 19:19:46,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1627866.6666666667, ans=0.1
2023-11-21 19:19:50,827 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.276e+01 8.879e+01 9.943e+01 1.926e+02, threshold=1.776e+02, percent-clipped=1.0
2023-11-21 19:19:52,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1627933.3333333333, ans=0.125
2023-11-21 19:20:01,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.64 vs. limit=15.0
2023-11-21 19:20:02,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244200
2023-11-21 19:20:06,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1628000.0, ans=0.125
2023-11-21 19:20:12,641 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=12.0
2023-11-21 19:20:14,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1628000.0, ans=0.2
2023-11-21 19:20:20,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1628066.6666666667, ans=0.125
2023-11-21 19:20:29,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1628133.3333333333, ans=0.125
2023-11-21 19:20:30,383 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3750, loss[loss=0.08419, simple_loss=0.1058, pruned_loss=0.02188, audio_tagging_loss=0.009392, over 15364.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.09707, pruned_loss=0.01648, audio_tagging_loss=0.009305, over 3056862.56 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:20:42,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1628200.0, ans=0.125
2023-11-21 19:20:49,218 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.31 vs. limit=15.0
2023-11-21 19:21:06,995 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244250
2023-11-21 19:21:14,277 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:21:20,999 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.63 vs. limit=6.0
2023-11-21 19:21:30,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1628400.0, ans=0.0
2023-11-21 19:21:35,251 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3800, loss[loss=0.06474, simple_loss=0.07984, pruned_loss=0.01438, audio_tagging_loss=0.01044, over 15966.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09619, pruned_loss=0.01636, audio_tagging_loss=0.009391, over 3055427.49 frames. ], batch size: 61, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:21:36,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1628466.6666666667, ans=0.125
2023-11-21 19:21:53,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1628533.3333333333, ans=0.125
2023-11-21 19:21:59,391 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.245e+01 8.842e+01 9.701e+01 1.541e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 19:22:07,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1628600.0, ans=0.2
2023-11-21 19:22:10,646 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244300
2023-11-21 19:22:30,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1628733.3333333333, ans=0.125
2023-11-21 19:22:39,719 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3850, loss[loss=0.1025, simple_loss=0.1319, pruned_loss=0.02648, audio_tagging_loss=0.01004, over 15099.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09576, pruned_loss=0.01611, audio_tagging_loss=0.009495, over 3053664.27 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:22:53,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1628866.6666666667, ans=0.125
2023-11-21 19:22:56,268 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.45 vs. limit=6.0
2023-11-21 19:23:00,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1628866.6666666667, ans=0.1
2023-11-21 19:23:04,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1628933.3333333333, ans=0.0
2023-11-21 19:23:11,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1628933.3333333333, ans=0.125
2023-11-21 19:23:15,508 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244350
2023-11-21 19:23:17,482 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:23:22,687 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.75 vs. limit=15.0
2023-11-21 19:23:28,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1629000.0, ans=0.125
2023-11-21 19:23:30,557 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.19 vs. limit=15.0
2023-11-21 19:23:43,386 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3900, loss[loss=0.09604, simple_loss=0.1167, pruned_loss=0.028, audio_tagging_loss=0.009681, over 14179.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09608, pruned_loss=0.01626, audio_tagging_loss=0.009568, over 3042019.67 frames. ], batch size: 55, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:23:48,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1629133.3333333333, ans=0.2
2023-11-21 19:24:08,372 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.764e+01 8.034e+01 8.736e+01 9.358e+01 1.610e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 19:24:19,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1629266.6666666667, ans=0.1
2023-11-21 19:24:20,167 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244400
2023-11-21 19:24:35,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1629400.0, ans=0.1
2023-11-21 19:24:48,539 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 3950, loss[loss=0.06419, simple_loss=0.08747, pruned_loss=0.01314, audio_tagging_loss=0.007311, over 15345.00 frames. ], tot_loss[loss=0.07368, simple_loss=0.09581, pruned_loss=0.01611, audio_tagging_loss=0.009659, over 3037568.93 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:24:59,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1629466.6666666667, ans=0.1
2023-11-21 19:25:23,552 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244450
2023-11-21 19:25:29,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1629666.6666666667, ans=0.0
2023-11-21 19:25:30,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1629666.6666666667, ans=0.125
2023-11-21 19:25:32,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1629666.6666666667, ans=0.125
2023-11-21 19:25:52,453 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4000, loss[loss=0.0651, simple_loss=0.08885, pruned_loss=0.01083, audio_tagging_loss=0.009845, over 14493.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09571, pruned_loss=0.01616, audio_tagging_loss=0.009745, over 3026657.93 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:25:58,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1629800.0, ans=0.1
2023-11-21 19:26:16,059 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.271e+01 8.886e+01 9.884e+01 1.294e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-21 19:26:24,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1629933.3333333333, ans=0.125
2023-11-21 19:26:28,142 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244500
2023-11-21 19:26:30,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1630000.0, ans=0.1
2023-11-21 19:26:56,266 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4050, loss[loss=0.0778, simple_loss=0.1069, pruned_loss=0.01456, audio_tagging_loss=0.009802, over 17345.00 frames. ], tot_loss[loss=0.07396, simple_loss=0.09576, pruned_loss=0.01623, audio_tagging_loss=0.009851, over 3035635.78 frames. ], batch size: 62, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:26:58,656 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:27:08,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1630200.0, ans=0.1
2023-11-21 19:27:32,281 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244550
2023-11-21 19:27:34,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1630333.3333333333, ans=0.07
2023-11-21 19:27:38,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=1630333.3333333333, ans=0.05
2023-11-21 19:27:42,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1630333.3333333333, ans=0.2
2023-11-21 19:28:00,859 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4100, loss[loss=0.05002, simple_loss=0.06551, pruned_loss=0.009081, audio_tagging_loss=0.008187, over 14638.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09543, pruned_loss=0.01613, audio_tagging_loss=0.009851, over 3033030.91 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:28:05,099 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.86 vs. limit=15.0
2023-11-21 19:28:18,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1630533.3333333333, ans=0.0
2023-11-21 19:28:23,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1630533.3333333333, ans=0.95
2023-11-21 19:28:25,948 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.257e+01 8.384e+01 8.914e+01 9.573e+01 1.296e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-21 19:28:35,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1630600.0, ans=0.125
2023-11-21 19:28:36,739 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244600
2023-11-21 19:28:42,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1630666.6666666667, ans=0.0
2023-11-21 19:28:49,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1630666.6666666667, ans=0.125
2023-11-21 19:29:06,020 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4150, loss[loss=0.0624, simple_loss=0.06995, pruned_loss=0.01519, audio_tagging_loss=0.01224, over 14869.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09572, pruned_loss=0.01619, audio_tagging_loss=0.009709, over 3039655.27 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:29:08,326 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.38 vs. limit=15.0
2023-11-21 19:29:16,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1630800.0, ans=0.125
2023-11-21 19:29:25,737 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:29:28,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1630866.6666666667, ans=0.0
2023-11-21 19:29:28,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.whiten.whitening_limit, batch_count=1630866.6666666667, ans=12.0
2023-11-21 19:29:42,539 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244650
2023-11-21 19:29:47,666 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:29:53,011 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:29:59,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1631066.6666666667, ans=0.125
2023-11-21 19:30:00,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1631066.6666666667, ans=0.0
2023-11-21 19:30:04,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1631066.6666666667, ans=0.125
2023-11-21 19:30:10,068 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4200, loss[loss=0.0666, simple_loss=0.06882, pruned_loss=0.01953, audio_tagging_loss=0.01266, over 15696.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.09643, pruned_loss=0.0164, audio_tagging_loss=0.00958, over 3043936.24 frames. ], batch size: 60, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:30:34,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1631200.0, ans=0.2
2023-11-21 19:30:36,268 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 7.875e+01 8.455e+01 9.386e+01 1.251e+02, threshold=1.691e+02, percent-clipped=0.0
2023-11-21 19:30:43,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1631266.6666666667, ans=0.125
2023-11-21 19:30:46,927 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244700
2023-11-21 19:30:52,267 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.46 vs. limit=15.0
2023-11-21 19:30:55,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1631333.3333333333, ans=0.1
2023-11-21 19:31:02,276 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.67 vs. limit=15.0
2023-11-21 19:31:04,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1631400.0, ans=0.125
2023-11-21 19:31:05,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1631400.0, ans=0.1
2023-11-21 19:31:06,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1631400.0, ans=0.125
2023-11-21 19:31:08,714 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.83 vs. limit=15.0
2023-11-21 19:31:15,167 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4250, loss[loss=0.07082, simple_loss=0.09103, pruned_loss=0.01553, audio_tagging_loss=0.009773, over 14873.00 frames. ], tot_loss[loss=0.07479, simple_loss=0.09738, pruned_loss=0.01661, audio_tagging_loss=0.009485, over 3048838.66 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:31:28,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1631533.3333333333, ans=0.125
2023-11-21 19:31:50,281 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244750
2023-11-21 19:31:54,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1631666.6666666667, ans=0.125
2023-11-21 19:32:09,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1631733.3333333333, ans=0.04949747468305833
2023-11-21 19:32:19,210 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4300, loss[loss=0.06291, simple_loss=0.07687, pruned_loss=0.01424, audio_tagging_loss=0.01024, over 15819.00 frames. ], tot_loss[loss=0.07523, simple_loss=0.09816, pruned_loss=0.01693, audio_tagging_loss=0.00922, over 3046724.23 frames. ], batch size: 60, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:32:30,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1631866.6666666667, ans=0.125
2023-11-21 19:32:32,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1631866.6666666667, ans=0.0
2023-11-21 19:32:43,222 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.395e+01 8.866e+01 9.565e+01 1.376e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-21 19:32:55,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244800
2023-11-21 19:33:03,066 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.81 vs. limit=12.0
2023-11-21 19:33:11,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1632066.6666666667, ans=0.125
2023-11-21 19:33:14,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.48 vs. limit=12.0
2023-11-21 19:33:19,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1632066.6666666667, ans=0.0
2023-11-21 19:33:22,514 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4350, loss[loss=0.06551, simple_loss=0.08367, pruned_loss=0.01356, audio_tagging_loss=0.01012, over 14600.00 frames. ], tot_loss[loss=0.07404, simple_loss=0.09671, pruned_loss=0.01646, audio_tagging_loss=0.00923, over 3042570.11 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 16.0
2023-11-21 19:33:27,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1632133.3333333333, ans=0.1
2023-11-21 19:33:36,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1632200.0, ans=0.0
2023-11-21 19:33:36,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1632200.0, ans=0.125
2023-11-21 19:33:42,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1632200.0, ans=0.0
2023-11-21 19:33:47,055 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.33 vs. limit=12.0
2023-11-21 19:33:56,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1632266.6666666667, ans=0.95
2023-11-21 19:33:58,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244850
2023-11-21 19:34:15,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1632400.0, ans=0.125
2023-11-21 19:34:27,303 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4400, loss[loss=0.07275, simple_loss=0.08563, pruned_loss=0.01809, audio_tagging_loss=0.01184, over 15509.00 frames. ], tot_loss[loss=0.07401, simple_loss=0.09666, pruned_loss=0.01644, audio_tagging_loss=0.009238, over 3038227.37 frames. ], batch size: 59, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:34:45,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1632533.3333333333, ans=0.07
2023-11-21 19:34:52,770 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.580e+01 8.144e+01 8.724e+01 9.531e+01 1.294e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 19:35:02,679 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244900
2023-11-21 19:35:03,073 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.90 vs. limit=15.0
2023-11-21 19:35:10,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1632666.6666666667, ans=0.0
2023-11-21 19:35:13,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1632666.6666666667, ans=0.0
2023-11-21 19:35:14,528 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.94 vs. limit=22.5
2023-11-21 19:35:26,931 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:35:30,047 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.25 vs. limit=8.0
2023-11-21 19:35:32,195 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4450, loss[loss=0.04829, simple_loss=0.05405, pruned_loss=0.01131, audio_tagging_loss=0.009952, over 15173.00 frames. ], tot_loss[loss=0.07422, simple_loss=0.09679, pruned_loss=0.01666, audio_tagging_loss=0.009162, over 3038692.76 frames. ], batch size: 59, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:35:35,477 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.10 vs. limit=15.0
2023-11-21 19:35:36,746 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.59 vs. limit=22.5
2023-11-21 19:35:38,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1632800.0, ans=0.0
2023-11-21 19:35:39,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1632800.0, ans=0.0
2023-11-21 19:35:45,157 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.18 vs. limit=15.0
2023-11-21 19:35:48,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1632866.6666666667, ans=0.125
2023-11-21 19:35:48,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1632866.6666666667, ans=0.0
2023-11-21 19:35:50,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.59 vs. limit=22.5
2023-11-21 19:35:55,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1632933.3333333333, ans=0.1
2023-11-21 19:36:07,796 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 244950
2023-11-21 19:36:22,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1633066.6666666667, ans=0.0
2023-11-21 19:36:25,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1633066.6666666667, ans=0.125
2023-11-21 19:36:29,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1633066.6666666667, ans=0.2
2023-11-21 19:36:34,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.95 vs. limit=15.0
2023-11-21 19:36:35,226 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4500, loss[loss=0.09346, simple_loss=0.1256, pruned_loss=0.02141, audio_tagging_loss=0.009232, over 14294.00 frames. ], tot_loss[loss=0.07514, simple_loss=0.09837, pruned_loss=0.01688, audio_tagging_loss=0.00908, over 3038977.95 frames. ], batch size: 53, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:36:45,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1633133.3333333333, ans=0.125
2023-11-21 19:36:54,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1633200.0, ans=0.125
2023-11-21 19:36:58,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1633200.0, ans=0.2
2023-11-21 19:37:01,325 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.155e+01 8.993e+01 9.811e+01 1.324e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-21 19:37:03,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1633266.6666666667, ans=0.125
2023-11-21 19:37:07,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1633266.6666666667, ans=0.0
2023-11-21 19:37:11,794 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245000
2023-11-21 19:37:39,069 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4550, loss[loss=0.07377, simple_loss=0.1079, pruned_loss=0.01185, audio_tagging_loss=0.007996, over 15241.00 frames. ], tot_loss[loss=0.07504, simple_loss=0.09799, pruned_loss=0.01686, audio_tagging_loss=0.009192, over 3038319.16 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:37:41,019 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.60 vs. limit=6.0
2023-11-21 19:37:46,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1633466.6666666667, ans=0.125
2023-11-21 19:38:16,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245050
2023-11-21 19:38:28,288 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 19:38:33,267 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.40 vs. limit=22.5
2023-11-21 19:38:38,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1633733.3333333333, ans=0.0
2023-11-21 19:38:44,456 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4600, loss[loss=0.08117, simple_loss=0.1049, pruned_loss=0.02095, audio_tagging_loss=0.007761, over 14881.00 frames. ], tot_loss[loss=0.07521, simple_loss=0.09803, pruned_loss=0.0169, audio_tagging_loss=0.0093, over 3040790.11 frames. ], batch size: 56, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:39:09,272 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.725e+01 8.114e+01 8.737e+01 9.348e+01 1.187e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 19:39:13,685 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.99 vs. limit=10.0
2023-11-21 19:39:18,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1633933.3333333333, ans=0.125
2023-11-21 19:39:19,877 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245100
2023-11-21 19:39:20,415 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.22 vs. limit=10.0
2023-11-21 19:39:25,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1634000.0, ans=0.125
2023-11-21 19:39:29,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1634000.0, ans=0.0
2023-11-21 19:39:33,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1634000.0, ans=0.2
2023-11-21 19:39:40,351 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.89 vs. limit=15.0
2023-11-21 19:39:43,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1634066.6666666667, ans=0.125
2023-11-21 19:39:46,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1634066.6666666667, ans=10.0
2023-11-21 19:39:48,095 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4650, loss[loss=0.07198, simple_loss=0.09344, pruned_loss=0.01797, audio_tagging_loss=0.007288, over 15477.00 frames. ], tot_loss[loss=0.07488, simple_loss=0.09717, pruned_loss=0.01685, audio_tagging_loss=0.009449, over 3045056.39 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:39:55,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1634133.3333333333, ans=0.125
2023-11-21 19:40:24,003 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245150
2023-11-21 19:40:33,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1634333.3333333333, ans=0.5
2023-11-21 19:40:42,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1634400.0, ans=0.0
2023-11-21 19:40:45,215 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:40:50,954 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4700, loss[loss=0.06324, simple_loss=0.07751, pruned_loss=0.0125, audio_tagging_loss=0.01198, over 15340.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09671, pruned_loss=0.01647, audio_tagging_loss=0.00961, over 3043831.92 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:41:06,644 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:41:17,147 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.203e+01 8.093e+01 8.785e+01 9.721e+01 1.198e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-21 19:41:24,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1634600.0, ans=0.0
2023-11-21 19:41:27,056 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245200
2023-11-21 19:41:31,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1634666.6666666667, ans=0.1
2023-11-21 19:41:55,453 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4750, loss[loss=0.08194, simple_loss=0.1123, pruned_loss=0.0165, audio_tagging_loss=0.009314, over 15314.00 frames. ], tot_loss[loss=0.07392, simple_loss=0.0961, pruned_loss=0.01627, audio_tagging_loss=0.00959, over 3045280.20 frames. ], batch size: 59, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:41:55,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1634800.0, ans=0.95
2023-11-21 19:42:00,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1634800.0, ans=0.0
2023-11-21 19:42:06,338 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.08 vs. limit=22.5
2023-11-21 19:42:18,980 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.05 vs. limit=22.5
2023-11-21 19:42:21,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1634933.3333333333, ans=0.1
2023-11-21 19:42:21,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1634933.3333333333, ans=0.2
2023-11-21 19:42:23,617 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.93 vs. limit=22.5
2023-11-21 19:42:30,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245250
2023-11-21 19:42:36,788 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.68 vs. limit=15.0
2023-11-21 19:42:40,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1635000.0, ans=0.1
2023-11-21 19:42:52,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1635066.6666666667, ans=0.0
2023-11-21 19:42:59,129 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4800, loss[loss=0.07846, simple_loss=0.1016, pruned_loss=0.01955, audio_tagging_loss=0.00812, over 14567.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09547, pruned_loss=0.01626, audio_tagging_loss=0.009763, over 3038159.53 frames. ], batch size: 54, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:43:05,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1635133.3333333333, ans=0.125
2023-11-21 19:43:18,016 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.52 vs. limit=15.0
2023-11-21 19:43:23,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1635266.6666666667, ans=0.125
2023-11-21 19:43:25,997 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.114e+01 8.825e+01 9.753e+01 1.179e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-21 19:43:34,735 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245300
2023-11-21 19:43:39,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1635333.3333333333, ans=0.1
2023-11-21 19:43:50,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1635400.0, ans=0.125
2023-11-21 19:43:54,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1635400.0, ans=0.2
2023-11-21 19:44:02,556 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4850, loss[loss=0.06078, simple_loss=0.07163, pruned_loss=0.01134, audio_tagging_loss=0.01362, over 14942.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09461, pruned_loss=0.01604, audio_tagging_loss=0.009802, over 3047446.62 frames. ], batch size: 57, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:44:09,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1635466.6666666667, ans=0.2
2023-11-21 19:44:19,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1635533.3333333333, ans=0.125
2023-11-21 19:44:31,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.38 vs. limit=15.0
2023-11-21 19:44:38,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1635600.0, ans=0.1
2023-11-21 19:44:39,427 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245350
2023-11-21 19:44:41,129 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.36 vs. limit=15.0
2023-11-21 19:44:49,618 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.47 vs. limit=15.0
2023-11-21 19:44:59,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1635733.3333333333, ans=0.125
2023-11-21 19:45:07,169 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4900, loss[loss=0.06518, simple_loss=0.07197, pruned_loss=0.01653, audio_tagging_loss=0.01266, over 15062.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09506, pruned_loss=0.01609, audio_tagging_loss=0.009616, over 3045485.77 frames. ], batch size: 61, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:45:34,467 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.513e+01 7.987e+01 8.445e+01 9.023e+01 1.315e+02, threshold=1.689e+02, percent-clipped=0.0
2023-11-21 19:45:39,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1635933.3333333333, ans=0.125
2023-11-21 19:45:43,190 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245400
2023-11-21 19:45:52,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1636000.0, ans=0.0
2023-11-21 19:45:55,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1636000.0, ans=0.125
2023-11-21 19:46:12,597 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 4950, loss[loss=0.05643, simple_loss=0.05013, pruned_loss=0.0164, audio_tagging_loss=0.01496, over 15085.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09453, pruned_loss=0.01603, audio_tagging_loss=0.009578, over 3041522.75 frames. ], batch size: 60, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:46:17,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1636133.3333333333, ans=0.2
2023-11-21 19:46:23,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1636200.0, ans=0.2
2023-11-21 19:46:39,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1636266.6666666667, ans=0.1
2023-11-21 19:46:48,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245450
2023-11-21 19:47:10,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1636400.0, ans=0.0
2023-11-21 19:47:16,834 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5000, loss[loss=0.05487, simple_loss=0.06776, pruned_loss=0.008179, audio_tagging_loss=0.01281, over 15678.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09444, pruned_loss=0.01601, audio_tagging_loss=0.009548, over 3044074.57 frames. ], batch size: 58, lr: 3.27e-03, grad_scale: 32.0
2023-11-21 19:47:17,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1636466.6666666667, ans=0.125
2023-11-21 19:47:35,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1636533.3333333333, ans=0.125
2023-11-21 19:47:39,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.00 vs. limit=15.0
2023-11-21 19:47:44,387 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.789e+01 8.147e+01 8.751e+01 9.572e+01 1.278e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-21 19:47:49,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1636600.0, ans=0.125
2023-11-21 19:47:53,498 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245500
2023-11-21 19:47:56,511 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.98 vs. limit=15.0
2023-11-21 19:48:21,027 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5050, loss[loss=0.1056, simple_loss=0.1353, pruned_loss=0.03152, audio_tagging_loss=0.006383, over 14837.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09518, pruned_loss=0.01613, audio_tagging_loss=0.009376, over 3047748.03 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:48:46,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1636933.3333333333, ans=0.05
2023-11-21 19:48:53,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1636933.3333333333, ans=0.125
2023-11-21 19:48:56,580 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245550
2023-11-21 19:49:00,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1637000.0, ans=0.1
2023-11-21 19:49:03,307 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 19:49:22,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1637066.6666666667, ans=10.0
2023-11-21 19:49:25,086 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5100, loss[loss=0.07563, simple_loss=0.09978, pruned_loss=0.01642, audio_tagging_loss=0.009319, over 14807.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09421, pruned_loss=0.01611, audio_tagging_loss=0.009446, over 3042258.65 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:49:50,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1637266.6666666667, ans=0.125
2023-11-21 19:49:51,612 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.776e+01 7.985e+01 8.659e+01 9.540e+01 1.416e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-21 19:49:53,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1637266.6666666667, ans=0.2
2023-11-21 19:49:59,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1637266.6666666667, ans=0.125
2023-11-21 19:50:00,764 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245600
2023-11-21 19:50:19,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.24 vs. limit=12.0
2023-11-21 19:50:19,773 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.64 vs. limit=5.0
2023-11-21 19:50:28,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1637466.6666666667, ans=0.125
2023-11-21 19:50:29,325 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5150, loss[loss=0.08302, simple_loss=0.1025, pruned_loss=0.01993, audio_tagging_loss=0.01184, over 14929.00 frames. ], tot_loss[loss=0.07303, simple_loss=0.09497, pruned_loss=0.01612, audio_tagging_loss=0.009418, over 3039153.78 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 19:51:03,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1637600.0, ans=0.1
2023-11-21 19:51:03,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1637600.0, ans=0.0
2023-11-21 19:51:05,574 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245650
2023-11-21 19:51:33,909 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5200, loss[loss=0.08681, simple_loss=0.1059, pruned_loss=0.02392, audio_tagging_loss=0.009939, over 14898.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.0953, pruned_loss=0.01609, audio_tagging_loss=0.009419, over 3042092.42 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:51:42,734 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.59 vs. limit=15.0
2023-11-21 19:51:53,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1637866.6666666667, ans=0.125
2023-11-21 19:52:01,155 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.548e+01 7.936e+01 8.716e+01 9.499e+01 2.064e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 19:52:09,263 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245700
2023-11-21 19:52:18,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1638000.0, ans=0.125
2023-11-21 19:52:23,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1638066.6666666667, ans=0.1
2023-11-21 19:52:38,024 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5250, loss[loss=0.06897, simple_loss=0.08867, pruned_loss=0.01384, audio_tagging_loss=0.01079, over 14857.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09576, pruned_loss=0.01613, audio_tagging_loss=0.009347, over 3043324.24 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:52:56,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=1638200.0, ans=22.5
2023-11-21 19:53:06,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1638266.6666666667, ans=0.125
2023-11-21 19:53:14,342 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245750
2023-11-21 19:53:34,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1638400.0, ans=0.125
2023-11-21 19:53:35,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1638400.0, ans=0.0
2023-11-21 19:53:41,699 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5300, loss[loss=0.07425, simple_loss=0.1044, pruned_loss=0.01473, audio_tagging_loss=0.007338, over 15021.00 frames. ], tot_loss[loss=0.074, simple_loss=0.09689, pruned_loss=0.01628, audio_tagging_loss=0.009272, over 3048063.44 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:53:47,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1638466.6666666667, ans=10.0
2023-11-21 19:54:04,048 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.25 vs. limit=15.0
2023-11-21 19:54:06,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1638533.3333333333, ans=0.1
2023-11-21 19:54:10,758 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.760e+01 8.198e+01 8.861e+01 9.238e+01 1.215e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-21 19:54:12,659 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.44 vs. limit=10.0
2023-11-21 19:54:18,890 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245800
2023-11-21 19:54:24,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1638666.6666666667, ans=0.125
2023-11-21 19:54:33,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1638733.3333333333, ans=0.125
2023-11-21 19:54:47,323 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5350, loss[loss=0.06451, simple_loss=0.08698, pruned_loss=0.01113, audio_tagging_loss=0.009892, over 15181.00 frames. ], tot_loss[loss=0.07387, simple_loss=0.09659, pruned_loss=0.01624, audio_tagging_loss=0.009331, over 3040656.34 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:54:54,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1638800.0, ans=0.125
2023-11-21 19:55:23,170 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245850
2023-11-21 19:55:41,535 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.50 vs. limit=12.0
2023-11-21 19:55:43,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1639066.6666666667, ans=0.07
2023-11-21 19:55:52,131 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5400, loss[loss=0.0614, simple_loss=0.07152, pruned_loss=0.01241, audio_tagging_loss=0.01323, over 15046.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09661, pruned_loss=0.01624, audio_tagging_loss=0.009295, over 3043220.78 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:56:07,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1639200.0, ans=0.125
2023-11-21 19:56:10,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1639200.0, ans=0.125
2023-11-21 19:56:19,850 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.098e+01 8.835e+01 9.334e+01 1.205e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 19:56:26,072 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.75 vs. limit=15.0
2023-11-21 19:56:28,472 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245900
2023-11-21 19:56:55,948 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5450, loss[loss=0.06942, simple_loss=0.08519, pruned_loss=0.01511, audio_tagging_loss=0.01171, over 14396.00 frames. ], tot_loss[loss=0.07428, simple_loss=0.09691, pruned_loss=0.01651, audio_tagging_loss=0.009315, over 3043570.51 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:57:03,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1639466.6666666667, ans=0.1
2023-11-21 19:57:32,967 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 245950
2023-11-21 19:57:52,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1639733.3333333333, ans=0.125
2023-11-21 19:57:55,566 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.50 vs. limit=15.0
2023-11-21 19:58:00,377 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5500, loss[loss=0.0567, simple_loss=0.07915, pruned_loss=0.009607, audio_tagging_loss=0.007515, over 15517.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09596, pruned_loss=0.0162, audio_tagging_loss=0.009397, over 3045826.09 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:58:12,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1639866.6666666667, ans=0.125
2023-11-21 19:58:22,289 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.67 vs. limit=22.5
2023-11-21 19:58:28,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.71 vs. limit=15.0
2023-11-21 19:58:28,913 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.083e+01 8.649e+01 9.254e+01 1.187e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 19:58:36,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246000
2023-11-21 19:58:36,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1639933.3333333333, ans=0.125
2023-11-21 19:58:44,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.01 vs. limit=22.5
2023-11-21 19:58:59,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1640066.6666666667, ans=0.125
2023-11-21 19:59:04,544 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5550, loss[loss=0.06555, simple_loss=0.07749, pruned_loss=0.01388, audio_tagging_loss=0.01293, over 14742.00 frames. ], tot_loss[loss=0.07429, simple_loss=0.09682, pruned_loss=0.01645, audio_tagging_loss=0.009433, over 3043282.94 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 19:59:14,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1640133.3333333333, ans=0.125
2023-11-21 19:59:32,948 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.20 vs. limit=15.0
2023-11-21 19:59:39,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1640266.6666666667, ans=0.125
2023-11-21 19:59:40,111 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246050
2023-11-21 19:59:53,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1640333.3333333333, ans=0.0
2023-11-21 20:00:02,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1640400.0, ans=0.1
2023-11-21 20:00:03,961 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:00:08,716 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5600, loss[loss=0.07465, simple_loss=0.09527, pruned_loss=0.02071, audio_tagging_loss=0.006302, over 14646.00 frames. ], tot_loss[loss=0.07491, simple_loss=0.09769, pruned_loss=0.01664, audio_tagging_loss=0.009431, over 3048025.88 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:00:27,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1640533.3333333333, ans=0.2
2023-11-21 20:00:37,207 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.033e+01 8.931e+01 9.623e+01 1.146e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-21 20:00:38,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1640600.0, ans=0.09899494936611666
2023-11-21 20:00:42,996 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.62 vs. limit=22.5
2023-11-21 20:00:44,748 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246100
2023-11-21 20:00:51,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1640666.6666666667, ans=0.95
2023-11-21 20:00:53,819 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:01:10,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1640800.0, ans=0.1
2023-11-21 20:01:11,919 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5650, loss[loss=0.04898, simple_loss=0.0681, pruned_loss=0.006883, audio_tagging_loss=0.008046, over 14829.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09693, pruned_loss=0.01659, audio_tagging_loss=0.009583, over 3053313.80 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:01:26,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1640866.6666666667, ans=0.125
2023-11-21 20:01:27,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1640866.6666666667, ans=0.1
2023-11-21 20:01:48,530 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246150
2023-11-21 20:02:06,671 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.80 vs. limit=15.0
2023-11-21 20:02:09,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1641066.6666666667, ans=0.0
2023-11-21 20:02:11,698 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:02:14,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.61 vs. limit=12.0
2023-11-21 20:02:16,870 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5700, loss[loss=0.09089, simple_loss=0.1149, pruned_loss=0.02629, audio_tagging_loss=0.007165, over 15249.00 frames. ], tot_loss[loss=0.07447, simple_loss=0.09676, pruned_loss=0.01655, audio_tagging_loss=0.009548, over 3048621.89 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:02:20,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1641133.3333333333, ans=0.2
2023-11-21 20:02:27,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1641133.3333333333, ans=0.0
2023-11-21 20:02:44,525 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.226e+01 8.873e+01 9.610e+01 1.173e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 20:02:52,782 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246200
2023-11-21 20:03:08,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1641400.0, ans=0.1
2023-11-21 20:03:08,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1641400.0, ans=0.125
2023-11-21 20:03:21,890 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5750, loss[loss=0.07006, simple_loss=0.08599, pruned_loss=0.01455, audio_tagging_loss=0.01251, over 15245.00 frames. ], tot_loss[loss=0.07431, simple_loss=0.09686, pruned_loss=0.01646, audio_tagging_loss=0.009426, over 3049206.25 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:03:57,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1641600.0, ans=0.125
2023-11-21 20:03:58,238 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246250
2023-11-21 20:04:11,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1641666.6666666667, ans=0.125
2023-11-21 20:04:11,599 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.49 vs. limit=15.0
2023-11-21 20:04:25,623 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5800, loss[loss=0.05148, simple_loss=0.06002, pruned_loss=0.008959, audio_tagging_loss=0.01251, over 16136.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09709, pruned_loss=0.01632, audio_tagging_loss=0.009289, over 3056794.00 frames. ], batch size: 63, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:04:45,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1641866.6666666667, ans=0.125
2023-11-21 20:04:55,494 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.128e+01 8.477e+01 9.291e+01 1.218e+02, threshold=1.695e+02, percent-clipped=0.0
2023-11-21 20:04:57,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1641933.3333333333, ans=0.125
2023-11-21 20:05:01,771 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246300
2023-11-21 20:05:19,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1642066.6666666667, ans=0.0
2023-11-21 20:05:29,849 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5850, loss[loss=0.08046, simple_loss=0.09987, pruned_loss=0.02031, audio_tagging_loss=0.01021, over 14787.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09574, pruned_loss=0.01618, audio_tagging_loss=0.009354, over 3049918.38 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:05:51,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1642200.0, ans=0.0
2023-11-21 20:06:05,301 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246350
2023-11-21 20:06:34,587 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5900, loss[loss=0.06918, simple_loss=0.08789, pruned_loss=0.01479, audio_tagging_loss=0.01045, over 14477.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.09558, pruned_loss=0.01606, audio_tagging_loss=0.009291, over 3044292.41 frames. ], batch size: 54, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:06:34,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1642466.6666666667, ans=0.125
2023-11-21 20:06:39,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1642466.6666666667, ans=0.2
2023-11-21 20:06:40,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1642466.6666666667, ans=0.0
2023-11-21 20:07:03,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1642600.0, ans=0.125
2023-11-21 20:07:04,003 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.116e+01 8.807e+01 9.373e+01 1.126e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 20:07:09,097 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:07:10,126 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246400
2023-11-21 20:07:37,647 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 5950, loss[loss=0.08912, simple_loss=0.1168, pruned_loss=0.01978, audio_tagging_loss=0.01096, over 15824.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09644, pruned_loss=0.01606, audio_tagging_loss=0.009265, over 3050157.98 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:07:57,909 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.46 vs. limit=15.0
2023-11-21 20:08:14,385 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246450
2023-11-21 20:08:17,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1643000.0, ans=0.125
2023-11-21 20:08:42,278 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6000, loss[loss=0.06556, simple_loss=0.08369, pruned_loss=0.01396, audio_tagging_loss=0.009757, over 15407.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09529, pruned_loss=0.01587, audio_tagging_loss=0.009347, over 3047755.72 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 32.0
2023-11-21 20:08:42,281 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 20:09:09,093 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.5889, 1.2328, 3.5782, 3.1059, 2.7044, 3.1595, 2.7357, 3.4235],
       device='cuda:0')
2023-11-21 20:09:23,098 INFO [train_asr.py:1253] (0/4) Epoch 21, validation: loss=0.05951, simple_loss=0.05205, pruned_loss=0.005242, audio_tagging_loss=0.02825, over 4681554.00 frames. 
2023-11-21 20:09:23,099 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 20:09:34,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1643200.0, ans=0.2
2023-11-21 20:09:46,998 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.71 vs. limit=6.0
2023-11-21 20:09:47,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1643266.6666666667, ans=0.125
2023-11-21 20:09:52,403 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.622e+01 7.949e+01 8.551e+01 9.154e+01 1.132e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-21 20:09:58,745 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246500
2023-11-21 20:10:05,719 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.75 vs. limit=15.0
2023-11-21 20:10:09,443 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:10:24,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1643400.0, ans=0.2
2023-11-21 20:10:26,634 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6050, loss[loss=0.08634, simple_loss=0.1089, pruned_loss=0.0236, audio_tagging_loss=0.008275, over 16126.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09504, pruned_loss=0.01588, audio_tagging_loss=0.009301, over 3050380.02 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:10:40,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1643533.3333333333, ans=0.125
2023-11-21 20:10:41,665 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.44 vs. limit=15.0
2023-11-21 20:10:57,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1643600.0, ans=0.1
2023-11-21 20:11:02,834 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246550
2023-11-21 20:11:05,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1643666.6666666667, ans=0.1
2023-11-21 20:11:05,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1643666.6666666667, ans=0.0
2023-11-21 20:11:08,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1643666.6666666667, ans=0.125
2023-11-21 20:11:17,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1643733.3333333333, ans=0.0
2023-11-21 20:11:30,731 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6100, loss[loss=0.07369, simple_loss=0.09697, pruned_loss=0.01599, audio_tagging_loss=0.009215, over 14986.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09555, pruned_loss=0.0161, audio_tagging_loss=0.009271, over 3045962.00 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:11:45,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.37 vs. limit=15.0
2023-11-21 20:11:47,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1643866.6666666667, ans=0.1
2023-11-21 20:12:01,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1643933.3333333333, ans=0.2
2023-11-21 20:12:03,544 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.714e+01 7.866e+01 8.426e+01 9.427e+01 1.141e+02, threshold=1.685e+02, percent-clipped=0.0
2023-11-21 20:12:06,083 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246600
2023-11-21 20:12:13,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1644000.0, ans=0.1
2023-11-21 20:12:18,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1644000.0, ans=0.125
2023-11-21 20:12:18,859 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.81 vs. limit=22.5
2023-11-21 20:12:34,402 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6150, loss[loss=0.06866, simple_loss=0.09187, pruned_loss=0.01335, audio_tagging_loss=0.00937, over 15468.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09479, pruned_loss=0.01599, audio_tagging_loss=0.00943, over 3036855.90 frames. ], batch size: 55, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:12:58,211 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.51 vs. limit=6.0
2023-11-21 20:13:05,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1644266.6666666667, ans=0.125
2023-11-21 20:13:10,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246650
2023-11-21 20:13:37,956 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6200, loss[loss=0.07329, simple_loss=0.09468, pruned_loss=0.0136, audio_tagging_loss=0.01235, over 16166.00 frames. ], tot_loss[loss=0.07276, simple_loss=0.09449, pruned_loss=0.01598, audio_tagging_loss=0.009534, over 3039786.73 frames. ], batch size: 60, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:13:53,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1644533.3333333333, ans=0.125
2023-11-21 20:13:59,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1644533.3333333333, ans=0.125
2023-11-21 20:14:00,360 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.33 vs. limit=12.0
2023-11-21 20:14:11,062 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.261e+01 8.724e+01 9.317e+01 1.600e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 20:14:11,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1644600.0, ans=0.125
2023-11-21 20:14:13,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246700
2023-11-21 20:14:14,237 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.54 vs. limit=15.0
2023-11-21 20:14:42,103 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6250, loss[loss=0.04598, simple_loss=0.06149, pruned_loss=0.006768, audio_tagging_loss=0.008468, over 15000.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09408, pruned_loss=0.01601, audio_tagging_loss=0.009645, over 3038879.94 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:14:47,559 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.46 vs. limit=15.0
2023-11-21 20:15:08,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1644933.3333333333, ans=15.0
2023-11-21 20:15:15,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1644933.3333333333, ans=0.0
2023-11-21 20:15:17,618 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246750
2023-11-21 20:15:23,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1645000.0, ans=0.125
2023-11-21 20:15:25,740 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.518e-03
2023-11-21 20:15:31,775 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.09 vs. limit=15.0
2023-11-21 20:15:45,985 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6300, loss[loss=0.04816, simple_loss=0.06281, pruned_loss=0.00731, audio_tagging_loss=0.009441, over 14934.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09519, pruned_loss=0.01623, audio_tagging_loss=0.009761, over 3042583.45 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:16:19,040 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.125e+01 8.837e+01 9.702e+01 1.272e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 20:16:20,062 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.73 vs. limit=15.0
2023-11-21 20:16:21,599 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246800
2023-11-21 20:16:29,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1645333.3333333333, ans=0.2
2023-11-21 20:16:44,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1645400.0, ans=0.125
2023-11-21 20:16:50,201 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6350, loss[loss=0.05812, simple_loss=0.07661, pruned_loss=0.01003, audio_tagging_loss=0.009778, over 15775.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09508, pruned_loss=0.01628, audio_tagging_loss=0.009815, over 3042830.98 frames. ], batch size: 63, lr: 3.26e-03, grad_scale: 8.0
2023-11-21 20:17:07,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1645533.3333333333, ans=0.125
2023-11-21 20:17:10,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1645533.3333333333, ans=0.1
2023-11-21 20:17:24,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1645600.0, ans=0.05
2023-11-21 20:17:26,546 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246850
2023-11-21 20:17:35,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1645666.6666666667, ans=0.035
2023-11-21 20:17:54,901 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.51 vs. limit=12.0
2023-11-21 20:17:55,215 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6400, loss[loss=0.07853, simple_loss=0.09215, pruned_loss=0.02086, audio_tagging_loss=0.01159, over 15190.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09572, pruned_loss=0.01642, audio_tagging_loss=0.009839, over 3035210.58 frames. ], batch size: 56, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:17:56,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1645800.0, ans=0.125
2023-11-21 20:18:06,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1645866.6666666667, ans=0.0
2023-11-21 20:18:22,457 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.43 vs. limit=15.0
2023-11-21 20:18:27,622 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.810e+01 7.956e+01 8.478e+01 9.124e+01 1.124e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-21 20:18:30,232 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246900
2023-11-21 20:18:37,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1646000.0, ans=15.0
2023-11-21 20:18:47,138 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.81 vs. limit=15.0
2023-11-21 20:18:58,485 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6450, loss[loss=0.06684, simple_loss=0.08634, pruned_loss=0.01314, audio_tagging_loss=0.01053, over 14594.00 frames. ], tot_loss[loss=0.07379, simple_loss=0.09535, pruned_loss=0.01629, audio_tagging_loss=0.009818, over 3032761.28 frames. ], batch size: 57, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:19:06,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1646133.3333333333, ans=0.125
2023-11-21 20:19:11,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1646200.0, ans=0.125
2023-11-21 20:19:18,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1646200.0, ans=0.125
2023-11-21 20:19:21,694 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.22 vs. limit=15.0
2023-11-21 20:19:34,204 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.61 vs. limit=15.0
2023-11-21 20:19:34,744 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 246950
2023-11-21 20:19:50,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1646400.0, ans=0.0
2023-11-21 20:19:54,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1646400.0, ans=0.035
2023-11-21 20:20:02,053 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6500, loss[loss=0.08465, simple_loss=0.1051, pruned_loss=0.02273, audio_tagging_loss=0.009358, over 15336.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09588, pruned_loss=0.0164, audio_tagging_loss=0.009706, over 3038373.21 frames. ], batch size: 58, lr: 3.26e-03, grad_scale: 16.0
2023-11-21 20:20:12,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1646466.6666666667, ans=0.0
2023-11-21 20:20:22,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1646533.3333333333, ans=0.125
2023-11-21 20:20:27,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1646600.0, ans=0.125
2023-11-21 20:20:35,573 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.211e+01 8.118e+01 8.877e+01 9.588e+01 1.151e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-21 20:20:38,755 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247000
2023-11-21 20:20:40,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1646666.6666666667, ans=0.1
2023-11-21 20:20:46,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1646666.6666666667, ans=0.125
2023-11-21 20:21:04,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1646733.3333333333, ans=0.0
2023-11-21 20:21:07,048 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6550, loss[loss=0.07524, simple_loss=0.1034, pruned_loss=0.01561, audio_tagging_loss=0.007916, over 15935.00 frames. ], tot_loss[loss=0.07393, simple_loss=0.09612, pruned_loss=0.01632, audio_tagging_loss=0.009552, over 3044178.40 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:21:14,755 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.04 vs. limit=15.0
2023-11-21 20:21:15,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1646800.0, ans=0.0
2023-11-21 20:21:41,111 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:21:43,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247050
2023-11-21 20:22:11,499 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6600, loss[loss=0.06773, simple_loss=0.09295, pruned_loss=0.01516, audio_tagging_loss=0.006096, over 14109.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09551, pruned_loss=0.01624, audio_tagging_loss=0.009415, over 3035635.45 frames. ], batch size: 53, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:22:13,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.14 vs. limit=15.0
2023-11-21 20:22:30,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1647200.0, ans=0.0
2023-11-21 20:22:31,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.74 vs. limit=22.5
2023-11-21 20:22:32,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1647200.0, ans=0.125
2023-11-21 20:22:44,400 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.213e+01 8.717e+01 9.748e+01 1.869e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 20:22:47,532 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247100
2023-11-21 20:23:15,561 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6650, loss[loss=0.05824, simple_loss=0.07639, pruned_loss=0.01115, audio_tagging_loss=0.008896, over 14553.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09552, pruned_loss=0.01604, audio_tagging_loss=0.009374, over 3035512.11 frames. ], batch size: 54, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:23:34,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1647533.3333333333, ans=0.125
2023-11-21 20:23:47,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1647600.0, ans=0.0
2023-11-21 20:23:49,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1647600.0, ans=0.0
2023-11-21 20:23:50,790 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247150
2023-11-21 20:24:03,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1647666.6666666667, ans=0.04949747468305833
2023-11-21 20:24:10,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1647733.3333333333, ans=0.1
2023-11-21 20:24:18,050 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6700, loss[loss=0.06241, simple_loss=0.07601, pruned_loss=0.01195, audio_tagging_loss=0.01245, over 14970.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.09534, pruned_loss=0.01606, audio_tagging_loss=0.009289, over 3035897.62 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:24:28,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1647800.0, ans=0.0
2023-11-21 20:24:34,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1647866.6666666667, ans=0.125
2023-11-21 20:24:35,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.97 vs. limit=15.0
2023-11-21 20:24:46,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1647933.3333333333, ans=0.125
2023-11-21 20:24:51,829 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.787e+01 8.102e+01 8.678e+01 9.415e+01 1.201e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-21 20:24:54,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247200
2023-11-21 20:25:10,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1648066.6666666667, ans=0.125
2023-11-21 20:25:20,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.50 vs. limit=22.5
2023-11-21 20:25:23,153 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6750, loss[loss=0.06738, simple_loss=0.09413, pruned_loss=0.01249, audio_tagging_loss=0.007831, over 16454.00 frames. ], tot_loss[loss=0.073, simple_loss=0.09512, pruned_loss=0.01616, audio_tagging_loss=0.00928, over 3030293.56 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:25:35,455 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.34 vs. limit=22.5
2023-11-21 20:25:43,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1648200.0, ans=0.125
2023-11-21 20:25:45,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1648200.0, ans=0.125
2023-11-21 20:25:47,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1648266.6666666667, ans=0.1
2023-11-21 20:25:48,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1648266.6666666667, ans=0.1
2023-11-21 20:25:58,240 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247250
2023-11-21 20:26:06,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1648333.3333333333, ans=0.125
2023-11-21 20:26:21,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1648400.0, ans=0.125
2023-11-21 20:26:26,350 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6800, loss[loss=0.07187, simple_loss=0.09919, pruned_loss=0.01387, audio_tagging_loss=0.008402, over 15751.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.09523, pruned_loss=0.01623, audio_tagging_loss=0.009296, over 3033446.10 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:26:57,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1648600.0, ans=0.0
2023-11-21 20:26:58,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1648600.0, ans=0.125
2023-11-21 20:26:59,646 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 7.919e+01 8.607e+01 9.295e+01 1.340e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-21 20:27:02,180 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247300
2023-11-21 20:27:29,484 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6850, loss[loss=0.05146, simple_loss=0.05525, pruned_loss=0.01135, audio_tagging_loss=0.01249, over 15972.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09567, pruned_loss=0.0162, audio_tagging_loss=0.009275, over 3035161.30 frames. ], batch size: 65, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:27:33,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1648800.0, ans=0.125
2023-11-21 20:27:56,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1648933.3333333333, ans=0.1
2023-11-21 20:28:06,028 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247350
2023-11-21 20:28:21,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1649066.6666666667, ans=0.125
2023-11-21 20:28:22,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1649066.6666666667, ans=0.0
2023-11-21 20:28:29,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1649066.6666666667, ans=0.2
2023-11-21 20:28:33,849 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6900, loss[loss=0.0749, simple_loss=0.09396, pruned_loss=0.01817, audio_tagging_loss=0.009755, over 13386.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09512, pruned_loss=0.016, audio_tagging_loss=0.009251, over 3040312.73 frames. ], batch size: 52, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:28:49,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1649200.0, ans=0.125
2023-11-21 20:29:08,800 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.897e+01 8.008e+01 8.614e+01 9.356e+01 1.357e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 20:29:08,944 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247400
2023-11-21 20:29:23,615 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:29:37,986 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 6950, loss[loss=0.05404, simple_loss=0.06952, pruned_loss=0.01042, audio_tagging_loss=0.008854, over 14879.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09536, pruned_loss=0.01595, audio_tagging_loss=0.009272, over 3039756.17 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:30:01,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1649600.0, ans=0.125
2023-11-21 20:30:02,637 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.56 vs. limit=5.0
2023-11-21 20:30:12,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1649600.0, ans=0.125
2023-11-21 20:30:13,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247450
2023-11-21 20:30:16,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1649666.6666666667, ans=0.125
2023-11-21 20:30:16,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1649666.6666666667, ans=0.125
2023-11-21 20:30:24,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1649666.6666666667, ans=0.0
2023-11-21 20:30:26,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.36 vs. limit=22.5
2023-11-21 20:30:41,559 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7000, loss[loss=0.0664, simple_loss=0.0803, pruned_loss=0.01379, audio_tagging_loss=0.01247, over 14130.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09506, pruned_loss=0.01585, audio_tagging_loss=0.009348, over 3038986.16 frames. ], batch size: 55, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:30:48,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1649800.0, ans=6.0
2023-11-21 20:30:49,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1649800.0, ans=0.0
2023-11-21 20:30:56,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1649866.6666666667, ans=0.125
2023-11-21 20:31:00,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1649866.6666666667, ans=0.1
2023-11-21 20:31:02,159 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.09 vs. limit=10.0
2023-11-21 20:31:03,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1649866.6666666667, ans=0.125
2023-11-21 20:31:07,941 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.76 vs. limit=6.0
2023-11-21 20:31:18,705 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 7.928e+01 8.493e+01 9.155e+01 1.182e+02, threshold=1.699e+02, percent-clipped=0.0
2023-11-21 20:31:18,859 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247500
2023-11-21 20:31:26,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1650000.0, ans=0.1
2023-11-21 20:31:35,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1650066.6666666667, ans=0.1
2023-11-21 20:31:46,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1650133.3333333333, ans=0.09899494936611666
2023-11-21 20:31:47,107 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7050, loss[loss=0.06574, simple_loss=0.07907, pruned_loss=0.01504, audio_tagging_loss=0.01117, over 15062.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.09458, pruned_loss=0.01582, audio_tagging_loss=0.009395, over 3042617.89 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:32:03,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1650200.0, ans=0.0
2023-11-21 20:32:16,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1650266.6666666667, ans=0.1
2023-11-21 20:32:22,488 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247550
2023-11-21 20:32:51,799 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7100, loss[loss=0.08116, simple_loss=0.115, pruned_loss=0.01513, audio_tagging_loss=0.00853, over 16405.00 frames. ], tot_loss[loss=0.07296, simple_loss=0.0952, pruned_loss=0.01591, audio_tagging_loss=0.009452, over 3046677.59 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:33:05,842 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.21 vs. limit=15.0
2023-11-21 20:33:10,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1650533.3333333333, ans=0.125
2023-11-21 20:33:26,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1650600.0, ans=0.125
2023-11-21 20:33:27,099 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.401e+01 8.093e+01 8.713e+01 9.346e+01 1.337e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 20:33:27,231 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247600
2023-11-21 20:33:38,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1650666.6666666667, ans=0.0
2023-11-21 20:33:52,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.44 vs. limit=15.0
2023-11-21 20:33:54,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1650800.0, ans=0.0
2023-11-21 20:33:55,490 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7150, loss[loss=0.06659, simple_loss=0.08729, pruned_loss=0.01268, audio_tagging_loss=0.01026, over 15793.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09505, pruned_loss=0.01583, audio_tagging_loss=0.009543, over 3047121.79 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:34:04,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1650800.0, ans=0.125
2023-11-21 20:34:21,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1650933.3333333333, ans=0.0
2023-11-21 20:34:32,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247650
2023-11-21 20:34:40,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1651000.0, ans=0.0
2023-11-21 20:35:00,016 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7200, loss[loss=0.06964, simple_loss=0.08656, pruned_loss=0.01609, audio_tagging_loss=0.01027, over 15182.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.09614, pruned_loss=0.01607, audio_tagging_loss=0.009564, over 3046861.18 frames. ], batch size: 58, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:35:06,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.30 vs. limit=15.0
2023-11-21 20:35:07,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1651133.3333333333, ans=15.0
2023-11-21 20:35:16,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1651200.0, ans=0.125
2023-11-21 20:35:18,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1651200.0, ans=0.125
2023-11-21 20:35:20,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1651200.0, ans=0.0
2023-11-21 20:35:35,370 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.318e+01 8.484e+01 9.159e+01 1.010e+02 1.295e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-21 20:35:35,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247700
2023-11-21 20:35:49,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1651333.3333333333, ans=0.125
2023-11-21 20:35:57,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1651400.0, ans=0.125
2023-11-21 20:36:03,892 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7250, loss[loss=0.07873, simple_loss=0.0997, pruned_loss=0.02067, audio_tagging_loss=0.008207, over 15303.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09541, pruned_loss=0.01591, audio_tagging_loss=0.009606, over 3044621.96 frames. ], batch size: 58, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:36:20,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1651533.3333333333, ans=0.015
2023-11-21 20:36:32,512 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.80 vs. limit=10.0
2023-11-21 20:36:39,742 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247750
2023-11-21 20:36:57,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1651733.3333333333, ans=0.1
2023-11-21 20:37:07,641 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7300, loss[loss=0.07542, simple_loss=0.09576, pruned_loss=0.0189, audio_tagging_loss=0.008635, over 15579.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09511, pruned_loss=0.0159, audio_tagging_loss=0.009603, over 3040946.41 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:37:16,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff2.min_abs, batch_count=1651800.0, ans=0.1
2023-11-21 20:37:28,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1651866.6666666667, ans=0.125
2023-11-21 20:37:43,491 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.100e+01 8.051e+01 8.700e+01 9.385e+01 1.347e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 20:37:43,637 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247800
2023-11-21 20:38:09,646 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.51 vs. limit=6.0
2023-11-21 20:38:12,560 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7350, loss[loss=0.06025, simple_loss=0.08021, pruned_loss=0.009577, audio_tagging_loss=0.01057, over 15052.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09404, pruned_loss=0.01575, audio_tagging_loss=0.009567, over 3035610.21 frames. ], batch size: 58, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:38:17,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1652133.3333333333, ans=0.1
2023-11-21 20:38:19,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=1652133.3333333333, ans=0.2
2023-11-21 20:38:21,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1652133.3333333333, ans=0.125
2023-11-21 20:38:21,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1652133.3333333333, ans=0.125
2023-11-21 20:38:30,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1652200.0, ans=0.0
2023-11-21 20:38:31,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1652200.0, ans=0.0
2023-11-21 20:38:34,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1652200.0, ans=0.125
2023-11-21 20:38:46,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1652266.6666666667, ans=0.2
2023-11-21 20:38:47,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247850
2023-11-21 20:39:15,989 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7400, loss[loss=0.05139, simple_loss=0.06022, pruned_loss=0.01052, audio_tagging_loss=0.01076, over 14443.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09382, pruned_loss=0.01558, audio_tagging_loss=0.009417, over 3031600.12 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:39:42,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1652600.0, ans=0.125
2023-11-21 20:39:51,553 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 8.103e+01 8.809e+01 9.626e+01 1.321e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 20:39:51,705 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247900
2023-11-21 20:40:00,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.92 vs. limit=15.0
2023-11-21 20:40:04,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1652666.6666666667, ans=0.125
2023-11-21 20:40:08,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1652733.3333333333, ans=0.2
2023-11-21 20:40:19,644 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7450, loss[loss=0.07147, simple_loss=0.1073, pruned_loss=0.01066, audio_tagging_loss=0.007144, over 15453.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09476, pruned_loss=0.01585, audio_tagging_loss=0.00937, over 3041642.43 frames. ], batch size: 55, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:40:55,210 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 247950
2023-11-21 20:40:57,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1653000.0, ans=0.125
2023-11-21 20:41:11,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1653066.6666666667, ans=0.125
2023-11-21 20:41:20,727 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.44 vs. limit=22.5
2023-11-21 20:41:22,574 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7500, loss[loss=0.06225, simple_loss=0.08452, pruned_loss=0.01388, audio_tagging_loss=0.006111, over 14738.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09474, pruned_loss=0.01588, audio_tagging_loss=0.009358, over 3032134.28 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:41:22,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1653133.3333333333, ans=0.125
2023-11-21 20:41:43,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1653200.0, ans=0.125
2023-11-21 20:41:49,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1653266.6666666667, ans=0.0
2023-11-21 20:41:57,967 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.210e+01 8.790e+01 9.485e+01 1.319e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 20:41:58,164 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248000
2023-11-21 20:42:00,314 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-248000.pt
2023-11-21 20:42:22,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1653400.0, ans=0.125
2023-11-21 20:42:29,577 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7550, loss[loss=0.07035, simple_loss=0.09117, pruned_loss=0.015, audio_tagging_loss=0.009767, over 15288.00 frames. ], tot_loss[loss=0.07288, simple_loss=0.09497, pruned_loss=0.01602, audio_tagging_loss=0.009372, over 3038717.68 frames. ], batch size: 57, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:42:29,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1653466.6666666667, ans=0.2
2023-11-21 20:42:36,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1653466.6666666667, ans=0.0
2023-11-21 20:42:41,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1653533.3333333333, ans=0.125
2023-11-21 20:42:44,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1653533.3333333333, ans=0.0
2023-11-21 20:42:51,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1653533.3333333333, ans=0.1
2023-11-21 20:43:05,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248050
2023-11-21 20:43:17,169 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.84 vs. limit=22.5
2023-11-21 20:43:19,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1653733.3333333333, ans=0.0
2023-11-21 20:43:23,816 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.80 vs. limit=15.0
2023-11-21 20:43:32,798 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7600, loss[loss=0.05791, simple_loss=0.07938, pruned_loss=0.007624, audio_tagging_loss=0.01059, over 15183.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09487, pruned_loss=0.01593, audio_tagging_loss=0.00941, over 3035766.70 frames. ], batch size: 55, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:43:55,179 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:44:08,216 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.765e+01 8.151e+01 8.758e+01 9.560e+01 1.334e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 20:44:08,380 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248100
2023-11-21 20:44:35,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1654133.3333333333, ans=0.125
2023-11-21 20:44:36,150 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7650, loss[loss=0.08787, simple_loss=0.1175, pruned_loss=0.01908, audio_tagging_loss=0.01003, over 16041.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09584, pruned_loss=0.01605, audio_tagging_loss=0.009314, over 3037238.31 frames. ], batch size: 59, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:44:53,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1654200.0, ans=0.125
2023-11-21 20:45:01,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1654266.6666666667, ans=0.125
2023-11-21 20:45:11,786 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248150
2023-11-21 20:45:16,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1654333.3333333333, ans=0.125
2023-11-21 20:45:29,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1654400.0, ans=0.125
2023-11-21 20:45:31,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1654400.0, ans=0.0
2023-11-21 20:45:40,028 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7700, loss[loss=0.0818, simple_loss=0.1058, pruned_loss=0.01917, audio_tagging_loss=0.009749, over 14603.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09567, pruned_loss=0.0162, audio_tagging_loss=0.009282, over 3036339.84 frames. ], batch size: 53, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:46:15,471 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.008e+01 8.521e+01 9.342e+01 1.167e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-21 20:46:15,622 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248200
2023-11-21 20:46:43,456 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7750, loss[loss=0.07067, simple_loss=0.09727, pruned_loss=0.01604, audio_tagging_loss=0.005997, over 15703.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09458, pruned_loss=0.01595, audio_tagging_loss=0.009463, over 3033142.66 frames. ], batch size: 60, lr: 3.25e-03, grad_scale: 32.0
2023-11-21 20:46:59,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1654866.6666666667, ans=0.0
2023-11-21 20:47:13,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1654933.3333333333, ans=0.1
2023-11-21 20:47:14,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1654933.3333333333, ans=0.09899494936611666
2023-11-21 20:47:19,528 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248250
2023-11-21 20:47:31,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1655000.0, ans=0.125
2023-11-21 20:47:31,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1655000.0, ans=0.125
2023-11-21 20:47:34,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1655066.6666666667, ans=0.0
2023-11-21 20:47:42,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1655066.6666666667, ans=0.125
2023-11-21 20:47:46,962 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7800, loss[loss=0.07175, simple_loss=0.08912, pruned_loss=0.01763, audio_tagging_loss=0.009564, over 14592.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09551, pruned_loss=0.01611, audio_tagging_loss=0.009512, over 3035334.72 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:48:04,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1655200.0, ans=0.0
2023-11-21 20:48:11,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1655200.0, ans=0.125
2023-11-21 20:48:23,359 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248300
2023-11-21 20:48:24,376 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.033e+01 8.572e+01 9.400e+01 1.167e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-21 20:48:50,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1655466.6666666667, ans=0.125
2023-11-21 20:48:51,006 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7850, loss[loss=0.06757, simple_loss=0.08263, pruned_loss=0.01249, audio_tagging_loss=0.01376, over 14674.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09555, pruned_loss=0.01615, audio_tagging_loss=0.009571, over 3031162.01 frames. ], batch size: 56, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:49:10,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1655533.3333333333, ans=0.07
2023-11-21 20:49:23,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1655600.0, ans=0.125
2023-11-21 20:49:25,726 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248350
2023-11-21 20:49:39,470 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.66 vs. limit=6.0
2023-11-21 20:49:41,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1655733.3333333333, ans=0.0
2023-11-21 20:49:53,826 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7900, loss[loss=0.06956, simple_loss=0.08976, pruned_loss=0.01452, audio_tagging_loss=0.01016, over 15051.00 frames. ], tot_loss[loss=0.07396, simple_loss=0.09628, pruned_loss=0.01625, audio_tagging_loss=0.009573, over 3036212.43 frames. ], batch size: 55, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:50:00,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1655800.0, ans=0.0
2023-11-21 20:50:03,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1655800.0, ans=0.125
2023-11-21 20:50:18,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1655933.3333333333, ans=0.125
2023-11-21 20:50:29,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248400
2023-11-21 20:50:32,497 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.166e+01 8.800e+01 9.422e+01 1.274e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-21 20:50:41,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1656000.0, ans=0.1
2023-11-21 20:50:47,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1656066.6666666667, ans=0.0
2023-11-21 20:50:57,271 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 7950, loss[loss=0.06468, simple_loss=0.08426, pruned_loss=0.01345, audio_tagging_loss=0.009109, over 15043.00 frames. ], tot_loss[loss=0.07418, simple_loss=0.09626, pruned_loss=0.01638, audio_tagging_loss=0.009662, over 3038872.16 frames. ], batch size: 54, lr: 3.25e-03, grad_scale: 8.0
2023-11-21 20:51:05,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1656133.3333333333, ans=0.2
2023-11-21 20:51:13,905 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:51:15,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1656200.0, ans=0.125
2023-11-21 20:51:25,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1656266.6666666667, ans=0.0
2023-11-21 20:51:34,138 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248450
2023-11-21 20:52:02,352 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8000, loss[loss=0.07186, simple_loss=0.09123, pruned_loss=0.01556, audio_tagging_loss=0.01069, over 15335.00 frames. ], tot_loss[loss=0.07469, simple_loss=0.09701, pruned_loss=0.01655, audio_tagging_loss=0.009639, over 3043783.20 frames. ], batch size: 62, lr: 3.25e-03, grad_scale: 16.0
2023-11-21 20:52:17,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1656533.3333333333, ans=0.125
2023-11-21 20:52:22,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1656533.3333333333, ans=0.125
2023-11-21 20:52:24,297 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.29 vs. limit=10.0
2023-11-21 20:52:27,983 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.24 vs. limit=15.0
2023-11-21 20:52:37,010 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248500
2023-11-21 20:52:39,318 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.016e+01 8.965e+01 9.786e+01 1.299e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 20:52:45,694 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=6.95 vs. limit=15.0
2023-11-21 20:52:55,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1656733.3333333333, ans=0.5
2023-11-21 20:53:01,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1656733.3333333333, ans=0.0
2023-11-21 20:53:02,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1656733.3333333333, ans=0.125
2023-11-21 20:53:05,621 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8050, loss[loss=0.06792, simple_loss=0.08992, pruned_loss=0.01474, audio_tagging_loss=0.008228, over 15750.00 frames. ], tot_loss[loss=0.07454, simple_loss=0.09676, pruned_loss=0.01648, audio_tagging_loss=0.009686, over 3043187.25 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:53:08,418 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 20:53:11,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1656800.0, ans=0.0
2023-11-21 20:53:20,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1656866.6666666667, ans=0.125
2023-11-21 20:53:34,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1656933.3333333333, ans=0.0
2023-11-21 20:53:40,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1656933.3333333333, ans=0.125
2023-11-21 20:53:41,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248550
2023-11-21 20:53:47,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1657000.0, ans=0.125
2023-11-21 20:54:06,810 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.21 vs. limit=15.0
2023-11-21 20:54:08,560 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8100, loss[loss=0.08054, simple_loss=0.1059, pruned_loss=0.01821, audio_tagging_loss=0.009403, over 15461.00 frames. ], tot_loss[loss=0.07463, simple_loss=0.09704, pruned_loss=0.0166, audio_tagging_loss=0.009505, over 3046364.33 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:54:18,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1657133.3333333333, ans=0.125
2023-11-21 20:54:44,701 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248600
2023-11-21 20:54:47,362 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.022e+01 8.595e+01 9.139e+01 1.165e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-21 20:54:47,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1657333.3333333333, ans=0.0
2023-11-21 20:54:50,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1657333.3333333333, ans=10.0
2023-11-21 20:55:02,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1657400.0, ans=0.0
2023-11-21 20:55:06,427 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.44 vs. limit=15.0
2023-11-21 20:55:10,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.26 vs. limit=15.0
2023-11-21 20:55:12,712 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8150, loss[loss=0.07663, simple_loss=0.1058, pruned_loss=0.01687, audio_tagging_loss=0.00685, over 16832.00 frames. ], tot_loss[loss=0.07396, simple_loss=0.09624, pruned_loss=0.01638, audio_tagging_loss=0.00945, over 3046938.36 frames. ], batch size: 61, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:55:20,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1657466.6666666667, ans=0.125
2023-11-21 20:55:23,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1657466.6666666667, ans=0.0
2023-11-21 20:55:47,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248650
2023-11-21 20:55:54,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1657666.6666666667, ans=0.1
2023-11-21 20:56:16,194 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8200, loss[loss=0.07726, simple_loss=0.09512, pruned_loss=0.01904, audio_tagging_loss=0.01066, over 14861.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09582, pruned_loss=0.01636, audio_tagging_loss=0.009375, over 3050957.17 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:56:16,256 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 20:56:21,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1657800.0, ans=0.125
2023-11-21 20:56:50,616 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248700
2023-11-21 20:56:50,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1657933.3333333333, ans=0.125
2023-11-21 20:56:52,860 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.429e+01 8.006e+01 8.771e+01 9.726e+01 1.237e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 20:56:57,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=1658000.0, ans=6.0
2023-11-21 20:57:11,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1658066.6666666667, ans=0.1
2023-11-21 20:57:18,315 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8250, loss[loss=0.07374, simple_loss=0.09046, pruned_loss=0.01702, audio_tagging_loss=0.01149, over 16884.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09556, pruned_loss=0.01632, audio_tagging_loss=0.009368, over 3051614.22 frames. ], batch size: 63, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:57:20,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1658133.3333333333, ans=0.1
2023-11-21 20:57:27,607 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.30 vs. limit=6.0
2023-11-21 20:57:54,936 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248750
2023-11-21 20:57:55,603 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.81 vs. limit=22.5
2023-11-21 20:57:56,710 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.24 vs. limit=15.0
2023-11-21 20:58:18,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1658400.0, ans=0.125
2023-11-21 20:58:22,525 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8300, loss[loss=0.0764, simple_loss=0.1026, pruned_loss=0.01783, audio_tagging_loss=0.007252, over 14440.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09573, pruned_loss=0.01618, audio_tagging_loss=0.009334, over 3050518.49 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:58:27,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1658466.6666666667, ans=0.0
2023-11-21 20:58:46,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1658533.3333333333, ans=0.0
2023-11-21 20:58:54,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1658600.0, ans=0.125
2023-11-21 20:58:58,048 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248800
2023-11-21 20:59:00,790 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.249e+01 8.152e+01 8.713e+01 9.319e+01 1.326e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 20:59:15,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1658733.3333333333, ans=0.2
2023-11-21 20:59:20,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1658733.3333333333, ans=0.125
2023-11-21 20:59:26,931 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8350, loss[loss=0.05769, simple_loss=0.06984, pruned_loss=0.01209, audio_tagging_loss=0.01067, over 15288.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09629, pruned_loss=0.01619, audio_tagging_loss=0.009235, over 3054443.39 frames. ], batch size: 57, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 20:59:27,680 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.71 vs. limit=6.0
2023-11-21 20:59:33,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1658800.0, ans=0.0
2023-11-21 20:59:36,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1658800.0, ans=0.2
2023-11-21 20:59:43,068 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.71 vs. limit=22.5
2023-11-21 20:59:47,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1658866.6666666667, ans=0.125
2023-11-21 20:59:51,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1658933.3333333333, ans=0.1
2023-11-21 20:59:54,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1658933.3333333333, ans=0.125
2023-11-21 21:00:02,722 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248850
2023-11-21 21:00:28,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1659066.6666666667, ans=0.0
2023-11-21 21:00:30,782 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8400, loss[loss=0.05924, simple_loss=0.0744, pruned_loss=0.01048, audio_tagging_loss=0.01156, over 16267.00 frames. ], tot_loss[loss=0.07314, simple_loss=0.0957, pruned_loss=0.01607, audio_tagging_loss=0.00922, over 3054558.93 frames. ], batch size: 62, lr: 3.24e-03, grad_scale: 32.0
2023-11-21 21:00:49,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.92 vs. limit=15.0
2023-11-21 21:01:00,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1659266.6666666667, ans=0.0
2023-11-21 21:01:03,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1659266.6666666667, ans=0.125
2023-11-21 21:01:03,435 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.76 vs. limit=12.0
2023-11-21 21:01:06,706 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248900
2023-11-21 21:01:10,210 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.422e+01 7.990e+01 8.773e+01 9.433e+01 1.119e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 21:01:16,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0
2023-11-21 21:01:22,705 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.65 vs. limit=12.0
2023-11-21 21:01:28,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1659400.0, ans=0.125
2023-11-21 21:01:29,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1659400.0, ans=0.125
2023-11-21 21:01:31,526 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.16 vs. limit=15.0
2023-11-21 21:01:33,140 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8450, loss[loss=0.08557, simple_loss=0.111, pruned_loss=0.0182, audio_tagging_loss=0.01188, over 15202.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09577, pruned_loss=0.01591, audio_tagging_loss=0.009303, over 3052877.51 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:01:36,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1659466.6666666667, ans=0.04949747468305833
2023-11-21 21:01:49,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1659533.3333333333, ans=0.025
2023-11-21 21:02:08,951 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 248950
2023-11-21 21:02:32,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1659733.3333333333, ans=0.0
2023-11-21 21:02:36,356 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8500, loss[loss=0.07198, simple_loss=0.09763, pruned_loss=0.01656, audio_tagging_loss=0.006603, over 15069.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.0954, pruned_loss=0.01593, audio_tagging_loss=0.009354, over 3052910.06 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:02:45,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1659800.0, ans=0.1
2023-11-21 21:02:52,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1659866.6666666667, ans=0.0
2023-11-21 21:02:59,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1659866.6666666667, ans=0.04949747468305833
2023-11-21 21:03:12,524 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249000
2023-11-21 21:03:15,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1660000.0, ans=0.125
2023-11-21 21:03:16,329 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.238e+01 7.932e+01 8.481e+01 9.270e+01 1.196e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-21 21:03:29,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1660066.6666666667, ans=0.1
2023-11-21 21:03:41,011 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8550, loss[loss=0.06452, simple_loss=0.09094, pruned_loss=0.01135, audio_tagging_loss=0.007698, over 15347.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09566, pruned_loss=0.01592, audio_tagging_loss=0.00949, over 3049568.06 frames. ], batch size: 60, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:03:54,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1660200.0, ans=0.0
2023-11-21 21:04:01,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1660200.0, ans=0.125
2023-11-21 21:04:04,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1660266.6666666667, ans=0.0
2023-11-21 21:04:11,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1660266.6666666667, ans=0.125
2023-11-21 21:04:12,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1660266.6666666667, ans=22.5
2023-11-21 21:04:16,332 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249050
2023-11-21 21:04:36,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1660400.0, ans=0.125
2023-11-21 21:04:39,323 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.48 vs. limit=15.0
2023-11-21 21:04:43,481 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8600, loss[loss=0.07002, simple_loss=0.0924, pruned_loss=0.01324, audio_tagging_loss=0.01057, over 15759.00 frames. ], tot_loss[loss=0.07384, simple_loss=0.09638, pruned_loss=0.01611, audio_tagging_loss=0.009549, over 3050528.48 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:04:49,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1660466.6666666667, ans=0.125
2023-11-21 21:04:59,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.91 vs. limit=10.0
2023-11-21 21:05:14,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1660600.0, ans=0.04949747468305833
2023-11-21 21:05:20,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249100
2023-11-21 21:05:24,798 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.123e+01 8.590e+01 9.454e+01 1.253e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-21 21:05:33,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1660733.3333333333, ans=0.125
2023-11-21 21:05:43,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1660733.3333333333, ans=0.125
2023-11-21 21:05:47,728 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8650, loss[loss=0.06071, simple_loss=0.07828, pruned_loss=0.01277, audio_tagging_loss=0.008801, over 14396.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09573, pruned_loss=0.01594, audio_tagging_loss=0.009611, over 3057768.29 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:05:59,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1660866.6666666667, ans=0.125
2023-11-21 21:06:23,160 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249150
2023-11-21 21:06:35,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1661000.0, ans=0.125
2023-11-21 21:06:35,887 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.45 vs. limit=15.0
2023-11-21 21:06:45,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1661066.6666666667, ans=0.025
2023-11-21 21:06:48,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.56 vs. limit=15.0
2023-11-21 21:06:51,492 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8700, loss[loss=0.07871, simple_loss=0.08741, pruned_loss=0.02137, audio_tagging_loss=0.01363, over 15908.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.09646, pruned_loss=0.01599, audio_tagging_loss=0.009582, over 3059009.09 frames. ], batch size: 61, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:06:56,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1661133.3333333333, ans=0.125
2023-11-21 21:07:04,983 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.71 vs. limit=22.5
2023-11-21 21:07:16,872 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.30 vs. limit=15.0
2023-11-21 21:07:27,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249200
2023-11-21 21:07:32,750 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.120e+01 8.963e+01 9.729e+01 1.356e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-21 21:07:55,269 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8750, loss[loss=0.0772, simple_loss=0.1007, pruned_loss=0.01745, audio_tagging_loss=0.009424, over 16344.00 frames. ], tot_loss[loss=0.07392, simple_loss=0.09648, pruned_loss=0.0161, audio_tagging_loss=0.009582, over 3053183.74 frames. ], batch size: 61, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:07:55,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1661466.6666666667, ans=0.0
2023-11-21 21:08:06,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1661466.6666666667, ans=0.125
2023-11-21 21:08:10,776 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.30 vs. limit=15.0
2023-11-21 21:08:12,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1661533.3333333333, ans=0.125
2023-11-21 21:08:14,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1661533.3333333333, ans=0.0
2023-11-21 21:08:27,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1661600.0, ans=0.1
2023-11-21 21:08:31,348 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249250
2023-11-21 21:08:31,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1661600.0, ans=0.125
2023-11-21 21:08:59,675 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8800, loss[loss=0.07875, simple_loss=0.1033, pruned_loss=0.01696, audio_tagging_loss=0.01012, over 14670.00 frames. ], tot_loss[loss=0.07397, simple_loss=0.09626, pruned_loss=0.01611, audio_tagging_loss=0.009731, over 3054299.72 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:09:21,722 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.94 vs. limit=15.0
2023-11-21 21:09:25,969 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.62 vs. limit=6.0
2023-11-21 21:09:35,375 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249300
2023-11-21 21:09:36,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1662000.0, ans=0.125
2023-11-21 21:09:40,734 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.249e+01 8.864e+01 9.702e+01 1.215e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-21 21:09:46,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1662000.0, ans=0.09899494936611666
2023-11-21 21:09:48,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1662000.0, ans=0.125
2023-11-21 21:10:03,857 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8850, loss[loss=0.0702, simple_loss=0.09985, pruned_loss=0.01042, audio_tagging_loss=0.009859, over 14659.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09549, pruned_loss=0.016, audio_tagging_loss=0.00986, over 3045607.87 frames. ], batch size: 53, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:10:04,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1662133.3333333333, ans=0.2
2023-11-21 21:10:06,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=1662133.3333333333, ans=0.2
2023-11-21 21:10:09,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1662133.3333333333, ans=0.07
2023-11-21 21:10:11,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=1662133.3333333333, ans=15.0
2023-11-21 21:10:15,538 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:10:20,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1662200.0, ans=0.125
2023-11-21 21:10:21,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1662200.0, ans=0.125
2023-11-21 21:10:28,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1662266.6666666667, ans=0.0
2023-11-21 21:10:31,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1662266.6666666667, ans=0.125
2023-11-21 21:10:39,609 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249350
2023-11-21 21:10:50,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1662333.3333333333, ans=0.125
2023-11-21 21:10:51,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1662333.3333333333, ans=0.125
2023-11-21 21:11:04,285 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.61 vs. limit=15.0
2023-11-21 21:11:07,387 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8900, loss[loss=0.07841, simple_loss=0.105, pruned_loss=0.01864, audio_tagging_loss=0.007285, over 15261.00 frames. ], tot_loss[loss=0.07375, simple_loss=0.09588, pruned_loss=0.01613, audio_tagging_loss=0.009683, over 3051646.32 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:11:34,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1662600.0, ans=0.125
2023-11-21 21:11:41,091 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.73 vs. limit=15.0
2023-11-21 21:11:43,047 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249400
2023-11-21 21:11:49,421 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.502e+01 8.199e+01 8.684e+01 9.586e+01 1.559e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 21:12:05,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1662733.3333333333, ans=0.125
2023-11-21 21:12:07,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1662733.3333333333, ans=0.125
2023-11-21 21:12:10,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1662733.3333333333, ans=0.125
2023-11-21 21:12:12,062 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 8950, loss[loss=0.07388, simple_loss=0.09141, pruned_loss=0.01662, audio_tagging_loss=0.01155, over 15574.00 frames. ], tot_loss[loss=0.07352, simple_loss=0.09563, pruned_loss=0.01614, audio_tagging_loss=0.009566, over 3053067.17 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:12:20,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1662800.0, ans=0.2
2023-11-21 21:12:35,326 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.97 vs. limit=10.0
2023-11-21 21:12:47,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249450
2023-11-21 21:13:15,728 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9000, loss[loss=0.09905, simple_loss=0.1365, pruned_loss=0.0236, audio_tagging_loss=0.007225, over 15829.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09549, pruned_loss=0.01606, audio_tagging_loss=0.009502, over 3054659.30 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:13:15,731 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 21:13:56,856 INFO [train_asr.py:1253] (0/4) Epoch 21, validation: loss=0.06003, simple_loss=0.05194, pruned_loss=0.005168, audio_tagging_loss=0.0289, over 4681554.00 frames. 
2023-11-21 21:13:56,856 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 21:14:19,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1663200.0, ans=0.035
2023-11-21 21:14:32,764 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249500
2023-11-21 21:14:38,735 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.929e+01 7.983e+01 8.777e+01 1.012e+02 1.174e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-21 21:14:45,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1663333.3333333333, ans=0.0
2023-11-21 21:15:01,139 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9050, loss[loss=0.07725, simple_loss=0.09325, pruned_loss=0.02186, audio_tagging_loss=0.008767, over 16129.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09649, pruned_loss=0.01607, audio_tagging_loss=0.009357, over 3064617.79 frames. ], batch size: 62, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:15:36,361 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.18 vs. limit=15.0
2023-11-21 21:15:36,756 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249550
2023-11-21 21:15:44,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1663666.6666666667, ans=0.125
2023-11-21 21:15:48,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1663666.6666666667, ans=0.125
2023-11-21 21:15:51,600 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.54 vs. limit=6.0
2023-11-21 21:16:04,958 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9100, loss[loss=0.08146, simple_loss=0.1134, pruned_loss=0.01732, audio_tagging_loss=0.007453, over 14954.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.09648, pruned_loss=0.01601, audio_tagging_loss=0.009289, over 3061309.04 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:16:07,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1663800.0, ans=0.0
2023-11-21 21:16:30,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1663933.3333333333, ans=0.125
2023-11-21 21:16:35,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1663933.3333333333, ans=0.125
2023-11-21 21:16:40,988 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249600
2023-11-21 21:16:47,972 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.138e+01 8.810e+01 9.417e+01 1.142e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 21:16:49,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1664000.0, ans=0.035
2023-11-21 21:16:52,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1664000.0, ans=0.0
2023-11-21 21:16:53,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=1664000.0, ans=15.0
2023-11-21 21:17:00,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1664066.6666666667, ans=0.07
2023-11-21 21:17:08,717 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9150, loss[loss=0.06855, simple_loss=0.08796, pruned_loss=0.0132, audio_tagging_loss=0.01137, over 14459.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09543, pruned_loss=0.01591, audio_tagging_loss=0.009324, over 3055202.46 frames. ], batch size: 54, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:17:27,363 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.30 vs. limit=10.0
2023-11-21 21:17:33,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1664200.0, ans=0.125
2023-11-21 21:17:45,998 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249650
2023-11-21 21:18:06,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1664400.0, ans=0.125
2023-11-21 21:18:14,538 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9200, loss[loss=0.06703, simple_loss=0.08552, pruned_loss=0.0148, audio_tagging_loss=0.009473, over 15872.00 frames. ], tot_loss[loss=0.07361, simple_loss=0.09664, pruned_loss=0.01611, audio_tagging_loss=0.009183, over 3065864.27 frames. ], batch size: 61, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:18:22,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1664466.6666666667, ans=0.1
2023-11-21 21:18:30,721 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.01 vs. limit=6.0
2023-11-21 21:18:48,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1664600.0, ans=0.125
2023-11-21 21:18:49,967 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249700
2023-11-21 21:18:50,571 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.82 vs. limit=10.0
2023-11-21 21:18:56,915 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.394e+01 8.125e+01 8.694e+01 9.340e+01 1.162e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 21:19:09,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1664733.3333333333, ans=10.0
2023-11-21 21:19:19,570 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9250, loss[loss=0.058, simple_loss=0.08075, pruned_loss=0.007909, audio_tagging_loss=0.009715, over 15153.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09602, pruned_loss=0.01597, audio_tagging_loss=0.00918, over 3065327.81 frames. ], batch size: 56, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:19:55,929 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249750
2023-11-21 21:20:09,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1665000.0, ans=0.2
2023-11-21 21:20:09,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1665000.0, ans=0.0
2023-11-21 21:20:10,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1665066.6666666667, ans=0.0
2023-11-21 21:20:14,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1665066.6666666667, ans=0.125
2023-11-21 21:20:23,916 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9300, loss[loss=0.07825, simple_loss=0.1091, pruned_loss=0.0175, audio_tagging_loss=0.006209, over 15675.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09569, pruned_loss=0.01588, audio_tagging_loss=0.009335, over 3065825.47 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:20:24,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1665133.3333333333, ans=0.0
2023-11-21 21:20:37,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1665200.0, ans=0.0
2023-11-21 21:21:00,372 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249800
2023-11-21 21:21:06,633 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.134e+01 8.648e+01 9.186e+01 1.314e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-21 21:21:24,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1665400.0, ans=0.2
2023-11-21 21:21:29,036 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9350, loss[loss=0.08748, simple_loss=0.1184, pruned_loss=0.01947, audio_tagging_loss=0.008822, over 15896.00 frames. ], tot_loss[loss=0.07379, simple_loss=0.09655, pruned_loss=0.01619, audio_tagging_loss=0.009319, over 3057080.88 frames. ], batch size: 58, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:21:34,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1665466.6666666667, ans=0.125
2023-11-21 21:21:43,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1665533.3333333333, ans=0.0
2023-11-21 21:21:53,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1665600.0, ans=0.125
2023-11-21 21:22:03,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1665600.0, ans=0.125
2023-11-21 21:22:03,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1665600.0, ans=0.2
2023-11-21 21:22:04,379 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249850
2023-11-21 21:22:12,898 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.52 vs. limit=22.5
2023-11-21 21:22:33,838 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9400, loss[loss=0.08359, simple_loss=0.1069, pruned_loss=0.0216, audio_tagging_loss=0.008527, over 15098.00 frames. ], tot_loss[loss=0.07399, simple_loss=0.09633, pruned_loss=0.01628, audio_tagging_loss=0.00955, over 3050627.02 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:23:00,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1665933.3333333333, ans=0.1
2023-11-21 21:23:07,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1665933.3333333333, ans=0.125
2023-11-21 21:23:09,763 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249900
2023-11-21 21:23:16,399 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.499e+01 7.941e+01 8.716e+01 9.449e+01 2.230e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-21 21:23:16,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1666000.0, ans=0.2
2023-11-21 21:23:23,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1666000.0, ans=0.125
2023-11-21 21:23:24,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1666066.6666666667, ans=0.125
2023-11-21 21:23:24,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1666066.6666666667, ans=0.1
2023-11-21 21:23:26,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1666066.6666666667, ans=0.0
2023-11-21 21:23:35,575 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:23:38,004 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9450, loss[loss=0.0718, simple_loss=0.09943, pruned_loss=0.01455, audio_tagging_loss=0.007535, over 14830.00 frames. ], tot_loss[loss=0.07404, simple_loss=0.09647, pruned_loss=0.01623, audio_tagging_loss=0.009576, over 3048648.45 frames. ], batch size: 55, lr: 3.24e-03, grad_scale: 16.0
2023-11-21 21:23:42,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1666133.3333333333, ans=0.125
2023-11-21 21:23:54,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1666200.0, ans=0.2
2023-11-21 21:24:00,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1666200.0, ans=0.125
2023-11-21 21:24:06,485 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.56 vs. limit=15.0
2023-11-21 21:24:14,527 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 249950
2023-11-21 21:24:29,825 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.35 vs. limit=22.5
2023-11-21 21:24:37,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1666400.0, ans=0.2
2023-11-21 21:24:38,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1666400.0, ans=0.125
2023-11-21 21:24:42,642 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9500, loss[loss=0.07204, simple_loss=0.09067, pruned_loss=0.01452, audio_tagging_loss=0.01218, over 15599.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09619, pruned_loss=0.01609, audio_tagging_loss=0.009664, over 3046742.78 frames. ], batch size: 60, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:24:44,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1666466.6666666667, ans=0.1
2023-11-21 21:24:47,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1666466.6666666667, ans=0.125
2023-11-21 21:24:50,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1666466.6666666667, ans=0.0
2023-11-21 21:24:51,880 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.57 vs. limit=6.0
2023-11-21 21:25:10,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1666600.0, ans=0.1
2023-11-21 21:25:16,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1666600.0, ans=0.125
2023-11-21 21:25:16,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.97 vs. limit=22.5
2023-11-21 21:25:18,561 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250000
2023-11-21 21:25:18,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1666600.0, ans=0.125
2023-11-21 21:25:21,987 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.29 vs. limit=15.0
2023-11-21 21:25:26,596 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.269e+01 8.539e+01 8.975e+01 9.633e+01 1.238e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-21 21:25:46,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1666800.0, ans=0.0
2023-11-21 21:25:47,976 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9550, loss[loss=0.07711, simple_loss=0.09755, pruned_loss=0.01825, audio_tagging_loss=0.01009, over 16264.00 frames. ], tot_loss[loss=0.07374, simple_loss=0.09605, pruned_loss=0.01609, audio_tagging_loss=0.009622, over 3044130.70 frames. ], batch size: 61, lr: 3.24e-03, grad_scale: 8.0
2023-11-21 21:25:57,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1666800.0, ans=0.125
2023-11-21 21:26:24,914 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250050
2023-11-21 21:26:53,395 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9600, loss[loss=0.05891, simple_loss=0.07689, pruned_loss=0.0109, audio_tagging_loss=0.009564, over 14511.00 frames. ], tot_loss[loss=0.07396, simple_loss=0.09619, pruned_loss=0.01625, audio_tagging_loss=0.009612, over 3042054.29 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:26:56,402 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.23 vs. limit=22.5
2023-11-21 21:27:09,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1667200.0, ans=0.0
2023-11-21 21:27:30,213 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250100
2023-11-21 21:27:38,191 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.555e+01 8.305e+01 8.780e+01 9.351e+01 2.082e+02, threshold=1.756e+02, percent-clipped=1.0
2023-11-21 21:27:58,567 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9650, loss[loss=0.06176, simple_loss=0.07141, pruned_loss=0.01321, audio_tagging_loss=0.01285, over 14960.00 frames. ], tot_loss[loss=0.07412, simple_loss=0.09666, pruned_loss=0.01623, audio_tagging_loss=0.009561, over 3039811.46 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:27:58,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1667466.6666666667, ans=0.1
2023-11-21 21:28:17,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1667533.3333333333, ans=0.0
2023-11-21 21:28:34,862 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250150
2023-11-21 21:28:47,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1667666.6666666667, ans=10.0
2023-11-21 21:28:49,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1667733.3333333333, ans=0.1
2023-11-21 21:29:02,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1667800.0, ans=0.0
2023-11-21 21:29:03,967 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9700, loss[loss=0.0655, simple_loss=0.09518, pruned_loss=0.01178, audio_tagging_loss=0.006125, over 14671.00 frames. ], tot_loss[loss=0.07421, simple_loss=0.09699, pruned_loss=0.01631, audio_tagging_loss=0.009402, over 3044325.75 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:29:34,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1667933.3333333333, ans=0.125
2023-11-21 21:29:40,201 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250200
2023-11-21 21:29:47,777 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.250e+01 8.691e+01 9.260e+01 1.203e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 21:30:09,302 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9750, loss[loss=0.06831, simple_loss=0.09341, pruned_loss=0.01105, audio_tagging_loss=0.01056, over 15002.00 frames. ], tot_loss[loss=0.07376, simple_loss=0.09679, pruned_loss=0.01609, audio_tagging_loss=0.009284, over 3039227.51 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:30:26,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1668200.0, ans=0.0
2023-11-21 21:30:46,080 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250250
2023-11-21 21:30:57,776 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.62 vs. limit=15.0
2023-11-21 21:31:00,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1668400.0, ans=0.0
2023-11-21 21:31:14,283 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9800, loss[loss=0.0997, simple_loss=0.1455, pruned_loss=0.02017, audio_tagging_loss=0.006795, over 15874.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.09765, pruned_loss=0.01633, audio_tagging_loss=0.009197, over 3042314.09 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:31:36,066 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.68 vs. limit=6.0
2023-11-21 21:31:43,567 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.34 vs. limit=15.0
2023-11-21 21:31:45,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1668600.0, ans=0.0
2023-11-21 21:31:51,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250300
2023-11-21 21:31:52,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1668666.6666666667, ans=0.125
2023-11-21 21:31:58,394 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 7.987e+01 8.487e+01 9.363e+01 1.387e+02, threshold=1.697e+02, percent-clipped=0.0
2023-11-21 21:32:11,906 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:32:18,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1668800.0, ans=0.09899494936611666
2023-11-21 21:32:19,232 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9850, loss[loss=0.09889, simple_loss=0.1363, pruned_loss=0.02402, audio_tagging_loss=0.0067, over 15545.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09744, pruned_loss=0.0162, audio_tagging_loss=0.00923, over 3043131.20 frames. ], batch size: 54, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:32:27,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1668800.0, ans=0.125
2023-11-21 21:32:47,713 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:32:55,319 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250350
2023-11-21 21:33:23,728 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9900, loss[loss=0.0809, simple_loss=0.1096, pruned_loss=0.01864, audio_tagging_loss=0.007474, over 14406.00 frames. ], tot_loss[loss=0.07378, simple_loss=0.09698, pruned_loss=0.01614, audio_tagging_loss=0.009147, over 3042199.72 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:33:38,526 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.39 vs. limit=15.0
2023-11-21 21:33:57,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1669266.6666666667, ans=0.125
2023-11-21 21:33:57,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1669266.6666666667, ans=0.0
2023-11-21 21:34:00,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250400
2023-11-21 21:34:03,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1669333.3333333333, ans=0.1
2023-11-21 21:34:08,500 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.048e+01 8.181e+01 8.919e+01 9.368e+01 2.171e+02, threshold=1.784e+02, percent-clipped=1.0
2023-11-21 21:34:19,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1669400.0, ans=0.2
2023-11-21 21:34:26,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1669400.0, ans=0.125
2023-11-21 21:34:28,657 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 9950, loss[loss=0.07345, simple_loss=0.09329, pruned_loss=0.01777, audio_tagging_loss=0.009032, over 15329.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.0965, pruned_loss=0.01609, audio_tagging_loss=0.009204, over 3040189.91 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:34:55,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1669600.0, ans=0.0
2023-11-21 21:35:04,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250450
2023-11-21 21:35:04,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1669600.0, ans=0.1
2023-11-21 21:35:27,575 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.54 vs. limit=15.0
2023-11-21 21:35:33,087 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10000, loss[loss=0.05918, simple_loss=0.07144, pruned_loss=0.01293, audio_tagging_loss=0.01053, over 14903.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09518, pruned_loss=0.01585, audio_tagging_loss=0.009227, over 3045327.56 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:35:52,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1669866.6666666667, ans=0.2
2023-11-21 21:36:09,521 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250500
2023-11-21 21:36:17,279 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.484e+01 8.244e+01 8.958e+01 9.849e+01 1.581e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-21 21:36:35,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1670066.6666666667, ans=0.125
2023-11-21 21:36:37,550 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10050, loss[loss=0.07998, simple_loss=0.1063, pruned_loss=0.01845, audio_tagging_loss=0.008373, over 14539.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09679, pruned_loss=0.01625, audio_tagging_loss=0.00917, over 3041612.69 frames. ], batch size: 52, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:36:37,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1670133.3333333333, ans=0.125
2023-11-21 21:36:40,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1670133.3333333333, ans=0.125
2023-11-21 21:36:48,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1670200.0, ans=0.125
2023-11-21 21:36:48,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1670200.0, ans=0.0
2023-11-21 21:37:05,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_na.min_abs, batch_count=1670266.6666666667, ans=0.02
2023-11-21 21:37:11,323 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.74 vs. limit=10.0
2023-11-21 21:37:13,264 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250550
2023-11-21 21:37:13,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1670266.6666666667, ans=0.0
2023-11-21 21:37:20,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1670333.3333333333, ans=0.1
2023-11-21 21:37:40,423 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10100, loss[loss=0.05057, simple_loss=0.06371, pruned_loss=0.007352, audio_tagging_loss=0.01136, over 17195.00 frames. ], tot_loss[loss=0.07417, simple_loss=0.09723, pruned_loss=0.01636, audio_tagging_loss=0.009196, over 3047752.83 frames. ], batch size: 67, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:37:40,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1670466.6666666667, ans=0.0
2023-11-21 21:38:01,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1670533.3333333333, ans=0.0
2023-11-21 21:38:03,675 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0
2023-11-21 21:38:09,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1670600.0, ans=0.125
2023-11-21 21:38:11,595 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.12 vs. limit=6.0
2023-11-21 21:38:12,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1670600.0, ans=0.2
2023-11-21 21:38:12,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1670600.0, ans=0.0
2023-11-21 21:38:16,932 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250600
2023-11-21 21:38:18,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1670666.6666666667, ans=0.1
2023-11-21 21:38:25,701 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.286e+01 8.086e+01 8.639e+01 9.260e+01 1.282e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-21 21:38:28,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1670666.6666666667, ans=0.0
2023-11-21 21:38:31,779 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:38:45,764 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10150, loss[loss=0.06984, simple_loss=0.09533, pruned_loss=0.01454, audio_tagging_loss=0.007637, over 16126.00 frames. ], tot_loss[loss=0.07371, simple_loss=0.09654, pruned_loss=0.01615, audio_tagging_loss=0.009295, over 3047477.69 frames. ], batch size: 59, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:39:12,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1670933.3333333333, ans=0.125
2023-11-21 21:39:14,486 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:39:20,740 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250650
2023-11-21 21:39:31,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1671000.0, ans=0.125
2023-11-21 21:39:43,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1671066.6666666667, ans=0.0
2023-11-21 21:39:49,976 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10200, loss[loss=0.04198, simple_loss=0.04391, pruned_loss=0.008775, audio_tagging_loss=0.01125, over 16157.00 frames. ], tot_loss[loss=0.07282, simple_loss=0.09507, pruned_loss=0.01586, audio_tagging_loss=0.009427, over 3049723.05 frames. ], batch size: 63, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:40:03,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1671200.0, ans=0.125
2023-11-21 21:40:12,606 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:40:16,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1671266.6666666667, ans=0.1
2023-11-21 21:40:21,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1671266.6666666667, ans=0.0
2023-11-21 21:40:25,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1671266.6666666667, ans=0.125
2023-11-21 21:40:26,534 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250700
2023-11-21 21:40:35,584 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.985e+01 8.067e+01 8.565e+01 9.348e+01 1.300e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-21 21:40:44,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1671400.0, ans=0.125
2023-11-21 21:40:44,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1671400.0, ans=0.0
2023-11-21 21:40:50,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1671400.0, ans=0.2
2023-11-21 21:40:54,069 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10250, loss[loss=0.07205, simple_loss=0.09628, pruned_loss=0.01488, audio_tagging_loss=0.009031, over 15142.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.0955, pruned_loss=0.01594, audio_tagging_loss=0.009477, over 3043011.43 frames. ], batch size: 60, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:40:56,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1671466.6666666667, ans=0.125
2023-11-21 21:41:30,155 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250750
2023-11-21 21:41:58,109 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10300, loss[loss=0.08273, simple_loss=0.1099, pruned_loss=0.0172, audio_tagging_loss=0.0106, over 15562.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.09593, pruned_loss=0.01609, audio_tagging_loss=0.009483, over 3045123.54 frames. ], batch size: 60, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:42:30,174 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.09 vs. limit=15.0
2023-11-21 21:42:33,142 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250800
2023-11-21 21:42:36,671 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.63 vs. limit=22.5
2023-11-21 21:42:42,610 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.369e+01 8.124e+01 8.842e+01 9.369e+01 1.260e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 21:42:49,674 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.38 vs. limit=22.5
2023-11-21 21:43:03,287 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10350, loss[loss=0.09549, simple_loss=0.1251, pruned_loss=0.02359, audio_tagging_loss=0.009342, over 13765.00 frames. ], tot_loss[loss=0.07415, simple_loss=0.09677, pruned_loss=0.01628, audio_tagging_loss=0.009491, over 3049814.69 frames. ], batch size: 53, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:43:11,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1672133.3333333333, ans=0.0
2023-11-21 21:43:39,853 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250850
2023-11-21 21:43:55,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1672400.0, ans=0.125
2023-11-21 21:43:56,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1672400.0, ans=0.0
2023-11-21 21:44:00,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1672400.0, ans=0.1
2023-11-21 21:44:07,053 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10400, loss[loss=0.06957, simple_loss=0.08687, pruned_loss=0.01245, audio_tagging_loss=0.01369, over 15519.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09612, pruned_loss=0.01609, audio_tagging_loss=0.009581, over 3046365.71 frames. ], batch size: 59, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:44:09,268 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.64 vs. limit=5.0
2023-11-21 21:44:14,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1672466.6666666667, ans=0.0
2023-11-21 21:44:28,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1672533.3333333333, ans=0.1
2023-11-21 21:44:43,951 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250900
2023-11-21 21:44:46,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1672666.6666666667, ans=0.0
2023-11-21 21:44:52,357 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.219e+01 8.868e+01 9.725e+01 1.469e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 21:44:53,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1672666.6666666667, ans=0.125
2023-11-21 21:44:54,375 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.80 vs. limit=6.0
2023-11-21 21:44:57,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1672733.3333333333, ans=0.125
2023-11-21 21:45:09,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1672733.3333333333, ans=0.025
2023-11-21 21:45:12,309 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10450, loss[loss=0.07182, simple_loss=0.09545, pruned_loss=0.01679, audio_tagging_loss=0.007308, over 15757.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.0961, pruned_loss=0.01621, audio_tagging_loss=0.00954, over 3044726.06 frames. ], batch size: 60, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:45:16,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1672800.0, ans=0.0
2023-11-21 21:45:27,240 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.75 vs. limit=12.0
2023-11-21 21:45:31,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1672866.6666666667, ans=0.125
2023-11-21 21:45:33,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1672866.6666666667, ans=0.0
2023-11-21 21:45:34,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1672866.6666666667, ans=0.125
2023-11-21 21:45:48,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 250950
2023-11-21 21:46:11,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1673066.6666666667, ans=0.125
2023-11-21 21:46:17,698 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10500, loss[loss=0.05982, simple_loss=0.0766, pruned_loss=0.01421, audio_tagging_loss=0.007301, over 14753.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09542, pruned_loss=0.01608, audio_tagging_loss=0.009426, over 3044199.43 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:46:28,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1673133.3333333333, ans=0.1
2023-11-21 21:46:54,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251000
2023-11-21 21:46:57,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1673333.3333333333, ans=0.125
2023-11-21 21:46:58,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1673333.3333333333, ans=0.125
2023-11-21 21:46:59,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1673333.3333333333, ans=0.0
2023-11-21 21:47:04,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1673333.3333333333, ans=0.125
2023-11-21 21:47:05,261 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.030e+01 8.294e+01 8.913e+01 9.437e+01 1.275e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-21 21:47:23,426 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10550, loss[loss=0.06798, simple_loss=0.08973, pruned_loss=0.01744, audio_tagging_loss=0.005675, over 15044.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09477, pruned_loss=0.01594, audio_tagging_loss=0.009299, over 3042686.97 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:47:34,008 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.46 vs. limit=15.0
2023-11-21 21:47:39,793 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.32 vs. limit=15.0
2023-11-21 21:47:49,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1673600.0, ans=0.2
2023-11-21 21:47:54,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1673600.0, ans=0.0
2023-11-21 21:47:56,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1673600.0, ans=0.125
2023-11-21 21:48:00,788 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251050
2023-11-21 21:48:08,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1673666.6666666667, ans=0.125
2023-11-21 21:48:09,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1673666.6666666667, ans=0.04949747468305833
2023-11-21 21:48:17,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1673733.3333333333, ans=0.0
2023-11-21 21:48:23,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1673733.3333333333, ans=0.125
2023-11-21 21:48:25,168 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.28 vs. limit=15.0
2023-11-21 21:48:28,731 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10600, loss[loss=0.08719, simple_loss=0.1102, pruned_loss=0.01892, audio_tagging_loss=0.01315, over 15449.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09545, pruned_loss=0.01598, audio_tagging_loss=0.009198, over 3046758.36 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:48:30,703 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.37 vs. limit=15.0
2023-11-21 21:48:55,335 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:48:56,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1673933.3333333333, ans=0.125
2023-11-21 21:49:05,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251100
2023-11-21 21:49:15,230 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.229e+01 8.713e+01 9.313e+01 1.246e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-21 21:49:24,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1674066.6666666667, ans=0.0
2023-11-21 21:49:27,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1674066.6666666667, ans=0.0
2023-11-21 21:49:33,270 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10650, loss[loss=0.0793, simple_loss=0.1064, pruned_loss=0.01832, audio_tagging_loss=0.007786, over 15036.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.09584, pruned_loss=0.01596, audio_tagging_loss=0.009145, over 3048666.22 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:50:09,598 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251150
2023-11-21 21:50:13,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1674333.3333333333, ans=22.5
2023-11-21 21:50:14,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1674333.3333333333, ans=0.1
2023-11-21 21:50:25,509 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:50:38,660 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10700, loss[loss=0.06269, simple_loss=0.0749, pruned_loss=0.0135, audio_tagging_loss=0.01174, over 14668.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.096, pruned_loss=0.01605, audio_tagging_loss=0.009144, over 3046373.22 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:50:49,298 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.78 vs. limit=15.0
2023-11-21 21:50:54,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1674533.3333333333, ans=0.2
2023-11-21 21:51:04,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1674600.0, ans=0.1
2023-11-21 21:51:11,012 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.83 vs. limit=15.0
2023-11-21 21:51:14,781 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251200
2023-11-21 21:51:25,544 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.625e+01 8.099e+01 8.796e+01 9.421e+01 3.240e+02, threshold=1.759e+02, percent-clipped=1.0
2023-11-21 21:51:39,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.72 vs. limit=10.0
2023-11-21 21:51:43,395 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10750, loss[loss=0.07661, simple_loss=0.105, pruned_loss=0.01668, audio_tagging_loss=0.007438, over 15483.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.09565, pruned_loss=0.01596, audio_tagging_loss=0.009242, over 3045705.21 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:52:00,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1674866.6666666667, ans=0.125
2023-11-21 21:52:19,738 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251250
2023-11-21 21:52:27,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1675000.0, ans=0.125
2023-11-21 21:52:33,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1675066.6666666667, ans=0.5
2023-11-21 21:52:44,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1675066.6666666667, ans=0.2
2023-11-21 21:52:47,591 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10800, loss[loss=0.09431, simple_loss=0.1253, pruned_loss=0.0202, audio_tagging_loss=0.01148, over 16482.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09539, pruned_loss=0.01593, audio_tagging_loss=0.009232, over 3052185.49 frames. ], batch size: 58, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:53:19,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1675266.6666666667, ans=0.125
2023-11-21 21:53:23,716 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251300
2023-11-21 21:53:29,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1675333.3333333333, ans=0.2
2023-11-21 21:53:34,028 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.560e+01 8.266e+01 8.735e+01 9.337e+01 1.123e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-21 21:53:52,968 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10850, loss[loss=0.06918, simple_loss=0.08715, pruned_loss=0.01557, audio_tagging_loss=0.01004, over 14373.00 frames. ], tot_loss[loss=0.07332, simple_loss=0.09595, pruned_loss=0.01609, audio_tagging_loss=0.009261, over 3045768.20 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:53:53,647 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.89 vs. limit=22.5
2023-11-21 21:53:54,904 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.06 vs. limit=15.0
2023-11-21 21:54:00,664 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:54:00,892 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.07 vs. limit=15.0
2023-11-21 21:54:16,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1675533.3333333333, ans=0.0
2023-11-21 21:54:19,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.00 vs. limit=10.0
2023-11-21 21:54:28,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251350
2023-11-21 21:54:31,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1675666.6666666667, ans=0.125
2023-11-21 21:54:40,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1675666.6666666667, ans=0.125
2023-11-21 21:54:52,756 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:54:56,562 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10900, loss[loss=0.08146, simple_loss=0.1021, pruned_loss=0.01892, audio_tagging_loss=0.0115, over 14655.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09539, pruned_loss=0.01591, audio_tagging_loss=0.009308, over 3045607.35 frames. ], batch size: 55, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:55:17,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1675866.6666666667, ans=0.0
2023-11-21 21:55:26,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1675933.3333333333, ans=0.125
2023-11-21 21:55:33,602 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251400
2023-11-21 21:55:38,269 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.97 vs. limit=15.0
2023-11-21 21:55:43,669 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.153e+01 8.721e+01 9.251e+01 1.356e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-21 21:55:57,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1676066.6666666667, ans=0.0
2023-11-21 21:55:57,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1676066.6666666667, ans=0.125
2023-11-21 21:55:59,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_na.min_abs, batch_count=1676066.6666666667, ans=0.02
2023-11-21 21:55:59,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1676066.6666666667, ans=0.0
2023-11-21 21:56:02,096 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 10950, loss[loss=0.0754, simple_loss=0.08946, pruned_loss=0.02199, audio_tagging_loss=0.00867, over 15729.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09517, pruned_loss=0.01578, audio_tagging_loss=0.009408, over 3051950.54 frames. ], batch size: 61, lr: 3.23e-03, grad_scale: 32.0
2023-11-21 21:56:03,998 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.13 vs. limit=6.0
2023-11-21 21:56:16,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1676200.0, ans=0.1
2023-11-21 21:56:27,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1676266.6666666667, ans=0.125
2023-11-21 21:56:32,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1676266.6666666667, ans=0.0
2023-11-21 21:56:37,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1676266.6666666667, ans=0.2
2023-11-21 21:56:38,002 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251450
2023-11-21 21:56:54,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1676400.0, ans=0.09899494936611666
2023-11-21 21:56:56,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1676400.0, ans=0.04949747468305833
2023-11-21 21:56:57,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1676400.0, ans=0.2
2023-11-21 21:56:58,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1676400.0, ans=0.125
2023-11-21 21:57:06,335 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11000, loss[loss=0.05309, simple_loss=0.06519, pruned_loss=0.01167, audio_tagging_loss=0.008821, over 14793.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09506, pruned_loss=0.01572, audio_tagging_loss=0.009411, over 3044392.94 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:57:16,920 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 21:57:22,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1676533.3333333333, ans=0.2
2023-11-21 21:57:38,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1676600.0, ans=0.125
2023-11-21 21:57:41,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251500
2023-11-21 21:57:46,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1676666.6666666667, ans=0.125
2023-11-21 21:57:53,222 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.659e+01 8.207e+01 8.781e+01 9.534e+01 1.219e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-21 21:58:07,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1676733.3333333333, ans=0.125
2023-11-21 21:58:09,755 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11050, loss[loss=0.08563, simple_loss=0.1253, pruned_loss=0.01504, audio_tagging_loss=0.007949, over 15535.00 frames. ], tot_loss[loss=0.07308, simple_loss=0.09565, pruned_loss=0.01578, audio_tagging_loss=0.009475, over 3055962.43 frames. ], batch size: 56, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:58:25,399 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:58:44,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=1676933.3333333333, ans=0.5
2023-11-21 21:58:45,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251550
2023-11-21 21:58:55,029 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 21:59:13,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1677133.3333333333, ans=0.125
2023-11-21 21:59:14,252 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11100, loss[loss=0.0644, simple_loss=0.0849, pruned_loss=0.01175, audio_tagging_loss=0.01021, over 15425.00 frames. ], tot_loss[loss=0.07333, simple_loss=0.09575, pruned_loss=0.01582, audio_tagging_loss=0.009646, over 3054659.44 frames. ], batch size: 57, lr: 3.23e-03, grad_scale: 16.0
2023-11-21 21:59:18,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1677133.3333333333, ans=0.2
2023-11-21 21:59:27,720 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.16 vs. limit=15.0
2023-11-21 21:59:35,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1677200.0, ans=0.0
2023-11-21 21:59:38,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1677266.6666666667, ans=0.125
2023-11-21 21:59:50,089 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251600
2023-11-21 21:59:58,051 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:00:02,563 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.524e+01 9.104e+01 1.000e+02 2.938e+02, threshold=1.821e+02, percent-clipped=1.0
2023-11-21 22:00:04,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1677333.3333333333, ans=0.0
2023-11-21 22:00:15,700 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.81 vs. limit=5.0
2023-11-21 22:00:18,955 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11150, loss[loss=0.08188, simple_loss=0.1031, pruned_loss=0.01788, audio_tagging_loss=0.01243, over 15761.00 frames. ], tot_loss[loss=0.07386, simple_loss=0.09636, pruned_loss=0.01593, audio_tagging_loss=0.009748, over 3052605.66 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:00:37,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.71 vs. limit=15.0
2023-11-21 22:00:42,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1677533.3333333333, ans=0.125
2023-11-21 22:00:55,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251650
2023-11-21 22:01:05,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1677666.6666666667, ans=0.125
2023-11-21 22:01:10,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1677733.3333333333, ans=0.125
2023-11-21 22:01:16,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1677733.3333333333, ans=0.125
2023-11-21 22:01:22,751 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11200, loss[loss=0.09066, simple_loss=0.1165, pruned_loss=0.02164, audio_tagging_loss=0.01076, over 15825.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09559, pruned_loss=0.01571, audio_tagging_loss=0.009852, over 3050270.58 frames. ], batch size: 57, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:01:28,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1677800.0, ans=0.2
2023-11-21 22:01:43,968 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.72 vs. limit=15.0
2023-11-21 22:01:49,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1677933.3333333333, ans=0.2
2023-11-21 22:01:55,023 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.64 vs. limit=15.0
2023-11-21 22:01:59,530 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251700
2023-11-21 22:02:10,494 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.694e+01 7.989e+01 8.616e+01 9.280e+01 1.646e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-21 22:02:28,506 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11250, loss[loss=0.06634, simple_loss=0.08669, pruned_loss=0.01453, audio_tagging_loss=0.008459, over 14748.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09471, pruned_loss=0.01566, audio_tagging_loss=0.009837, over 3041509.13 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:03:02,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1678266.6666666667, ans=0.2
2023-11-21 22:03:03,371 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251750
2023-11-21 22:03:17,636 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.94 vs. limit=22.5
2023-11-21 22:03:24,397 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.563e-03
2023-11-21 22:03:32,030 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11300, loss[loss=0.06445, simple_loss=0.08445, pruned_loss=0.01438, audio_tagging_loss=0.007848, over 15958.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09574, pruned_loss=0.01567, audio_tagging_loss=0.009626, over 3048432.71 frames. ], batch size: 63, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:03:38,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1678466.6666666667, ans=0.125
2023-11-21 22:03:45,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1678533.3333333333, ans=0.0
2023-11-21 22:03:47,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1678533.3333333333, ans=0.1
2023-11-21 22:03:53,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1678533.3333333333, ans=0.5
2023-11-21 22:04:01,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1678600.0, ans=0.125
2023-11-21 22:04:08,518 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251800
2023-11-21 22:04:11,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1678666.6666666667, ans=0.1
2023-11-21 22:04:15,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1678666.6666666667, ans=0.1
2023-11-21 22:04:19,981 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.032e+01 8.626e+01 9.245e+01 1.282e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-21 22:04:20,784 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.07 vs. limit=15.0
2023-11-21 22:04:36,729 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11350, loss[loss=0.06117, simple_loss=0.08143, pruned_loss=0.01272, audio_tagging_loss=0.007741, over 15639.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09541, pruned_loss=0.01569, audio_tagging_loss=0.009518, over 3044510.25 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:04:48,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.76 vs. limit=10.0
2023-11-21 22:05:12,703 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251850
2023-11-21 22:05:17,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1679000.0, ans=0.2
2023-11-21 22:05:29,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1679066.6666666667, ans=0.125
2023-11-21 22:05:40,483 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11400, loss[loss=0.07963, simple_loss=0.1124, pruned_loss=0.01542, audio_tagging_loss=0.00802, over 15146.00 frames. ], tot_loss[loss=0.07358, simple_loss=0.09603, pruned_loss=0.01609, audio_tagging_loss=0.009477, over 3043092.70 frames. ], batch size: 56, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:05:43,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1679133.3333333333, ans=0.0
2023-11-21 22:05:50,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1679133.3333333333, ans=0.125
2023-11-21 22:06:09,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1679266.6666666667, ans=0.125
2023-11-21 22:06:15,756 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251900
2023-11-21 22:06:17,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1679333.3333333333, ans=0.125
2023-11-21 22:06:19,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1679333.3333333333, ans=0.0
2023-11-21 22:06:27,613 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.296e+01 8.125e+01 8.630e+01 9.501e+01 1.337e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-21 22:06:44,325 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11450, loss[loss=0.0623, simple_loss=0.08326, pruned_loss=0.01294, audio_tagging_loss=0.007721, over 15667.00 frames. ], tot_loss[loss=0.07333, simple_loss=0.09562, pruned_loss=0.01602, audio_tagging_loss=0.009489, over 3041042.69 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:06:53,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1679466.6666666667, ans=0.2
2023-11-21 22:06:59,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1679533.3333333333, ans=0.1
2023-11-21 22:07:09,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1679600.0, ans=0.0
2023-11-21 22:07:13,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1679600.0, ans=0.125
2023-11-21 22:07:20,654 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.98 vs. limit=12.0
2023-11-21 22:07:21,342 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 251950
2023-11-21 22:07:25,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1679666.6666666667, ans=0.125
2023-11-21 22:07:28,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1679666.6666666667, ans=0.1
2023-11-21 22:07:36,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1679733.3333333333, ans=0.125
2023-11-21 22:07:48,712 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11500, loss[loss=0.1026, simple_loss=0.1405, pruned_loss=0.02427, audio_tagging_loss=0.00805, over 16149.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09542, pruned_loss=0.01607, audio_tagging_loss=0.009453, over 3039894.96 frames. ], batch size: 59, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:07:50,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1679800.0, ans=0.0
2023-11-21 22:08:25,035 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252000
2023-11-21 22:08:26,511 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-252000.pt
2023-11-21 22:08:32,019 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:08:33,705 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.94 vs. limit=15.0
2023-11-21 22:08:39,030 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.783e+01 7.996e+01 8.554e+01 9.237e+01 1.174e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 22:08:43,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.71 vs. limit=12.0
2023-11-21 22:08:56,378 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11550, loss[loss=0.07246, simple_loss=0.0911, pruned_loss=0.01524, audio_tagging_loss=0.01167, over 15877.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09549, pruned_loss=0.01597, audio_tagging_loss=0.009443, over 3041199.87 frames. ], batch size: 59, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:09:05,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1680133.3333333333, ans=0.125
2023-11-21 22:09:12,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1680200.0, ans=0.1
2023-11-21 22:09:17,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1680200.0, ans=0.125
2023-11-21 22:09:19,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1680200.0, ans=0.125
2023-11-21 22:09:27,853 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.87 vs. limit=22.5
2023-11-21 22:09:30,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.87 vs. limit=15.0
2023-11-21 22:09:31,198 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.87 vs. limit=15.0
2023-11-21 22:09:31,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252050
2023-11-21 22:09:34,303 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 22:09:34,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1680333.3333333333, ans=0.2
2023-11-21 22:09:52,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.27 vs. limit=22.5
2023-11-21 22:10:00,348 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11600, loss[loss=0.09783, simple_loss=0.1387, pruned_loss=0.01887, audio_tagging_loss=0.009629, over 16152.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09518, pruned_loss=0.01591, audio_tagging_loss=0.009422, over 3043928.09 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:10:03,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1680466.6666666667, ans=0.1
2023-11-21 22:10:29,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1680600.0, ans=0.125
2023-11-21 22:10:36,285 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252100
2023-11-21 22:10:46,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1680666.6666666667, ans=0.125
2023-11-21 22:10:47,819 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.780e+01 8.258e+01 8.947e+01 9.707e+01 1.482e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-21 22:10:50,718 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.050e-02
2023-11-21 22:10:52,136 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.25 vs. limit=22.5
2023-11-21 22:10:57,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1680733.3333333333, ans=0.125
2023-11-21 22:11:04,549 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11650, loss[loss=0.08002, simple_loss=0.1016, pruned_loss=0.02095, audio_tagging_loss=0.008273, over 14586.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09453, pruned_loss=0.01585, audio_tagging_loss=0.009462, over 3039370.29 frames. ], batch size: 53, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:11:12,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1680800.0, ans=0.035
2023-11-21 22:11:24,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1680866.6666666667, ans=0.1
2023-11-21 22:11:34,255 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.07 vs. limit=12.0
2023-11-21 22:11:40,812 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252150
2023-11-21 22:11:46,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.14 vs. limit=12.0
2023-11-21 22:12:08,135 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11700, loss[loss=0.05951, simple_loss=0.07325, pruned_loss=0.01285, audio_tagging_loss=0.01004, over 14685.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09459, pruned_loss=0.01574, audio_tagging_loss=0.009398, over 3048010.55 frames. ], batch size: 57, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:12:13,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1681133.3333333333, ans=0.125
2023-11-21 22:12:29,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1681200.0, ans=0.125
2023-11-21 22:12:38,297 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.67 vs. limit=22.5
2023-11-21 22:12:42,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1681266.6666666667, ans=0.05
2023-11-21 22:12:45,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252200
2023-11-21 22:12:54,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1681333.3333333333, ans=0.125
2023-11-21 22:12:56,198 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.227e+01 8.797e+01 9.747e+01 1.315e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-21 22:13:11,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1681400.0, ans=0.05
2023-11-21 22:13:14,118 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11750, loss[loss=0.0627, simple_loss=0.08277, pruned_loss=0.01171, audio_tagging_loss=0.009609, over 14854.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09473, pruned_loss=0.01569, audio_tagging_loss=0.00949, over 3053109.84 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:13:38,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1681600.0, ans=0.125
2023-11-21 22:13:50,113 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252250
2023-11-21 22:14:02,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=1681666.6666666667, ans=15.0
2023-11-21 22:14:10,249 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:14:19,024 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11800, loss[loss=0.07632, simple_loss=0.1008, pruned_loss=0.01799, audio_tagging_loss=0.007946, over 15939.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.095, pruned_loss=0.01568, audio_tagging_loss=0.00951, over 3052489.83 frames. ], batch size: 61, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:14:19,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1681800.0, ans=0.1
2023-11-21 22:14:26,075 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.52 vs. limit=10.0
2023-11-21 22:14:28,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1681800.0, ans=0.125
2023-11-21 22:14:35,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1681866.6666666667, ans=0.035
2023-11-21 22:14:53,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1681933.3333333333, ans=0.1
2023-11-21 22:14:54,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=1681933.3333333333, ans=15.0
2023-11-21 22:14:54,893 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252300
2023-11-21 22:14:59,026 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.70 vs. limit=22.5
2023-11-21 22:15:07,681 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 7.961e+01 8.555e+01 9.322e+01 1.275e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 22:15:09,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1682066.6666666667, ans=0.015
2023-11-21 22:15:19,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1682066.6666666667, ans=0.0
2023-11-21 22:15:22,866 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11850, loss[loss=0.08304, simple_loss=0.1164, pruned_loss=0.01623, audio_tagging_loss=0.008639, over 15798.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09575, pruned_loss=0.01582, audio_tagging_loss=0.009546, over 3053641.46 frames. ], batch size: 58, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:15:34,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1682200.0, ans=0.125
2023-11-21 22:15:49,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1682266.6666666667, ans=0.125
2023-11-21 22:15:50,387 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.17 vs. limit=6.0
2023-11-21 22:16:00,040 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252350
2023-11-21 22:16:01,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1682333.3333333333, ans=0.125
2023-11-21 22:16:27,871 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11900, loss[loss=0.08503, simple_loss=0.1145, pruned_loss=0.01891, audio_tagging_loss=0.008884, over 15139.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09597, pruned_loss=0.01594, audio_tagging_loss=0.009484, over 3056933.56 frames. ], batch size: 57, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:16:36,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1682466.6666666667, ans=0.0
2023-11-21 22:16:45,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.00 vs. limit=10.0
2023-11-21 22:16:56,062 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.41 vs. limit=10.0
2023-11-21 22:17:01,146 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.61 vs. limit=15.0
2023-11-21 22:17:04,126 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252400
2023-11-21 22:17:17,493 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.240e+01 7.885e+01 8.457e+01 9.193e+01 1.226e+02, threshold=1.691e+02, percent-clipped=0.0
2023-11-21 22:17:33,598 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 11950, loss[loss=0.06527, simple_loss=0.08261, pruned_loss=0.0133, audio_tagging_loss=0.01067, over 15232.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09586, pruned_loss=0.01603, audio_tagging_loss=0.009634, over 3051756.47 frames. ], batch size: 56, lr: 3.22e-03, grad_scale: 16.0
2023-11-21 22:17:57,688 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.90 vs. limit=15.0
2023-11-21 22:18:09,278 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252450
2023-11-21 22:18:09,952 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0
2023-11-21 22:18:10,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1683000.0, ans=0.125
2023-11-21 22:18:14,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.25 vs. limit=15.0
2023-11-21 22:18:15,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1683000.0, ans=0.125
2023-11-21 22:18:26,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1683066.6666666667, ans=0.1
2023-11-21 22:18:29,057 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.74 vs. limit=22.5
2023-11-21 22:18:36,451 INFO [train_asr.py:1221] (0/4) Epoch 21, batch 12000, loss[loss=0.05464, simple_loss=0.06808, pruned_loss=0.01023, audio_tagging_loss=0.01038, over 15870.00 frames. ], tot_loss[loss=0.07323, simple_loss=0.09524, pruned_loss=0.01583, audio_tagging_loss=0.009778, over 3055872.96 frames. ], batch size: 63, lr: 3.22e-03, grad_scale: 32.0
2023-11-21 22:18:36,455 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 22:19:19,212 INFO [train_asr.py:1253] (0/4) Epoch 21, validation: loss=0.05938, simple_loss=0.05195, pruned_loss=0.005201, audio_tagging_loss=0.02821, over 4681554.00 frames. 
2023-11-21 22:19:19,213 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 22:19:34,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1683200.0, ans=0.0
2023-11-21 22:19:40,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1683200.0, ans=0.125
2023-11-21 22:19:49,480 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-21.pt
2023-11-21 22:20:22,654 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 0, loss[loss=0.08497, simple_loss=0.1029, pruned_loss=0.01275, audio_tagging_loss=0.02079, over 14920.00 frames. ], tot_loss[loss=0.08497, simple_loss=0.1029, pruned_loss=0.01275, audio_tagging_loss=0.02079, over 14920.00 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:20:22,657 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 22:20:59,036 INFO [train_asr.py:1253] (0/4) Epoch 22, validation: loss=0.05904, simple_loss=0.0519, pruned_loss=0.0051, audio_tagging_loss=0.02799, over 4681554.00 frames. 
2023-11-21 22:20:59,037 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 22:21:03,957 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252500
2023-11-21 22:21:10,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=1683360.0, ans=0.1
2023-11-21 22:21:16,300 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.182e+01 9.159e+01 9.846e+01 1.292e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-21 22:21:26,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1683426.6666666667, ans=0.125
2023-11-21 22:21:59,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1683560.0, ans=0.125
2023-11-21 22:22:02,775 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 50, loss[loss=0.08892, simple_loss=0.1134, pruned_loss=0.01554, audio_tagging_loss=0.01668, over 15208.00 frames. ], tot_loss[loss=0.08575, simple_loss=0.1011, pruned_loss=0.01741, audio_tagging_loss=0.01778, over 694445.01 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:22:07,699 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252550
2023-11-21 22:22:24,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.52 vs. limit=15.0
2023-11-21 22:22:31,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1683760.0, ans=0.0
2023-11-21 22:23:06,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1683960.0, ans=0.125
2023-11-21 22:23:06,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1683960.0, ans=0.07
2023-11-21 22:23:07,467 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 100, loss[loss=0.07206, simple_loss=0.09346, pruned_loss=0.01111, audio_tagging_loss=0.01422, over 14659.00 frames. ], tot_loss[loss=0.08135, simple_loss=0.09606, pruned_loss=0.0161, audio_tagging_loss=0.01722, over 1208420.75 frames. ], batch size: 53, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:23:09,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1683960.0, ans=0.2
2023-11-21 22:23:09,341 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.61 vs. limit=12.0
2023-11-21 22:23:12,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252600
2023-11-21 22:23:17,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff2.min_abs, batch_count=1683960.0, ans=0.1
2023-11-21 22:23:24,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1684026.6666666667, ans=0.05
2023-11-21 22:23:25,408 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.481e+01 8.773e+01 9.405e+01 1.016e+02 1.413e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-21 22:23:42,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1684093.3333333333, ans=0.1
2023-11-21 22:23:44,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1684160.0, ans=0.5
2023-11-21 22:23:50,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1684160.0, ans=0.1
2023-11-21 22:24:11,242 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 150, loss[loss=0.07115, simple_loss=0.08423, pruned_loss=0.01707, audio_tagging_loss=0.01196, over 14548.00 frames. ], tot_loss[loss=0.07914, simple_loss=0.09541, pruned_loss=0.016, audio_tagging_loss=0.01544, over 1617436.02 frames. ], batch size: 54, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:24:16,694 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252650
2023-11-21 22:24:39,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.14 vs. limit=15.0
2023-11-21 22:24:44,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1684426.6666666667, ans=0.125
2023-11-21 22:24:44,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.40 vs. limit=15.0
2023-11-21 22:25:15,870 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 200, loss[loss=0.08383, simple_loss=0.1121, pruned_loss=0.01459, audio_tagging_loss=0.01321, over 14928.00 frames. ], tot_loss[loss=0.07759, simple_loss=0.09578, pruned_loss=0.01599, audio_tagging_loss=0.01371, over 1938058.50 frames. ], batch size: 53, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:25:20,926 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252700
2023-11-21 22:25:26,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1684626.6666666667, ans=0.125
2023-11-21 22:25:26,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1684626.6666666667, ans=0.125
2023-11-21 22:25:28,501 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:25:31,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1684693.3333333333, ans=0.125
2023-11-21 22:25:33,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1684693.3333333333, ans=0.125
2023-11-21 22:25:34,373 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.318e+01 8.693e+01 9.716e+01 1.201e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 22:25:51,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1684760.0, ans=0.5
2023-11-21 22:26:03,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=1684826.6666666667, ans=15.0
2023-11-21 22:26:17,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1684893.3333333333, ans=0.125
2023-11-21 22:26:21,132 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 250, loss[loss=0.07466, simple_loss=0.1019, pruned_loss=0.01589, audio_tagging_loss=0.007798, over 14335.00 frames. ], tot_loss[loss=0.07657, simple_loss=0.09647, pruned_loss=0.01594, audio_tagging_loss=0.01239, over 2178024.23 frames. ], batch size: 56, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:26:26,028 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252750
2023-11-21 22:26:59,619 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.19 vs. limit=15.0
2023-11-21 22:27:22,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1685226.6666666667, ans=0.125
2023-11-21 22:27:24,801 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 300, loss[loss=0.07655, simple_loss=0.1021, pruned_loss=0.01333, audio_tagging_loss=0.01218, over 15506.00 frames. ], tot_loss[loss=0.07565, simple_loss=0.0966, pruned_loss=0.01589, audio_tagging_loss=0.01146, over 2380642.51 frames. ], batch size: 59, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:27:25,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1685293.3333333333, ans=0.125
2023-11-21 22:27:30,622 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252800
2023-11-21 22:27:43,811 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.749e+01 8.171e+01 8.868e+01 9.880e+01 1.210e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 22:27:55,837 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.06 vs. limit=12.0
2023-11-21 22:27:59,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1685426.6666666667, ans=0.125
2023-11-21 22:28:06,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1685493.3333333333, ans=0.0
2023-11-21 22:28:06,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1685493.3333333333, ans=0.125
2023-11-21 22:28:09,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1685493.3333333333, ans=0.07
2023-11-21 22:28:09,097 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.531e-03
2023-11-21 22:28:22,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1685560.0, ans=0.0
2023-11-21 22:28:27,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1685560.0, ans=0.125
2023-11-21 22:28:30,576 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 350, loss[loss=0.04804, simple_loss=0.05858, pruned_loss=0.009593, audio_tagging_loss=0.009153, over 14544.00 frames. ], tot_loss[loss=0.07533, simple_loss=0.09684, pruned_loss=0.01604, audio_tagging_loss=0.01088, over 2529407.14 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:28:33,911 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:28:36,328 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252850
2023-11-21 22:28:44,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1685693.3333333333, ans=0.1
2023-11-21 22:28:51,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1685693.3333333333, ans=0.0
2023-11-21 22:29:15,496 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0
2023-11-21 22:29:28,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1685893.3333333333, ans=0.125
2023-11-21 22:29:36,489 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 400, loss[loss=0.07256, simple_loss=0.08928, pruned_loss=0.01918, audio_tagging_loss=0.008742, over 14183.00 frames. ], tot_loss[loss=0.07476, simple_loss=0.09621, pruned_loss=0.01606, audio_tagging_loss=0.01059, over 2645284.37 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:29:40,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1685960.0, ans=0.1
2023-11-21 22:29:42,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252900
2023-11-21 22:29:51,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1686026.6666666667, ans=0.2
2023-11-21 22:29:55,040 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.919e+01 8.135e+01 8.839e+01 9.372e+01 1.171e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 22:29:59,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1686026.6666666667, ans=0.125
2023-11-21 22:30:01,366 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.82 vs. limit=15.0
2023-11-21 22:30:11,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1686093.3333333333, ans=0.125
2023-11-21 22:30:13,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1686093.3333333333, ans=0.125
2023-11-21 22:30:14,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1686160.0, ans=0.0
2023-11-21 22:30:41,865 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.59 vs. limit=6.0
2023-11-21 22:30:42,525 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 450, loss[loss=0.07283, simple_loss=0.1006, pruned_loss=0.01522, audio_tagging_loss=0.007322, over 16351.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09571, pruned_loss=0.01583, audio_tagging_loss=0.0104, over 2735871.49 frames. ], batch size: 59, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:30:44,539 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.75 vs. limit=22.5
2023-11-21 22:30:48,338 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 252950
2023-11-21 22:30:57,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1686360.0, ans=0.125
2023-11-21 22:31:20,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1686426.6666666667, ans=0.0
2023-11-21 22:31:25,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1686493.3333333333, ans=0.1
2023-11-21 22:31:31,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1686493.3333333333, ans=0.125
2023-11-21 22:31:46,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.36 vs. limit=15.0
2023-11-21 22:31:48,760 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 500, loss[loss=0.06026, simple_loss=0.08364, pruned_loss=0.01182, audio_tagging_loss=0.00662, over 15860.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09561, pruned_loss=0.01592, audio_tagging_loss=0.01018, over 2803628.54 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:31:53,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253000
2023-11-21 22:31:58,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1686626.6666666667, ans=0.0
2023-11-21 22:32:01,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1686693.3333333333, ans=0.07
2023-11-21 22:32:01,606 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.80 vs. limit=10.0
2023-11-21 22:32:04,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1686693.3333333333, ans=0.125
2023-11-21 22:32:06,977 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.743e+01 8.271e+01 8.753e+01 9.475e+01 1.190e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 22:32:13,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1686693.3333333333, ans=0.1
2023-11-21 22:32:38,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1686826.6666666667, ans=0.0
2023-11-21 22:32:45,459 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.42 vs. limit=15.0
2023-11-21 22:32:54,278 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 550, loss[loss=0.06435, simple_loss=0.08572, pruned_loss=0.01181, audio_tagging_loss=0.009672, over 14478.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09498, pruned_loss=0.01583, audio_tagging_loss=0.01013, over 2858509.52 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:33:00,569 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253050
2023-11-21 22:33:16,308 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.27 vs. limit=15.0
2023-11-21 22:34:00,164 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 600, loss[loss=0.07918, simple_loss=0.1103, pruned_loss=0.01511, audio_tagging_loss=0.008932, over 14877.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09517, pruned_loss=0.01596, audio_tagging_loss=0.01001, over 2897197.24 frames. ], batch size: 53, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:34:05,303 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253100
2023-11-21 22:34:18,144 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.004e+01 8.758e+01 9.434e+01 1.366e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-21 22:34:24,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1687426.6666666667, ans=0.0
2023-11-21 22:34:29,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1687426.6666666667, ans=0.125
2023-11-21 22:34:35,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1687426.6666666667, ans=0.025
2023-11-21 22:34:42,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1687493.3333333333, ans=0.1
2023-11-21 22:34:48,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1687493.3333333333, ans=15.0
2023-11-21 22:34:49,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1687493.3333333333, ans=0.125
2023-11-21 22:35:05,439 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 650, loss[loss=0.0573, simple_loss=0.07132, pruned_loss=0.01121, audio_tagging_loss=0.01043, over 14910.00 frames. ], tot_loss[loss=0.0735, simple_loss=0.09556, pruned_loss=0.01584, audio_tagging_loss=0.00988, over 2922700.54 frames. ], batch size: 56, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:35:10,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253150
2023-11-21 22:35:44,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1687826.6666666667, ans=0.125
2023-11-21 22:35:57,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1687893.3333333333, ans=0.0
2023-11-21 22:36:01,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.57 vs. limit=15.0
2023-11-21 22:36:09,021 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 700, loss[loss=0.07878, simple_loss=0.1057, pruned_loss=0.0151, audio_tagging_loss=0.01083, over 16206.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09545, pruned_loss=0.0157, audio_tagging_loss=0.009773, over 2958978.71 frames. ], batch size: 58, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:36:11,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1687960.0, ans=0.125
2023-11-21 22:36:12,010 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.13 vs. limit=15.0
2023-11-21 22:36:13,970 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253200
2023-11-21 22:36:28,396 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 8.099e+01 8.868e+01 9.467e+01 1.457e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-21 22:36:36,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1688093.3333333333, ans=0.125
2023-11-21 22:36:45,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1688093.3333333333, ans=0.0
2023-11-21 22:37:02,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1688226.6666666667, ans=0.1
2023-11-21 22:37:13,499 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 750, loss[loss=0.07292, simple_loss=0.09838, pruned_loss=0.01472, audio_tagging_loss=0.009016, over 16202.00 frames. ], tot_loss[loss=0.07297, simple_loss=0.09543, pruned_loss=0.01562, audio_tagging_loss=0.00963, over 2980656.00 frames. ], batch size: 60, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:37:18,555 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253250
2023-11-21 22:37:21,643 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.14 vs. limit=15.0
2023-11-21 22:37:46,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1688426.6666666667, ans=0.0
2023-11-21 22:37:50,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1688493.3333333333, ans=0.0
2023-11-21 22:37:52,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1688493.3333333333, ans=0.0
2023-11-21 22:38:11,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1688560.0, ans=0.1
2023-11-21 22:38:13,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1688560.0, ans=0.04949747468305833
2023-11-21 22:38:17,563 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 800, loss[loss=0.08276, simple_loss=0.1095, pruned_loss=0.02125, audio_tagging_loss=0.006759, over 15843.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09601, pruned_loss=0.01583, audio_tagging_loss=0.009613, over 2997355.34 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:38:22,592 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253300
2023-11-21 22:38:35,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1688693.3333333333, ans=0.0
2023-11-21 22:38:36,168 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.171e+01 8.697e+01 9.274e+01 1.215e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-21 22:38:41,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1688760.0, ans=0.04949747468305833
2023-11-21 22:39:11,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1688893.3333333333, ans=0.125
2023-11-21 22:39:11,929 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.73 vs. limit=22.5
2023-11-21 22:39:20,960 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 850, loss[loss=0.09649, simple_loss=0.1177, pruned_loss=0.02776, audio_tagging_loss=0.009873, over 14373.00 frames. ], tot_loss[loss=0.07369, simple_loss=0.09599, pruned_loss=0.016, audio_tagging_loss=0.009696, over 3010858.22 frames. ], batch size: 54, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:39:25,983 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253350
2023-11-21 22:39:29,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1688960.0, ans=0.0
2023-11-21 22:39:44,674 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:39:51,765 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.55 vs. limit=22.5
2023-11-21 22:39:59,192 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.15 vs. limit=15.0
2023-11-21 22:40:03,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1689160.0, ans=0.1
2023-11-21 22:40:04,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1689160.0, ans=0.125
2023-11-21 22:40:09,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1689160.0, ans=0.125
2023-11-21 22:40:24,931 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 900, loss[loss=0.073, simple_loss=0.1023, pruned_loss=0.01609, audio_tagging_loss=0.005731, over 16885.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09613, pruned_loss=0.01618, audio_tagging_loss=0.009802, over 3019971.32 frames. ], batch size: 62, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:40:30,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253400
2023-11-21 22:40:44,870 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.175e+01 9.057e+01 9.659e+01 1.593e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-21 22:40:45,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1689360.0, ans=0.1
2023-11-21 22:40:56,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1689426.6666666667, ans=0.125
2023-11-21 22:40:57,852 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.56 vs. limit=12.0
2023-11-21 22:41:00,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1689426.6666666667, ans=0.125
2023-11-21 22:41:30,288 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 950, loss[loss=0.05142, simple_loss=0.06106, pruned_loss=0.008724, audio_tagging_loss=0.01216, over 14416.00 frames. ], tot_loss[loss=0.07406, simple_loss=0.09632, pruned_loss=0.01617, audio_tagging_loss=0.009724, over 3031442.84 frames. ], batch size: 54, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:41:35,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253450
2023-11-21 22:41:46,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1689693.3333333333, ans=0.1
2023-11-21 22:41:47,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1689693.3333333333, ans=0.2
2023-11-21 22:42:05,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1689760.0, ans=0.125
2023-11-21 22:42:22,752 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.19 vs. limit=15.0
2023-11-21 22:42:24,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1689893.3333333333, ans=0.125
2023-11-21 22:42:29,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1689893.3333333333, ans=0.0
2023-11-21 22:42:32,965 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1000, loss[loss=0.07129, simple_loss=0.09061, pruned_loss=0.01779, audio_tagging_loss=0.008193, over 15317.00 frames. ], tot_loss[loss=0.07405, simple_loss=0.09651, pruned_loss=0.01619, audio_tagging_loss=0.0096, over 3037338.71 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:42:37,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253500
2023-11-21 22:42:51,807 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.522e+01 8.090e+01 8.690e+01 9.714e+01 1.265e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 22:42:53,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1690026.6666666667, ans=0.125
2023-11-21 22:42:59,984 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 22:43:37,582 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1050, loss[loss=0.08349, simple_loss=0.1097, pruned_loss=0.01944, audio_tagging_loss=0.009219, over 15892.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09582, pruned_loss=0.01603, audio_tagging_loss=0.009461, over 3037410.63 frames. ], batch size: 58, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:43:37,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1690293.3333333333, ans=0.0
2023-11-21 22:43:42,574 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253550
2023-11-21 22:43:42,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=1690293.3333333333, ans=0.95
2023-11-21 22:44:05,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1690426.6666666667, ans=0.0
2023-11-21 22:44:12,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1690426.6666666667, ans=0.125
2023-11-21 22:44:12,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1690426.6666666667, ans=0.0
2023-11-21 22:44:13,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1690426.6666666667, ans=0.0
2023-11-21 22:44:17,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1690493.3333333333, ans=0.0
2023-11-21 22:44:42,865 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1100, loss[loss=0.07803, simple_loss=0.1005, pruned_loss=0.01842, audio_tagging_loss=0.009333, over 14220.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09501, pruned_loss=0.01579, audio_tagging_loss=0.009418, over 3038570.85 frames. ], batch size: 55, lr: 3.14e-03, grad_scale: 32.0
2023-11-21 22:44:45,362 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 22:44:45,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1690626.6666666667, ans=0.1
2023-11-21 22:44:47,903 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253600
2023-11-21 22:45:01,571 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.847e+01 8.128e+01 8.851e+01 9.302e+01 1.185e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-21 22:45:09,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1690760.0, ans=0.0
2023-11-21 22:45:26,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1690826.6666666667, ans=0.125
2023-11-21 22:45:35,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1690893.3333333333, ans=0.125
2023-11-21 22:45:42,657 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:45:42,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1690893.3333333333, ans=0.0
2023-11-21 22:45:46,083 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1150, loss[loss=0.07143, simple_loss=0.1009, pruned_loss=0.01251, audio_tagging_loss=0.008471, over 15047.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09507, pruned_loss=0.01596, audio_tagging_loss=0.009421, over 3041467.34 frames. ], batch size: 56, lr: 3.14e-03, grad_scale: 8.0
2023-11-21 22:45:48,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1690960.0, ans=0.125
2023-11-21 22:45:51,096 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253650
2023-11-21 22:46:06,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1691026.6666666667, ans=0.2
2023-11-21 22:46:07,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1691026.6666666667, ans=0.2
2023-11-21 22:46:50,825 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1200, loss[loss=0.06849, simple_loss=0.08475, pruned_loss=0.0178, audio_tagging_loss=0.008314, over 15206.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09543, pruned_loss=0.01605, audio_tagging_loss=0.009431, over 3043033.96 frames. ], batch size: 57, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:46:55,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253700
2023-11-21 22:47:02,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1691360.0, ans=0.0
2023-11-21 22:47:04,957 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.27 vs. limit=15.0
2023-11-21 22:47:12,909 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.597e+01 8.044e+01 8.657e+01 9.227e+01 1.115e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 22:47:51,741 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.10 vs. limit=12.0
2023-11-21 22:47:54,541 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1250, loss[loss=0.06542, simple_loss=0.08155, pruned_loss=0.01455, audio_tagging_loss=0.01009, over 14673.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09514, pruned_loss=0.01599, audio_tagging_loss=0.009369, over 3036013.74 frames. ], batch size: 56, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:48:00,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253750
2023-11-21 22:48:12,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1691693.3333333333, ans=0.125
2023-11-21 22:48:25,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1691760.0, ans=0.1
2023-11-21 22:48:39,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1691826.6666666667, ans=0.125
2023-11-21 22:48:42,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1691826.6666666667, ans=0.125
2023-11-21 22:48:59,691 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1300, loss[loss=0.06972, simple_loss=0.09992, pruned_loss=0.014, audio_tagging_loss=0.005756, over 14152.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.0945, pruned_loss=0.01585, audio_tagging_loss=0.009481, over 3037014.98 frames. ], batch size: 54, lr: 3.14e-03, grad_scale: 16.0
2023-11-21 22:49:04,731 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253800
2023-11-21 22:49:16,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1692026.6666666667, ans=0.125
2023-11-21 22:49:21,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1692026.6666666667, ans=0.125
2023-11-21 22:49:22,326 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 7.957e+01 8.431e+01 9.281e+01 1.105e+02, threshold=1.686e+02, percent-clipped=0.0
2023-11-21 22:49:22,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1692026.6666666667, ans=0.04949747468305833
2023-11-21 22:50:04,314 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1350, loss[loss=0.07118, simple_loss=0.09442, pruned_loss=0.01368, audio_tagging_loss=0.01029, over 13910.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09477, pruned_loss=0.0159, audio_tagging_loss=0.009379, over 3037504.79 frames. ], batch size: 53, lr: 3.14e-03, grad_scale: 8.0
2023-11-21 22:50:10,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253850
2023-11-21 22:50:45,505 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.68 vs. limit=15.0
2023-11-21 22:50:51,073 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 22:51:00,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.79 vs. limit=15.0
2023-11-21 22:51:04,778 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:51:05,120 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.87 vs. limit=12.0
2023-11-21 22:51:09,551 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1400, loss[loss=0.06183, simple_loss=0.07137, pruned_loss=0.01326, audio_tagging_loss=0.01289, over 16488.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09368, pruned_loss=0.01575, audio_tagging_loss=0.009486, over 3039177.69 frames. ], batch size: 62, lr: 3.14e-03, grad_scale: 8.0
2023-11-21 22:51:13,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1692626.6666666667, ans=0.125
2023-11-21 22:51:15,231 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253900
2023-11-21 22:51:32,905 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.193e+01 8.723e+01 9.483e+01 1.296e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-21 22:51:48,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1692826.6666666667, ans=0.0
2023-11-21 22:52:15,022 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1450, loss[loss=0.07471, simple_loss=0.09416, pruned_loss=0.01632, audio_tagging_loss=0.01131, over 15688.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09335, pruned_loss=0.01557, audio_tagging_loss=0.009602, over 3044696.06 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 8.0
2023-11-21 22:52:19,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 253950
2023-11-21 22:52:22,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1692960.0, ans=0.0
2023-11-21 22:52:35,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1693026.6666666667, ans=0.125
2023-11-21 22:52:41,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1693093.3333333333, ans=0.125
2023-11-21 22:52:49,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1693093.3333333333, ans=0.125
2023-11-21 22:52:58,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1693160.0, ans=0.0
2023-11-21 22:53:08,672 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0
2023-11-21 22:53:15,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1693226.6666666667, ans=0.0
2023-11-21 22:53:19,090 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1500, loss[loss=0.0741, simple_loss=0.1005, pruned_loss=0.01431, audio_tagging_loss=0.009557, over 15860.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09424, pruned_loss=0.01587, audio_tagging_loss=0.009564, over 3040997.55 frames. ], batch size: 59, lr: 3.13e-03, grad_scale: 8.0
2023-11-21 22:53:24,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254000
2023-11-21 22:53:27,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1693293.3333333333, ans=0.125
2023-11-21 22:53:35,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1693360.0, ans=0.125
2023-11-21 22:53:37,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1693360.0, ans=0.0
2023-11-21 22:53:42,808 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.497e+01 8.377e+01 8.863e+01 9.600e+01 1.162e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-21 22:53:51,882 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.17 vs. limit=15.0
2023-11-21 22:54:08,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1693493.3333333333, ans=0.125
2023-11-21 22:54:23,597 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1550, loss[loss=0.07395, simple_loss=0.08827, pruned_loss=0.01857, audio_tagging_loss=0.01124, over 15474.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09394, pruned_loss=0.01597, audio_tagging_loss=0.009646, over 3040370.24 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 8.0
2023-11-21 22:54:25,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1693626.6666666667, ans=0.0
2023-11-21 22:54:28,520 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254050
2023-11-21 22:54:52,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1693760.0, ans=0.0
2023-11-21 22:55:16,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1693893.3333333333, ans=0.1
2023-11-21 22:55:27,427 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1600, loss[loss=0.05724, simple_loss=0.07129, pruned_loss=0.01282, audio_tagging_loss=0.008773, over 14703.00 frames. ], tot_loss[loss=0.07282, simple_loss=0.09396, pruned_loss=0.01604, audio_tagging_loss=0.009799, over 3042698.80 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:55:32,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254100
2023-11-21 22:55:36,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1693960.0, ans=0.1
2023-11-21 22:55:40,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1694026.6666666667, ans=0.0
2023-11-21 22:55:50,495 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 8.195e+01 8.712e+01 9.406e+01 1.124e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-21 22:56:12,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1694160.0, ans=0.125
2023-11-21 22:56:23,358 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0
2023-11-21 22:56:30,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1694293.3333333333, ans=0.1
2023-11-21 22:56:31,739 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1650, loss[loss=0.05294, simple_loss=0.06451, pruned_loss=0.01064, audio_tagging_loss=0.01004, over 14262.00 frames. ], tot_loss[loss=0.07285, simple_loss=0.09395, pruned_loss=0.01609, audio_tagging_loss=0.009786, over 3044776.29 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:56:36,636 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254150
2023-11-21 22:56:45,728 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.72 vs. limit=15.0
2023-11-21 22:56:48,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1694360.0, ans=0.0
2023-11-21 22:56:50,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1694360.0, ans=0.1
2023-11-21 22:56:54,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1694360.0, ans=0.125
2023-11-21 22:57:06,001 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.17 vs. limit=5.0
2023-11-21 22:57:36,314 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1700, loss[loss=0.07414, simple_loss=0.09391, pruned_loss=0.01554, audio_tagging_loss=0.01165, over 15813.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09378, pruned_loss=0.01594, audio_tagging_loss=0.009793, over 3046253.24 frames. ], batch size: 61, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:57:37,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1694626.6666666667, ans=0.125
2023-11-21 22:57:41,383 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254200
2023-11-21 22:57:44,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1694626.6666666667, ans=0.125
2023-11-21 22:57:58,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1694693.3333333333, ans=0.125
2023-11-21 22:57:59,284 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.322e+01 8.804e+01 9.363e+01 1.175e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-21 22:58:05,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1694760.0, ans=0.0
2023-11-21 22:58:18,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1694826.6666666667, ans=0.125
2023-11-21 22:58:36,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.52 vs. limit=15.0
2023-11-21 22:58:41,033 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1750, loss[loss=0.06873, simple_loss=0.08472, pruned_loss=0.01411, audio_tagging_loss=0.01226, over 14811.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.09432, pruned_loss=0.01591, audio_tagging_loss=0.009761, over 3048170.32 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:58:45,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254250
2023-11-21 22:58:46,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1694960.0, ans=0.05
2023-11-21 22:58:47,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1694960.0, ans=0.0
2023-11-21 22:58:48,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1694960.0, ans=0.125
2023-11-21 22:58:53,335 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 22:59:44,766 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1800, loss[loss=0.07099, simple_loss=0.1028, pruned_loss=0.01324, audio_tagging_loss=0.006365, over 15560.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09496, pruned_loss=0.01596, audio_tagging_loss=0.009637, over 3042617.60 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 22:59:50,303 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254300
2023-11-21 22:59:54,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1695293.3333333333, ans=0.125
2023-11-21 23:00:06,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1695360.0, ans=0.0
2023-11-21 23:00:08,561 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.157e+01 8.741e+01 9.334e+01 1.416e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-21 23:00:15,782 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.05 vs. limit=22.5
2023-11-21 23:00:35,550 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.84 vs. limit=15.0
2023-11-21 23:00:46,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1695560.0, ans=0.1
2023-11-21 23:00:49,756 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1850, loss[loss=0.09039, simple_loss=0.1131, pruned_loss=0.0254, audio_tagging_loss=0.008454, over 15118.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09564, pruned_loss=0.0162, audio_tagging_loss=0.009574, over 3041849.37 frames. ], batch size: 55, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:00:54,755 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254350
2023-11-21 23:01:08,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1695693.3333333333, ans=10.0
2023-11-21 23:01:20,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1695760.0, ans=0.125
2023-11-21 23:01:54,046 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1900, loss[loss=0.06464, simple_loss=0.07642, pruned_loss=0.01643, audio_tagging_loss=0.01001, over 15458.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09586, pruned_loss=0.01618, audio_tagging_loss=0.009488, over 3045942.86 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:01:58,483 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.88 vs. limit=15.0
2023-11-21 23:01:59,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254400
2023-11-21 23:02:06,068 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.15 vs. limit=15.0
2023-11-21 23:02:16,781 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.534e+01 8.079e+01 8.685e+01 9.740e+01 1.437e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-21 23:02:27,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1696093.3333333333, ans=0.0
2023-11-21 23:02:32,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1696160.0, ans=0.125
2023-11-21 23:02:33,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1696160.0, ans=10.0
2023-11-21 23:02:43,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1696160.0, ans=0.1
2023-11-21 23:02:58,596 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 1950, loss[loss=0.05777, simple_loss=0.07009, pruned_loss=0.01461, audio_tagging_loss=0.008116, over 16348.00 frames. ], tot_loss[loss=0.07301, simple_loss=0.09486, pruned_loss=0.01607, audio_tagging_loss=0.009507, over 3053428.82 frames. ], batch size: 62, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:03:00,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1696293.3333333333, ans=0.1
2023-11-21 23:03:03,545 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254450
2023-11-21 23:03:25,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1696426.6666666667, ans=0.07
2023-11-21 23:03:50,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1696560.0, ans=0.125
2023-11-21 23:04:02,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1696560.0, ans=0.0
2023-11-21 23:04:04,668 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2000, loss[loss=0.08495, simple_loss=0.1189, pruned_loss=0.01919, audio_tagging_loss=0.006296, over 14472.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09463, pruned_loss=0.01588, audio_tagging_loss=0.009491, over 3043132.58 frames. ], batch size: 54, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:04:09,651 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254500
2023-11-21 23:04:19,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1696693.3333333333, ans=0.125
2023-11-21 23:04:23,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1696693.3333333333, ans=0.125
2023-11-21 23:04:27,268 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 7.983e+01 8.610e+01 9.428e+01 1.340e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-21 23:04:40,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1696760.0, ans=0.1
2023-11-21 23:04:42,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1696826.6666666667, ans=0.1
2023-11-21 23:04:55,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1696893.3333333333, ans=0.125
2023-11-21 23:05:03,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1696893.3333333333, ans=0.0
2023-11-21 23:05:05,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1696893.3333333333, ans=0.125
2023-11-21 23:05:07,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1696960.0, ans=0.07
2023-11-21 23:05:09,027 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2050, loss[loss=0.08257, simple_loss=0.1071, pruned_loss=0.01847, audio_tagging_loss=0.01052, over 14791.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09547, pruned_loss=0.01601, audio_tagging_loss=0.009406, over 3042108.39 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:05:12,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1696960.0, ans=0.125
2023-11-21 23:05:14,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254550
2023-11-21 23:05:48,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1697160.0, ans=0.125
2023-11-21 23:06:04,547 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.60 vs. limit=22.5
2023-11-21 23:06:12,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1697293.3333333333, ans=0.0
2023-11-21 23:06:13,807 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2100, loss[loss=0.06723, simple_loss=0.09322, pruned_loss=0.0114, audio_tagging_loss=0.009225, over 14960.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.0956, pruned_loss=0.01601, audio_tagging_loss=0.009336, over 3040977.53 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:06:18,714 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254600
2023-11-21 23:06:37,860 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 8.098e+01 8.657e+01 9.385e+01 1.571e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-21 23:06:38,403 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.25 vs. limit=15.0
2023-11-21 23:06:42,140 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.45 vs. limit=15.0
2023-11-21 23:06:51,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0
2023-11-21 23:07:07,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=1697560.0, ans=0.5
2023-11-21 23:07:18,035 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2150, loss[loss=0.07297, simple_loss=0.09165, pruned_loss=0.01499, audio_tagging_loss=0.01215, over 15297.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09548, pruned_loss=0.01597, audio_tagging_loss=0.009333, over 3034378.31 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:07:23,690 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254650
2023-11-21 23:07:52,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1697760.0, ans=0.125
2023-11-21 23:07:56,836 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:07:58,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1697826.6666666667, ans=0.125
2023-11-21 23:08:23,095 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2200, loss[loss=0.07799, simple_loss=0.1089, pruned_loss=0.01543, audio_tagging_loss=0.008095, over 15059.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09579, pruned_loss=0.01575, audio_tagging_loss=0.009441, over 3037273.06 frames. ], batch size: 55, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:08:28,713 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254700
2023-11-21 23:08:47,049 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.502e+01 8.027e+01 8.507e+01 9.488e+01 1.436e+02, threshold=1.701e+02, percent-clipped=0.0
2023-11-21 23:08:56,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1698093.3333333333, ans=0.125
2023-11-21 23:09:08,258 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.32 vs. limit=15.0
2023-11-21 23:09:19,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1698226.6666666667, ans=0.0
2023-11-21 23:09:27,601 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2250, loss[loss=0.09627, simple_loss=0.133, pruned_loss=0.02255, audio_tagging_loss=0.007228, over 15024.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09612, pruned_loss=0.01581, audio_tagging_loss=0.009514, over 3035838.60 frames. ], batch size: 54, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:09:32,672 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254750
2023-11-21 23:09:35,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1698293.3333333333, ans=0.125
2023-11-21 23:10:00,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1698426.6666666667, ans=0.125
2023-11-21 23:10:01,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1698426.6666666667, ans=0.0
2023-11-21 23:10:12,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1698493.3333333333, ans=0.125
2023-11-21 23:10:12,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1698493.3333333333, ans=0.125
2023-11-21 23:10:15,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1698493.3333333333, ans=0.2
2023-11-21 23:10:19,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1698560.0, ans=0.0
2023-11-21 23:10:25,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.whiten.whitening_limit, batch_count=1698560.0, ans=15.0
2023-11-21 23:10:26,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1698560.0, ans=0.1
2023-11-21 23:10:32,046 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2300, loss[loss=0.07401, simple_loss=0.09075, pruned_loss=0.01748, audio_tagging_loss=0.01115, over 14609.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09567, pruned_loss=0.01573, audio_tagging_loss=0.009606, over 3038078.90 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:10:38,122 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254800
2023-11-21 23:10:41,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1698626.6666666667, ans=0.2
2023-11-21 23:10:54,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1698693.3333333333, ans=0.1
2023-11-21 23:10:57,737 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.992e+01 8.213e+01 8.771e+01 9.442e+01 1.167e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-21 23:11:05,729 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.32 vs. limit=10.0
2023-11-21 23:11:08,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.04 vs. limit=10.0
2023-11-21 23:11:23,639 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=22.5
2023-11-21 23:11:29,827 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:11:34,278 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.90 vs. limit=15.0
2023-11-21 23:11:37,158 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2350, loss[loss=0.07195, simple_loss=0.1021, pruned_loss=0.01317, audio_tagging_loss=0.007748, over 16625.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09565, pruned_loss=0.01574, audio_tagging_loss=0.009636, over 3033719.92 frames. ], batch size: 62, lr: 3.13e-03, grad_scale: 16.0
2023-11-21 23:11:43,431 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254850
2023-11-21 23:11:47,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1698960.0, ans=0.1
2023-11-21 23:11:49,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1699026.6666666667, ans=0.125
2023-11-21 23:11:50,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1699026.6666666667, ans=0.2
2023-11-21 23:12:05,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1699093.3333333333, ans=0.1
2023-11-21 23:12:30,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1699226.6666666667, ans=0.05
2023-11-21 23:12:42,282 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2400, loss[loss=0.08167, simple_loss=0.105, pruned_loss=0.01834, audio_tagging_loss=0.01082, over 15439.00 frames. ], tot_loss[loss=0.07313, simple_loss=0.09517, pruned_loss=0.01583, audio_tagging_loss=0.009718, over 3036486.55 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:12:47,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254900
2023-11-21 23:12:53,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1699360.0, ans=15.0
2023-11-21 23:13:04,152 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.08 vs. limit=6.0
2023-11-21 23:13:06,531 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 7.976e+01 8.691e+01 9.517e+01 1.439e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-21 23:13:29,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.whiten.whitening_limit, batch_count=1699493.3333333333, ans=15.0
2023-11-21 23:13:34,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1699560.0, ans=0.0
2023-11-21 23:13:36,171 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0
2023-11-21 23:13:41,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1699560.0, ans=0.025
2023-11-21 23:13:45,575 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2450, loss[loss=0.07587, simple_loss=0.09375, pruned_loss=0.01713, audio_tagging_loss=0.01188, over 15620.00 frames. ], tot_loss[loss=0.07312, simple_loss=0.09467, pruned_loss=0.01593, audio_tagging_loss=0.009856, over 3045693.53 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:13:51,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 254950
2023-11-21 23:13:56,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1699626.6666666667, ans=0.125
2023-11-21 23:14:03,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.62 vs. limit=10.0
2023-11-21 23:14:29,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1699826.6666666667, ans=0.125
2023-11-21 23:14:33,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1699826.6666666667, ans=0.0
2023-11-21 23:14:41,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1699893.3333333333, ans=0.0
2023-11-21 23:14:43,302 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.74 vs. limit=15.0
2023-11-21 23:14:50,015 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2500, loss[loss=0.0628, simple_loss=0.08381, pruned_loss=0.01356, audio_tagging_loss=0.007334, over 15872.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09451, pruned_loss=0.01575, audio_tagging_loss=0.0099, over 3048737.84 frames. ], batch size: 60, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:14:54,950 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255000
2023-11-21 23:15:04,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1700026.6666666667, ans=0.0
2023-11-21 23:15:10,762 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.56 vs. limit=22.5
2023-11-21 23:15:14,911 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.176e+01 8.833e+01 9.682e+01 1.336e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-21 23:15:32,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1700160.0, ans=0.025
2023-11-21 23:15:55,687 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2550, loss[loss=0.07358, simple_loss=0.08944, pruned_loss=0.01768, audio_tagging_loss=0.01118, over 14682.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09447, pruned_loss=0.01584, audio_tagging_loss=0.009801, over 3048017.30 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:16:00,766 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255050
2023-11-21 23:16:31,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1700426.6666666667, ans=0.1
2023-11-21 23:16:36,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1700493.3333333333, ans=0.1
2023-11-21 23:16:39,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1700493.3333333333, ans=0.0
2023-11-21 23:16:56,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1700560.0, ans=0.0
2023-11-21 23:17:00,031 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2600, loss[loss=0.07199, simple_loss=0.08829, pruned_loss=0.01671, audio_tagging_loss=0.01113, over 15017.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09476, pruned_loss=0.01572, audio_tagging_loss=0.009587, over 3048879.17 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:17:01,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1700626.6666666667, ans=0.2
2023-11-21 23:17:05,182 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255100
2023-11-21 23:17:19,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1700693.3333333333, ans=0.125
2023-11-21 23:17:23,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1700693.3333333333, ans=0.1
2023-11-21 23:17:24,817 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.035e+01 8.256e+01 8.734e+01 9.365e+01 1.832e+02, threshold=1.747e+02, percent-clipped=1.0
2023-11-21 23:17:58,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1700893.3333333333, ans=0.125
2023-11-21 23:18:05,215 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2650, loss[loss=0.07763, simple_loss=0.08926, pruned_loss=0.0218, audio_tagging_loss=0.0112, over 15131.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.0954, pruned_loss=0.01589, audio_tagging_loss=0.009475, over 3048833.07 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:18:10,040 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255150
2023-11-21 23:18:20,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1701026.6666666667, ans=0.125
2023-11-21 23:19:00,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1701226.6666666667, ans=0.0
2023-11-21 23:19:00,842 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.12 vs. limit=22.5
2023-11-21 23:19:09,750 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2700, loss[loss=0.06013, simple_loss=0.07839, pruned_loss=0.01234, audio_tagging_loss=0.008598, over 13733.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.0954, pruned_loss=0.01586, audio_tagging_loss=0.009472, over 3051190.70 frames. ], batch size: 55, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:19:10,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1701293.3333333333, ans=0.125
2023-11-21 23:19:14,910 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255200
2023-11-21 23:19:27,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1701360.0, ans=0.125
2023-11-21 23:19:34,273 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.128e+01 8.561e+01 9.314e+01 1.273e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 23:19:40,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1701426.6666666667, ans=0.125
2023-11-21 23:19:51,257 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.63 vs. limit=15.0
2023-11-21 23:20:15,122 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2750, loss[loss=0.07105, simple_loss=0.08309, pruned_loss=0.01804, audio_tagging_loss=0.01146, over 16375.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.0945, pruned_loss=0.01565, audio_tagging_loss=0.009563, over 3043708.47 frames. ], batch size: 62, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:20:20,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255250
2023-11-21 23:20:50,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1701760.0, ans=0.1
2023-11-21 23:21:02,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1701826.6666666667, ans=0.07
2023-11-21 23:21:06,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1701893.3333333333, ans=0.125
2023-11-21 23:21:07,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1701893.3333333333, ans=0.2
2023-11-21 23:21:08,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1701893.3333333333, ans=0.125
2023-11-21 23:21:09,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1701893.3333333333, ans=0.0
2023-11-21 23:21:10,819 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:21:16,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1701893.3333333333, ans=0.125
2023-11-21 23:21:16,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1701893.3333333333, ans=0.125
2023-11-21 23:21:18,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1701893.3333333333, ans=0.1
2023-11-21 23:21:20,711 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2800, loss[loss=0.06952, simple_loss=0.09245, pruned_loss=0.01432, audio_tagging_loss=0.008967, over 15343.00 frames. ], tot_loss[loss=0.07199, simple_loss=0.09398, pruned_loss=0.01549, audio_tagging_loss=0.009513, over 3042360.54 frames. ], batch size: 59, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:21:25,724 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255300
2023-11-21 23:21:28,338 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:21:44,828 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.479e+01 7.892e+01 8.472e+01 9.341e+01 1.172e+02, threshold=1.694e+02, percent-clipped=0.0
2023-11-21 23:21:45,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1702093.3333333333, ans=0.0
2023-11-21 23:21:48,868 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:22:03,374 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.02 vs. limit=15.0
2023-11-21 23:22:11,962 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.32 vs. limit=12.0
2023-11-21 23:22:13,925 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.62 vs. limit=10.0
2023-11-21 23:22:15,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.24 vs. limit=6.0
2023-11-21 23:22:25,135 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2850, loss[loss=0.08203, simple_loss=0.1111, pruned_loss=0.01801, audio_tagging_loss=0.008491, over 14488.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09431, pruned_loss=0.01556, audio_tagging_loss=0.009411, over 3043454.89 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:22:30,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255350
2023-11-21 23:22:50,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1702426.6666666667, ans=0.125
2023-11-21 23:23:28,845 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2900, loss[loss=0.07751, simple_loss=0.1074, pruned_loss=0.01458, audio_tagging_loss=0.00925, over 15407.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09524, pruned_loss=0.01569, audio_tagging_loss=0.009366, over 3042069.67 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:23:29,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1702626.6666666667, ans=0.125
2023-11-21 23:23:32,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1702626.6666666667, ans=0.125
2023-11-21 23:23:34,403 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255400
2023-11-21 23:23:34,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1702626.6666666667, ans=0.2
2023-11-21 23:23:54,131 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.570e+01 8.449e+01 9.076e+01 9.852e+01 1.265e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-21 23:23:54,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1702760.0, ans=0.0
2023-11-21 23:23:55,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1702760.0, ans=0.2
2023-11-21 23:24:33,956 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 2950, loss[loss=0.08854, simple_loss=0.1099, pruned_loss=0.02331, audio_tagging_loss=0.0103, over 15561.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09505, pruned_loss=0.0157, audio_tagging_loss=0.009367, over 3043200.66 frames. ], batch size: 57, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:24:38,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255450
2023-11-21 23:24:40,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1702960.0, ans=0.0
2023-11-21 23:24:58,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1703093.3333333333, ans=0.125
2023-11-21 23:24:59,271 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:25:21,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1703160.0, ans=0.125
2023-11-21 23:25:26,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1703226.6666666667, ans=0.125
2023-11-21 23:25:28,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1703226.6666666667, ans=0.1
2023-11-21 23:25:37,738 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3000, loss[loss=0.05614, simple_loss=0.06676, pruned_loss=0.01195, audio_tagging_loss=0.01081, over 15330.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09538, pruned_loss=0.01595, audio_tagging_loss=0.009582, over 3045761.56 frames. ], batch size: 58, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:25:37,741 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-21 23:26:16,406 INFO [train_asr.py:1253] (0/4) Epoch 22, validation: loss=0.05907, simple_loss=0.0519, pruned_loss=0.005126, audio_tagging_loss=0.02799, over 4681554.00 frames. 
2023-11-21 23:26:16,407 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-21 23:26:21,960 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255500
2023-11-21 23:26:25,249 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:26:28,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1703360.0, ans=0.2
2023-11-21 23:26:41,247 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 7.989e+01 8.698e+01 9.491e+01 1.260e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-21 23:26:43,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1703426.6666666667, ans=0.125
2023-11-21 23:27:00,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1703493.3333333333, ans=0.125
2023-11-21 23:27:21,990 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3050, loss[loss=0.07189, simple_loss=0.09166, pruned_loss=0.01822, audio_tagging_loss=0.00784, over 14275.00 frames. ], tot_loss[loss=0.07365, simple_loss=0.09589, pruned_loss=0.01612, audio_tagging_loss=0.009589, over 3039084.50 frames. ], batch size: 56, lr: 3.13e-03, grad_scale: 32.0
2023-11-21 23:27:22,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1703626.6666666667, ans=0.125
2023-11-21 23:27:26,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255550
2023-11-21 23:27:28,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1703626.6666666667, ans=0.125
2023-11-21 23:27:32,577 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:27:36,594 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.24 vs. limit=15.0
2023-11-21 23:27:50,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1703760.0, ans=0.2
2023-11-21 23:27:57,683 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:27:59,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1703826.6666666667, ans=0.125
2023-11-21 23:28:25,795 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3100, loss[loss=0.07641, simple_loss=0.1002, pruned_loss=0.01618, audio_tagging_loss=0.01012, over 15553.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09572, pruned_loss=0.01606, audio_tagging_loss=0.009681, over 3045517.13 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:28:25,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1703960.0, ans=0.0
2023-11-21 23:28:30,899 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255600
2023-11-21 23:28:36,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1703960.0, ans=0.1
2023-11-21 23:28:43,862 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.82 vs. limit=15.0
2023-11-21 23:28:48,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1704026.6666666667, ans=0.125
2023-11-21 23:28:51,160 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.488e+01 7.990e+01 8.561e+01 9.403e+01 1.205e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-21 23:29:01,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1704093.3333333333, ans=0.125
2023-11-21 23:29:14,435 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.08 vs. limit=6.0
2023-11-21 23:29:16,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1704226.6666666667, ans=0.0
2023-11-21 23:29:26,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1704226.6666666667, ans=0.125
2023-11-21 23:29:29,639 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3150, loss[loss=0.05131, simple_loss=0.06633, pruned_loss=0.007911, audio_tagging_loss=0.01023, over 15934.00 frames. ], tot_loss[loss=0.07404, simple_loss=0.09627, pruned_loss=0.01621, audio_tagging_loss=0.009689, over 3049053.71 frames. ], batch size: 60, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:29:34,686 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255650
2023-11-21 23:29:37,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1704293.3333333333, ans=0.0
2023-11-21 23:29:51,561 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.88 vs. limit=15.0
2023-11-21 23:30:12,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1704493.3333333333, ans=0.125
2023-11-21 23:30:18,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1704493.3333333333, ans=0.0
2023-11-21 23:30:35,319 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3200, loss[loss=0.08388, simple_loss=0.1052, pruned_loss=0.02171, audio_tagging_loss=0.00958, over 15021.00 frames. ], tot_loss[loss=0.07432, simple_loss=0.09679, pruned_loss=0.01616, audio_tagging_loss=0.009755, over 3052500.86 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:30:40,408 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255700
2023-11-21 23:30:51,289 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.05 vs. limit=15.0
2023-11-21 23:31:01,652 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.275e+01 8.900e+01 9.462e+01 1.368e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-21 23:31:29,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1704893.3333333333, ans=0.0
2023-11-21 23:31:30,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1704893.3333333333, ans=0.125
2023-11-21 23:31:35,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1704893.3333333333, ans=0.0
2023-11-21 23:31:39,799 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3250, loss[loss=0.08289, simple_loss=0.1102, pruned_loss=0.0205, audio_tagging_loss=0.007289, over 14884.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.0957, pruned_loss=0.01587, audio_tagging_loss=0.009805, over 3048477.74 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:31:44,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255750
2023-11-21 23:31:47,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1704960.0, ans=0.0
2023-11-21 23:31:50,464 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.62 vs. limit=12.0
2023-11-21 23:32:07,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1705093.3333333333, ans=0.2
2023-11-21 23:32:09,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1705093.3333333333, ans=0.125
2023-11-21 23:32:16,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1705093.3333333333, ans=0.2
2023-11-21 23:32:29,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1705160.0, ans=0.125
2023-11-21 23:32:30,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1705226.6666666667, ans=0.125
2023-11-21 23:32:35,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1705226.6666666667, ans=0.125
2023-11-21 23:32:38,558 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.71 vs. limit=12.0
2023-11-21 23:32:43,662 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3300, loss[loss=0.06634, simple_loss=0.08097, pruned_loss=0.01338, audio_tagging_loss=0.01247, over 15567.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09471, pruned_loss=0.01581, audio_tagging_loss=0.009902, over 3054188.06 frames. ], batch size: 61, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:32:48,599 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255800
2023-11-21 23:33:03,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1705360.0, ans=0.125
2023-11-21 23:33:11,149 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.185e+01 8.316e+01 8.844e+01 9.343e+01 1.264e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-21 23:33:17,239 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.69 vs. limit=12.0
2023-11-21 23:33:34,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1705560.0, ans=0.1
2023-11-21 23:33:38,885 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.07 vs. limit=15.0
2023-11-21 23:33:47,419 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3350, loss[loss=0.07166, simple_loss=0.09327, pruned_loss=0.01586, audio_tagging_loss=0.009161, over 15629.00 frames. ], tot_loss[loss=0.07239, simple_loss=0.09398, pruned_loss=0.01558, audio_tagging_loss=0.00982, over 3055972.22 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:33:52,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1705626.6666666667, ans=0.125
2023-11-21 23:33:52,994 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255850
2023-11-21 23:34:05,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1705693.3333333333, ans=0.0
2023-11-21 23:34:12,755 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.40 vs. limit=15.0
2023-11-21 23:34:52,876 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3400, loss[loss=0.07503, simple_loss=0.1016, pruned_loss=0.0165, audio_tagging_loss=0.007732, over 14837.00 frames. ], tot_loss[loss=0.07346, simple_loss=0.09561, pruned_loss=0.01599, audio_tagging_loss=0.00966, over 3057164.04 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:34:57,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255900
2023-11-21 23:35:11,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1706026.6666666667, ans=0.125
2023-11-21 23:35:16,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1706093.3333333333, ans=0.125
2023-11-21 23:35:18,273 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.503e+01 8.132e+01 8.792e+01 9.497e+01 1.160e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-21 23:35:45,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1706226.6666666667, ans=0.0
2023-11-21 23:35:50,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1706226.6666666667, ans=0.125
2023-11-21 23:35:56,092 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3450, loss[loss=0.03742, simple_loss=0.04502, pruned_loss=0.00579, audio_tagging_loss=0.009123, over 14963.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09589, pruned_loss=0.01603, audio_tagging_loss=0.009477, over 3044130.24 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:35:56,824 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.12 vs. limit=10.0
2023-11-21 23:35:57,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1706293.3333333333, ans=0.125
2023-11-21 23:36:01,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 255950
2023-11-21 23:36:23,181 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:36:42,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1706493.3333333333, ans=0.125
2023-11-21 23:36:48,912 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.97 vs. limit=15.0
2023-11-21 23:36:48,978 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.23 vs. limit=15.0
2023-11-21 23:36:59,405 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3500, loss[loss=0.08587, simple_loss=0.1063, pruned_loss=0.02293, audio_tagging_loss=0.009804, over 15322.00 frames. ], tot_loss[loss=0.07324, simple_loss=0.09546, pruned_loss=0.01604, audio_tagging_loss=0.009463, over 3036530.20 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:37:05,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256000
2023-11-21 23:37:05,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.51 vs. limit=15.0
2023-11-21 23:37:07,037 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-256000.pt
2023-11-21 23:37:15,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1706693.3333333333, ans=0.125
2023-11-21 23:37:31,121 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.462e+01 7.990e+01 8.553e+01 9.404e+01 1.412e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-21 23:37:37,249 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:37:43,895 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.02 vs. limit=12.0
2023-11-21 23:37:59,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1706893.3333333333, ans=0.0
2023-11-21 23:38:08,315 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3550, loss[loss=0.06558, simple_loss=0.08897, pruned_loss=0.01213, audio_tagging_loss=0.008967, over 15668.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.0953, pruned_loss=0.01591, audio_tagging_loss=0.009433, over 3037640.42 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:38:13,916 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256050
2023-11-21 23:38:16,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1706960.0, ans=0.0
2023-11-21 23:38:18,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1706960.0, ans=0.1
2023-11-21 23:38:28,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1707026.6666666667, ans=0.5
2023-11-21 23:39:12,559 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3600, loss[loss=0.04887, simple_loss=0.05874, pruned_loss=0.01011, audio_tagging_loss=0.009397, over 14967.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09549, pruned_loss=0.01598, audio_tagging_loss=0.009459, over 3040726.39 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:39:15,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1707293.3333333333, ans=0.125
2023-11-21 23:39:17,476 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256100
2023-11-21 23:39:18,104 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.53 vs. limit=15.0
2023-11-21 23:39:21,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1707293.3333333333, ans=0.2
2023-11-21 23:39:37,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1707426.6666666667, ans=0.125
2023-11-21 23:39:39,537 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.698e+01 8.079e+01 8.522e+01 9.097e+01 1.276e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-21 23:39:41,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1707426.6666666667, ans=0.125
2023-11-21 23:39:51,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1707493.3333333333, ans=0.0
2023-11-21 23:39:57,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1707493.3333333333, ans=0.125
2023-11-21 23:40:06,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1707560.0, ans=0.1
2023-11-21 23:40:16,172 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3650, loss[loss=0.07244, simple_loss=0.09558, pruned_loss=0.01613, audio_tagging_loss=0.008521, over 15752.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09548, pruned_loss=0.01609, audio_tagging_loss=0.009426, over 3040449.63 frames. ], batch size: 61, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:40:21,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256150
2023-11-21 23:40:37,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1707693.3333333333, ans=0.125
2023-11-21 23:40:50,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1707760.0, ans=0.0
2023-11-21 23:40:54,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1707826.6666666667, ans=0.1
2023-11-21 23:40:57,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1707826.6666666667, ans=0.0
2023-11-21 23:41:11,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1707893.3333333333, ans=0.125
2023-11-21 23:41:21,385 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3700, loss[loss=0.06508, simple_loss=0.07592, pruned_loss=0.01455, audio_tagging_loss=0.01257, over 14917.00 frames. ], tot_loss[loss=0.07312, simple_loss=0.09539, pruned_loss=0.01602, audio_tagging_loss=0.009404, over 3045539.86 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:41:27,188 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256200
2023-11-21 23:41:41,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1708026.6666666667, ans=0.125
2023-11-21 23:41:50,115 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.207e+01 8.757e+01 9.528e+01 1.164e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-21 23:42:00,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1708160.0, ans=0.0
2023-11-21 23:42:08,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1708160.0, ans=0.125
2023-11-21 23:42:12,035 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.31 vs. limit=12.0
2023-11-21 23:42:26,878 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3750, loss[loss=0.08469, simple_loss=0.1057, pruned_loss=0.01993, audio_tagging_loss=0.0119, over 15900.00 frames. ], tot_loss[loss=0.07301, simple_loss=0.09515, pruned_loss=0.01608, audio_tagging_loss=0.009357, over 3049888.28 frames. ], batch size: 61, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:42:31,909 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256250
2023-11-21 23:42:38,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1708360.0, ans=0.125
2023-11-21 23:42:49,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1708360.0, ans=0.125
2023-11-21 23:43:05,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1708493.3333333333, ans=0.1
2023-11-21 23:43:11,574 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:43:30,483 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3800, loss[loss=0.06657, simple_loss=0.08495, pruned_loss=0.0107, audio_tagging_loss=0.0134, over 15887.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09493, pruned_loss=0.016, audio_tagging_loss=0.009472, over 3051638.60 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:43:31,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1708626.6666666667, ans=0.125
2023-11-21 23:43:36,275 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256300
2023-11-21 23:43:37,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1708626.6666666667, ans=0.125
2023-11-21 23:43:55,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1708693.3333333333, ans=0.0
2023-11-21 23:43:59,430 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.560e+01 8.403e+01 8.924e+01 9.481e+01 1.120e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-21 23:43:59,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1708760.0, ans=0.0
2023-11-21 23:44:35,822 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3850, loss[loss=0.08034, simple_loss=0.1048, pruned_loss=0.01929, audio_tagging_loss=0.008651, over 15252.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.0953, pruned_loss=0.01593, audio_tagging_loss=0.009509, over 3048920.95 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:44:40,813 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256350
2023-11-21 23:44:48,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1709026.6666666667, ans=0.0
2023-11-21 23:45:00,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1709093.3333333333, ans=0.2
2023-11-21 23:45:05,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1709093.3333333333, ans=0.0
2023-11-21 23:45:33,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1709226.6666666667, ans=0.0
2023-11-21 23:45:35,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1709226.6666666667, ans=0.125
2023-11-21 23:45:40,176 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3900, loss[loss=0.09254, simple_loss=0.1266, pruned_loss=0.0166, audio_tagging_loss=0.01264, over 15564.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09471, pruned_loss=0.01581, audio_tagging_loss=0.009549, over 3043481.55 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:45:40,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1709293.3333333333, ans=0.0
2023-11-21 23:45:45,206 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256400
2023-11-21 23:45:52,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1709360.0, ans=0.125
2023-11-21 23:46:05,761 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0
2023-11-21 23:46:09,329 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.848e+01 8.019e+01 8.673e+01 9.477e+01 1.418e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-21 23:46:16,051 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.80 vs. limit=15.0
2023-11-21 23:46:20,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1709493.3333333333, ans=0.0
2023-11-21 23:46:26,281 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.67 vs. limit=10.0
2023-11-21 23:46:30,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1709560.0, ans=0.125
2023-11-21 23:46:33,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1709560.0, ans=0.2
2023-11-21 23:46:43,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1709560.0, ans=0.0
2023-11-21 23:46:45,103 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 3950, loss[loss=0.08968, simple_loss=0.1268, pruned_loss=0.0189, audio_tagging_loss=0.007406, over 15832.00 frames. ], tot_loss[loss=0.07307, simple_loss=0.09543, pruned_loss=0.01574, audio_tagging_loss=0.009614, over 3049864.81 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 16.0
2023-11-21 23:46:50,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256450
2023-11-21 23:47:04,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1709693.3333333333, ans=0.0
2023-11-21 23:47:20,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1709760.0, ans=0.2
2023-11-21 23:47:44,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1709893.3333333333, ans=0.05
2023-11-21 23:47:49,844 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4000, loss[loss=0.1014, simple_loss=0.1336, pruned_loss=0.02538, audio_tagging_loss=0.00921, over 15360.00 frames. ], tot_loss[loss=0.07419, simple_loss=0.09689, pruned_loss=0.01612, audio_tagging_loss=0.009622, over 3055961.13 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:47:54,751 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256500
2023-11-21 23:47:55,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1709960.0, ans=0.125
2023-11-21 23:48:08,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.75 vs. limit=15.0
2023-11-21 23:48:10,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1710026.6666666667, ans=0.2
2023-11-21 23:48:10,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1710026.6666666667, ans=0.125
2023-11-21 23:48:11,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1710026.6666666667, ans=0.1
2023-11-21 23:48:17,177 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.007e+01 8.495e+01 8.892e+01 9.387e+01 1.240e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-21 23:48:50,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1710226.6666666667, ans=0.04949747468305833
2023-11-21 23:48:51,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.61 vs. limit=15.0
2023-11-21 23:48:52,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1710293.3333333333, ans=0.0
2023-11-21 23:48:53,486 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4050, loss[loss=0.06071, simple_loss=0.07757, pruned_loss=0.01149, audio_tagging_loss=0.01044, over 15177.00 frames. ], tot_loss[loss=0.07392, simple_loss=0.09639, pruned_loss=0.01607, audio_tagging_loss=0.009656, over 3050472.17 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:48:57,255 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:48:58,526 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256550
2023-11-21 23:49:04,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1710360.0, ans=0.95
2023-11-21 23:49:27,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1710426.6666666667, ans=0.125
2023-11-21 23:49:42,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1710493.3333333333, ans=0.125
2023-11-21 23:49:57,005 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4100, loss[loss=0.09514, simple_loss=0.1337, pruned_loss=0.02143, audio_tagging_loss=0.006843, over 14796.00 frames. ], tot_loss[loss=0.07404, simple_loss=0.09674, pruned_loss=0.01607, audio_tagging_loss=0.009594, over 3051618.38 frames. ], batch size: 54, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:50:02,570 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256600
2023-11-21 23:50:26,956 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.295e+01 8.154e+01 8.730e+01 9.411e+01 1.441e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-21 23:50:27,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1710760.0, ans=0.125
2023-11-21 23:50:28,774 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=5.07 vs. limit=15.0
2023-11-21 23:50:32,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1710760.0, ans=0.125
2023-11-21 23:50:54,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1710893.3333333333, ans=0.1
2023-11-21 23:50:55,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1710893.3333333333, ans=0.125
2023-11-21 23:50:55,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1710893.3333333333, ans=0.0
2023-11-21 23:51:00,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1710893.3333333333, ans=0.2
2023-11-21 23:51:03,034 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4150, loss[loss=0.09436, simple_loss=0.1321, pruned_loss=0.02204, audio_tagging_loss=0.006271, over 15711.00 frames. ], tot_loss[loss=0.07385, simple_loss=0.09673, pruned_loss=0.01606, audio_tagging_loss=0.00942, over 3051183.55 frames. ], batch size: 55, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:51:08,163 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256650
2023-11-21 23:51:12,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1710960.0, ans=0.0
2023-11-21 23:51:12,508 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.54 vs. limit=6.0
2023-11-21 23:51:22,674 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-21 23:51:25,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1711026.6666666667, ans=0.125
2023-11-21 23:51:31,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1711093.3333333333, ans=0.1
2023-11-21 23:51:33,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1711093.3333333333, ans=0.0
2023-11-21 23:51:50,790 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-21 23:52:05,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1711226.6666666667, ans=0.125
2023-11-21 23:52:08,112 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4200, loss[loss=0.08079, simple_loss=0.1058, pruned_loss=0.01952, audio_tagging_loss=0.008368, over 15452.00 frames. ], tot_loss[loss=0.0737, simple_loss=0.0967, pruned_loss=0.01608, audio_tagging_loss=0.009281, over 3046539.70 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:52:13,158 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256700
2023-11-21 23:52:36,097 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.790e+01 7.996e+01 8.840e+01 9.314e+01 1.225e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-21 23:52:43,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1711426.6666666667, ans=0.2
2023-11-21 23:52:44,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1711426.6666666667, ans=0.125
2023-11-21 23:53:02,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1711560.0, ans=0.1
2023-11-21 23:53:11,831 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4250, loss[loss=0.08798, simple_loss=0.1199, pruned_loss=0.01906, audio_tagging_loss=0.008987, over 15923.00 frames. ], tot_loss[loss=0.07446, simple_loss=0.09775, pruned_loss=0.01641, audio_tagging_loss=0.009181, over 3044418.55 frames. ], batch size: 60, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:53:16,999 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256750
2023-11-21 23:53:17,772 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.23 vs. limit=22.5
2023-11-21 23:53:35,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1711693.3333333333, ans=0.125
2023-11-21 23:54:02,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1711893.3333333333, ans=0.125
2023-11-21 23:54:08,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1711893.3333333333, ans=0.125
2023-11-21 23:54:11,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1711893.3333333333, ans=0.125
2023-11-21 23:54:15,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1711960.0, ans=0.025
2023-11-21 23:54:16,097 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4300, loss[loss=0.06537, simple_loss=0.08178, pruned_loss=0.01472, audio_tagging_loss=0.009767, over 16661.00 frames. ], tot_loss[loss=0.07444, simple_loss=0.09794, pruned_loss=0.01634, audio_tagging_loss=0.009137, over 3052871.85 frames. ], batch size: 64, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:54:22,308 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256800
2023-11-21 23:54:45,106 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.276e+01 8.988e+01 9.758e+01 2.160e+02, threshold=1.798e+02, percent-clipped=1.0
2023-11-21 23:54:55,858 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=11.59 vs. limit=12.0
2023-11-21 23:55:12,528 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.12 vs. limit=12.0
2023-11-21 23:55:16,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1712226.6666666667, ans=0.125
2023-11-21 23:55:17,188 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.13 vs. limit=22.5
2023-11-21 23:55:22,075 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4350, loss[loss=0.05698, simple_loss=0.07325, pruned_loss=0.01043, audio_tagging_loss=0.009927, over 15271.00 frames. ], tot_loss[loss=0.07453, simple_loss=0.09801, pruned_loss=0.0164, audio_tagging_loss=0.009122, over 3052155.36 frames. ], batch size: 57, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:55:22,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1712293.3333333333, ans=0.125
2023-11-21 23:55:26,964 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256850
2023-11-21 23:55:50,520 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.05 vs. limit=15.0
2023-11-21 23:56:06,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1712493.3333333333, ans=0.0
2023-11-21 23:56:19,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1712560.0, ans=0.125
2023-11-21 23:56:25,331 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4400, loss[loss=0.08277, simple_loss=0.09424, pruned_loss=0.02596, audio_tagging_loss=0.009691, over 13572.00 frames. ], tot_loss[loss=0.07383, simple_loss=0.09721, pruned_loss=0.01611, audio_tagging_loss=0.009116, over 3053538.67 frames. ], batch size: 52, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:56:29,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1712626.6666666667, ans=0.125
2023-11-21 23:56:30,357 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256900
2023-11-21 23:56:33,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1712626.6666666667, ans=0.125
2023-11-21 23:56:51,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1712760.0, ans=0.125
2023-11-21 23:56:55,224 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.141e+01 8.812e+01 9.440e+01 1.228e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-21 23:57:29,763 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4450, loss[loss=0.05725, simple_loss=0.06515, pruned_loss=0.01401, audio_tagging_loss=0.01067, over 15104.00 frames. ], tot_loss[loss=0.07356, simple_loss=0.09676, pruned_loss=0.01609, audio_tagging_loss=0.009091, over 3049900.38 frames. ], batch size: 58, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:57:35,223 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 256950
2023-11-21 23:57:41,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1712960.0, ans=0.2
2023-11-21 23:57:50,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1713026.6666666667, ans=0.0
2023-11-21 23:58:12,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1713160.0, ans=0.2
2023-11-21 23:58:19,327 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.78 vs. limit=15.0
2023-11-21 23:58:34,710 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4500, loss[loss=0.06098, simple_loss=0.07645, pruned_loss=0.01285, audio_tagging_loss=0.009907, over 14677.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09583, pruned_loss=0.01595, audio_tagging_loss=0.00918, over 3050217.49 frames. ], batch size: 54, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:58:35,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1713293.3333333333, ans=0.125
2023-11-21 23:58:40,247 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257000
2023-11-21 23:58:54,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1713360.0, ans=0.125
2023-11-21 23:59:03,859 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.360e+01 9.049e+01 9.861e+01 1.318e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-21 23:59:04,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1713426.6666666667, ans=0.0
2023-11-21 23:59:16,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1713493.3333333333, ans=0.125
2023-11-21 23:59:39,441 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4550, loss[loss=0.07096, simple_loss=0.09615, pruned_loss=0.01245, audio_tagging_loss=0.01043, over 15432.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09545, pruned_loss=0.01596, audio_tagging_loss=0.009272, over 3049229.03 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 32.0
2023-11-21 23:59:44,364 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257050
2023-11-21 23:59:48,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1713626.6666666667, ans=0.125
2023-11-22 00:00:13,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1713760.0, ans=0.2
2023-11-22 00:00:26,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1713826.6666666667, ans=0.1
2023-11-22 00:00:27,896 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 00:00:42,546 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4600, loss[loss=0.09631, simple_loss=0.1299, pruned_loss=0.0249, audio_tagging_loss=0.00648, over 16047.00 frames. ], tot_loss[loss=0.07291, simple_loss=0.09523, pruned_loss=0.01595, audio_tagging_loss=0.009345, over 3053667.18 frames. ], batch size: 59, lr: 3.12e-03, grad_scale: 32.0
2023-11-22 00:00:44,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1713960.0, ans=0.1
2023-11-22 00:00:48,179 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257100
2023-11-22 00:00:52,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1713960.0, ans=10.0
2023-11-22 00:00:54,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1714026.6666666667, ans=0.125
2023-11-22 00:00:56,830 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.25 vs. limit=10.0
2023-11-22 00:01:13,015 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.884e+01 7.906e+01 8.473e+01 9.187e+01 1.183e+02, threshold=1.695e+02, percent-clipped=0.0
2023-11-22 00:01:26,120 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.12 vs. limit=6.0
2023-11-22 00:01:37,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1714226.6666666667, ans=0.0
2023-11-22 00:01:39,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1714226.6666666667, ans=0.0
2023-11-22 00:01:47,030 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4650, loss[loss=0.08841, simple_loss=0.1207, pruned_loss=0.01914, audio_tagging_loss=0.008915, over 14995.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09566, pruned_loss=0.01607, audio_tagging_loss=0.009372, over 3057693.08 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 32.0
2023-11-22 00:01:53,240 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257150
2023-11-22 00:01:53,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1714293.3333333333, ans=0.05
2023-11-22 00:01:54,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1714293.3333333333, ans=0.0
2023-11-22 00:02:11,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1714426.6666666667, ans=0.125
2023-11-22 00:02:16,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1714426.6666666667, ans=0.125
2023-11-22 00:02:51,274 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4700, loss[loss=0.05561, simple_loss=0.07028, pruned_loss=0.01162, audio_tagging_loss=0.008846, over 14615.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09447, pruned_loss=0.01585, audio_tagging_loss=0.009507, over 3057428.50 frames. ], batch size: 56, lr: 3.12e-03, grad_scale: 32.0
2023-11-22 00:02:56,417 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257200
2023-11-22 00:03:07,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1714693.3333333333, ans=0.0
2023-11-22 00:03:07,864 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:03:21,300 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.209e+01 8.969e+01 9.577e+01 1.305e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 00:03:45,479 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.14 vs. limit=22.5
2023-11-22 00:03:56,083 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4750, loss[loss=0.07931, simple_loss=0.1077, pruned_loss=0.0156, audio_tagging_loss=0.009861, over 13798.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09415, pruned_loss=0.01579, audio_tagging_loss=0.009576, over 3045168.70 frames. ], batch size: 54, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:03:58,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1714960.0, ans=0.125
2023-11-22 00:04:01,705 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257250
2023-11-22 00:04:09,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1715026.6666666667, ans=0.05
2023-11-22 00:04:10,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1715026.6666666667, ans=0.0
2023-11-22 00:04:17,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1715026.6666666667, ans=0.0
2023-11-22 00:04:17,480 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.63 vs. limit=22.5
2023-11-22 00:04:32,200 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.68 vs. limit=15.0
2023-11-22 00:04:54,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1715226.6666666667, ans=0.0
2023-11-22 00:04:58,863 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.35 vs. limit=22.5
2023-11-22 00:05:00,624 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4800, loss[loss=0.068, simple_loss=0.08455, pruned_loss=0.01342, audio_tagging_loss=0.0123, over 15495.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09469, pruned_loss=0.01575, audio_tagging_loss=0.009606, over 3054198.72 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:05:05,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257300
2023-11-22 00:05:05,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1715293.3333333333, ans=0.07
2023-11-22 00:05:18,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1715360.0, ans=0.125
2023-11-22 00:05:30,004 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.329e+01 8.144e+01 8.815e+01 9.467e+01 1.388e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 00:05:31,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1715426.6666666667, ans=0.125
2023-11-22 00:05:34,338 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.21 vs. limit=15.0
2023-11-22 00:05:56,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1715560.0, ans=0.0
2023-11-22 00:06:05,300 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4850, loss[loss=0.06932, simple_loss=0.08701, pruned_loss=0.01483, audio_tagging_loss=0.01099, over 14719.00 frames. ], tot_loss[loss=0.073, simple_loss=0.095, pruned_loss=0.01574, audio_tagging_loss=0.009764, over 3056498.69 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:06:10,257 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257350
2023-11-22 00:07:06,006 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.12 vs. limit=15.0
2023-11-22 00:07:09,100 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4900, loss[loss=0.07444, simple_loss=0.107, pruned_loss=0.01017, audio_tagging_loss=0.01078, over 15435.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09439, pruned_loss=0.01556, audio_tagging_loss=0.00985, over 3045328.47 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:07:14,209 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257400
2023-11-22 00:07:16,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1715960.0, ans=0.125
2023-11-22 00:07:17,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1715960.0, ans=0.125
2023-11-22 00:07:27,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.34 vs. limit=12.0
2023-11-22 00:07:38,990 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.481e+01 8.110e+01 8.831e+01 9.694e+01 1.184e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 00:07:44,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1716093.3333333333, ans=0.125
2023-11-22 00:07:47,763 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.76 vs. limit=15.0
2023-11-22 00:07:48,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1716160.0, ans=0.125
2023-11-22 00:07:55,347 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.96 vs. limit=15.0
2023-11-22 00:07:57,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1716160.0, ans=0.1
2023-11-22 00:08:02,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1716226.6666666667, ans=0.125
2023-11-22 00:08:07,029 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=2.92 vs. limit=12.0
2023-11-22 00:08:13,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1716293.3333333333, ans=0.125
2023-11-22 00:08:13,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.95 vs. limit=10.0
2023-11-22 00:08:14,175 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 4950, loss[loss=0.05555, simple_loss=0.07765, pruned_loss=0.009922, audio_tagging_loss=0.006798, over 14131.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09396, pruned_loss=0.01548, audio_tagging_loss=0.009638, over 3045057.98 frames. ], batch size: 52, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:08:16,116 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.97 vs. limit=15.0
2023-11-22 00:08:19,139 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257450
2023-11-22 00:08:24,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1716293.3333333333, ans=0.0
2023-11-22 00:08:53,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1716493.3333333333, ans=0.1
2023-11-22 00:09:17,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1716626.6666666667, ans=0.125
2023-11-22 00:09:18,240 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5000, loss[loss=0.07452, simple_loss=0.09045, pruned_loss=0.01707, audio_tagging_loss=0.01223, over 15811.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09445, pruned_loss=0.01552, audio_tagging_loss=0.009518, over 3041202.83 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:09:23,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257500
2023-11-22 00:09:25,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1716626.6666666667, ans=0.2
2023-11-22 00:09:30,400 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.30 vs. limit=15.0
2023-11-22 00:09:33,695 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:09:48,025 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.146e+01 8.929e+01 9.625e+01 1.123e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 00:09:48,816 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.85 vs. limit=22.5
2023-11-22 00:09:53,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1716760.0, ans=0.125
2023-11-22 00:09:54,866 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.69 vs. limit=10.0
2023-11-22 00:10:12,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1716893.3333333333, ans=0.125
2023-11-22 00:10:13,261 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:10:22,307 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5050, loss[loss=0.06409, simple_loss=0.08113, pruned_loss=0.01132, audio_tagging_loss=0.01221, over 15602.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09438, pruned_loss=0.01546, audio_tagging_loss=0.009419, over 3043628.54 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:10:27,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257550
2023-11-22 00:10:33,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1716960.0, ans=0.125
2023-11-22 00:10:51,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1717093.3333333333, ans=0.125
2023-11-22 00:10:53,743 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.00 vs. limit=15.0
2023-11-22 00:11:04,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1717160.0, ans=0.125
2023-11-22 00:11:22,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1717226.6666666667, ans=0.125
2023-11-22 00:11:28,006 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5100, loss[loss=0.04994, simple_loss=0.0655, pruned_loss=0.005491, audio_tagging_loss=0.0117, over 14817.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09403, pruned_loss=0.01548, audio_tagging_loss=0.009381, over 3039632.51 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:11:33,100 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257600
2023-11-22 00:11:33,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1717293.3333333333, ans=0.2
2023-11-22 00:11:51,067 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.80 vs. limit=15.0
2023-11-22 00:11:53,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1717426.6666666667, ans=0.0
2023-11-22 00:11:58,354 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.309e+01 7.923e+01 8.751e+01 9.756e+01 1.347e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 00:11:58,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1717426.6666666667, ans=0.125
2023-11-22 00:12:03,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1717426.6666666667, ans=0.125
2023-11-22 00:12:06,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1717493.3333333333, ans=10.0
2023-11-22 00:12:34,121 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5150, loss[loss=0.07366, simple_loss=0.09787, pruned_loss=0.01786, audio_tagging_loss=0.006871, over 16339.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.0937, pruned_loss=0.01543, audio_tagging_loss=0.009463, over 3042225.52 frames. ], batch size: 60, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:12:39,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257650
2023-11-22 00:12:57,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1717693.3333333333, ans=0.125
2023-11-22 00:13:09,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1717760.0, ans=0.125
2023-11-22 00:13:26,385 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:13:36,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1717893.3333333333, ans=0.0
2023-11-22 00:13:39,406 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5200, loss[loss=0.07909, simple_loss=0.09498, pruned_loss=0.02026, audio_tagging_loss=0.01134, over 15224.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09455, pruned_loss=0.01568, audio_tagging_loss=0.009461, over 3043559.64 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:13:42,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1717960.0, ans=0.125
2023-11-22 00:13:44,470 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257700
2023-11-22 00:13:47,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1717960.0, ans=0.0
2023-11-22 00:13:54,188 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.77 vs. limit=15.0
2023-11-22 00:14:09,393 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.143e+01 8.728e+01 9.361e+01 2.351e+02, threshold=1.746e+02, percent-clipped=1.0
2023-11-22 00:14:43,507 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5250, loss[loss=0.08115, simple_loss=0.1121, pruned_loss=0.01633, audio_tagging_loss=0.008761, over 15220.00 frames. ], tot_loss[loss=0.07282, simple_loss=0.09539, pruned_loss=0.01585, audio_tagging_loss=0.009269, over 3045105.10 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:14:49,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257750
2023-11-22 00:14:49,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1718293.3333333333, ans=0.0
2023-11-22 00:14:55,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1718360.0, ans=0.0
2023-11-22 00:15:12,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1718426.6666666667, ans=0.1
2023-11-22 00:15:20,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1718493.3333333333, ans=0.0
2023-11-22 00:15:28,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1718493.3333333333, ans=0.0
2023-11-22 00:15:39,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1718560.0, ans=0.125
2023-11-22 00:15:48,167 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5300, loss[loss=0.08464, simple_loss=0.1064, pruned_loss=0.02197, audio_tagging_loss=0.009479, over 14781.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.09456, pruned_loss=0.01583, audio_tagging_loss=0.009381, over 3045971.28 frames. ], batch size: 52, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:15:53,217 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257800
2023-11-22 00:15:59,037 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0
2023-11-22 00:16:02,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1718693.3333333333, ans=0.1
2023-11-22 00:16:17,183 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.827e+01 8.246e+01 8.912e+01 9.804e+01 1.159e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 00:16:26,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=1718826.6666666667, ans=15.0
2023-11-22 00:16:46,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1718893.3333333333, ans=0.2
2023-11-22 00:16:52,131 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5350, loss[loss=0.06747, simple_loss=0.08952, pruned_loss=0.01367, audio_tagging_loss=0.009039, over 15617.00 frames. ], tot_loss[loss=0.07348, simple_loss=0.09614, pruned_loss=0.01612, audio_tagging_loss=0.009292, over 3044462.39 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:16:57,282 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257850
2023-11-22 00:17:02,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.15 vs. limit=22.5
2023-11-22 00:17:20,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1719093.3333333333, ans=0.125
2023-11-22 00:17:27,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1719093.3333333333, ans=0.125
2023-11-22 00:17:36,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1719160.0, ans=0.125
2023-11-22 00:17:49,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1719226.6666666667, ans=0.0
2023-11-22 00:17:57,089 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5400, loss[loss=0.0944, simple_loss=0.1231, pruned_loss=0.0234, audio_tagging_loss=0.009469, over 15939.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09594, pruned_loss=0.0161, audio_tagging_loss=0.009337, over 3046995.32 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:18:02,564 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257900
2023-11-22 00:18:20,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1719360.0, ans=0.125
2023-11-22 00:18:22,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1719426.6666666667, ans=0.0
2023-11-22 00:18:27,285 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.62 vs. limit=15.0
2023-11-22 00:18:27,541 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 7.882e+01 8.565e+01 9.189e+01 1.192e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-22 00:18:44,729 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.48 vs. limit=6.0
2023-11-22 00:18:45,978 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.83 vs. limit=10.0
2023-11-22 00:19:01,136 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5450, loss[loss=0.07967, simple_loss=0.09646, pruned_loss=0.01772, audio_tagging_loss=0.01372, over 15091.00 frames. ], tot_loss[loss=0.07382, simple_loss=0.09632, pruned_loss=0.01621, audio_tagging_loss=0.009453, over 3054027.54 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:19:06,705 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 257950
2023-11-22 00:19:23,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1719693.3333333333, ans=0.0
2023-11-22 00:20:05,636 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5500, loss[loss=0.09319, simple_loss=0.1175, pruned_loss=0.02671, audio_tagging_loss=0.007723, over 15614.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09658, pruned_loss=0.01625, audio_tagging_loss=0.009541, over 3055367.07 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:20:10,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258000
2023-11-22 00:20:10,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1719960.0, ans=0.1
2023-11-22 00:20:14,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1719960.0, ans=0.04949747468305833
2023-11-22 00:20:18,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1720026.6666666667, ans=0.125
2023-11-22 00:20:20,356 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.57 vs. limit=8.0
2023-11-22 00:20:36,646 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.285e+01 8.796e+01 9.521e+01 1.194e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 00:20:56,519 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.51 vs. limit=15.0
2023-11-22 00:21:03,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1720226.6666666667, ans=0.1
2023-11-22 00:21:09,035 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5550, loss[loss=0.09073, simple_loss=0.1173, pruned_loss=0.01989, audio_tagging_loss=0.01218, over 14771.00 frames. ], tot_loss[loss=0.07393, simple_loss=0.09638, pruned_loss=0.01619, audio_tagging_loss=0.009559, over 3059761.32 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:21:15,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258050
2023-11-22 00:21:17,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten.whitening_limit, batch_count=1720293.3333333333, ans=22.5
2023-11-22 00:21:26,183 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.91 vs. limit=15.0
2023-11-22 00:21:36,672 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.89 vs. limit=15.0
2023-11-22 00:22:05,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1720560.0, ans=0.5
2023-11-22 00:22:13,632 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5600, loss[loss=0.06528, simple_loss=0.08674, pruned_loss=0.01499, audio_tagging_loss=0.006922, over 14949.00 frames. ], tot_loss[loss=0.07342, simple_loss=0.09579, pruned_loss=0.01589, audio_tagging_loss=0.00963, over 3051928.31 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:22:17,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1720626.6666666667, ans=0.0
2023-11-22 00:22:19,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258100
2023-11-22 00:22:36,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1720693.3333333333, ans=0.125
2023-11-22 00:22:43,449 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 7.788e+01 8.528e+01 9.166e+01 1.353e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-22 00:22:51,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1720826.6666666667, ans=0.125
2023-11-22 00:22:59,869 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 00:23:00,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1720826.6666666667, ans=0.0
2023-11-22 00:23:16,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1720960.0, ans=0.0
2023-11-22 00:23:17,594 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5650, loss[loss=0.06854, simple_loss=0.08586, pruned_loss=0.01173, audio_tagging_loss=0.01388, over 15706.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.096, pruned_loss=0.01584, audio_tagging_loss=0.009697, over 3046723.08 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:23:22,664 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258150
2023-11-22 00:23:35,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1721026.6666666667, ans=0.1
2023-11-22 00:23:49,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1721093.3333333333, ans=0.2
2023-11-22 00:23:50,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1721093.3333333333, ans=0.125
2023-11-22 00:24:20,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1721293.3333333333, ans=0.0
2023-11-22 00:24:21,012 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5700, loss[loss=0.06868, simple_loss=0.07793, pruned_loss=0.01686, audio_tagging_loss=0.01285, over 14751.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09565, pruned_loss=0.01567, audio_tagging_loss=0.009673, over 3048002.21 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:24:21,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1721293.3333333333, ans=0.0
2023-11-22 00:24:25,940 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258200
2023-11-22 00:24:39,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1721360.0, ans=0.125
2023-11-22 00:24:39,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1721360.0, ans=0.1
2023-11-22 00:24:52,712 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.074e+01 8.821e+01 9.592e+01 1.274e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 00:24:56,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1721426.6666666667, ans=0.1
2023-11-22 00:25:12,081 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.35 vs. limit=15.0
2023-11-22 00:25:26,484 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5750, loss[loss=0.09046, simple_loss=0.125, pruned_loss=0.02198, audio_tagging_loss=0.005998, over 15082.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.09588, pruned_loss=0.0158, audio_tagging_loss=0.009479, over 3048765.99 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:25:26,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1721626.6666666667, ans=0.0
2023-11-22 00:25:31,519 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258250
2023-11-22 00:25:38,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1721693.3333333333, ans=0.125
2023-11-22 00:25:54,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1721760.0, ans=0.0
2023-11-22 00:26:14,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1721826.6666666667, ans=0.125
2023-11-22 00:26:24,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1721893.3333333333, ans=0.0
2023-11-22 00:26:29,998 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5800, loss[loss=0.07369, simple_loss=0.09198, pruned_loss=0.01611, audio_tagging_loss=0.0116, over 14752.00 frames. ], tot_loss[loss=0.07344, simple_loss=0.09614, pruned_loss=0.01599, audio_tagging_loss=0.009379, over 3047668.51 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:26:32,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1721960.0, ans=0.0
2023-11-22 00:26:34,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258300
2023-11-22 00:26:35,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1721960.0, ans=0.2
2023-11-22 00:26:38,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1721960.0, ans=0.125
2023-11-22 00:26:43,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1722026.6666666667, ans=0.125
2023-11-22 00:26:55,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1722093.3333333333, ans=0.1
2023-11-22 00:27:01,820 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.499e+01 8.125e+01 8.820e+01 9.414e+01 1.363e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 00:27:07,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1722160.0, ans=0.025
2023-11-22 00:27:16,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=1722160.0, ans=0.05
2023-11-22 00:27:23,369 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.87 vs. limit=22.5
2023-11-22 00:27:33,621 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5850, loss[loss=0.06503, simple_loss=0.08068, pruned_loss=0.01351, audio_tagging_loss=0.01118, over 15752.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09492, pruned_loss=0.01582, audio_tagging_loss=0.009421, over 3042351.10 frames. ], batch size: 63, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:27:38,585 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258350
2023-11-22 00:27:38,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1722293.3333333333, ans=0.125
2023-11-22 00:27:45,655 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:27:45,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1722360.0, ans=0.125
2023-11-22 00:27:47,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1722360.0, ans=0.125
2023-11-22 00:28:27,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1722560.0, ans=0.2
2023-11-22 00:28:32,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1722560.0, ans=0.0
2023-11-22 00:28:38,110 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5900, loss[loss=0.05553, simple_loss=0.07471, pruned_loss=0.009491, audio_tagging_loss=0.008681, over 14639.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09492, pruned_loss=0.01581, audio_tagging_loss=0.009339, over 3041364.36 frames. ], batch size: 55, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:28:43,187 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258400
2023-11-22 00:29:03,665 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.70 vs. limit=15.0
2023-11-22 00:29:10,522 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.841e+01 8.273e+01 8.751e+01 9.457e+01 2.979e+02, threshold=1.750e+02, percent-clipped=1.0
2023-11-22 00:29:31,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1722893.3333333333, ans=0.1
2023-11-22 00:29:43,333 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 5950, loss[loss=0.06073, simple_loss=0.07689, pruned_loss=0.01353, audio_tagging_loss=0.008753, over 15324.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09534, pruned_loss=0.01589, audio_tagging_loss=0.00928, over 3045483.49 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 16.0
2023-11-22 00:29:48,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258450
2023-11-22 00:29:48,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1722960.0, ans=0.125
2023-11-22 00:29:53,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1722960.0, ans=0.0
2023-11-22 00:29:59,783 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.34 vs. limit=22.5
2023-11-22 00:30:10,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1723093.3333333333, ans=0.125
2023-11-22 00:30:16,718 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:30:31,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1723160.0, ans=0.05
2023-11-22 00:30:36,437 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.48 vs. limit=15.0
2023-11-22 00:30:38,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.42 vs. limit=10.0
2023-11-22 00:30:46,888 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6000, loss[loss=0.06436, simple_loss=0.09457, pruned_loss=0.01074, audio_tagging_loss=0.006338, over 14925.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09507, pruned_loss=0.01573, audio_tagging_loss=0.009176, over 3047560.24 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:30:46,891 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 00:31:22,786 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7419, 5.7998, 5.8507, 5.8103], device='cuda:0')
2023-11-22 00:31:27,182 INFO [train_asr.py:1253] (0/4) Epoch 22, validation: loss=0.05958, simple_loss=0.05193, pruned_loss=0.005178, audio_tagging_loss=0.02843, over 4681554.00 frames. 
2023-11-22 00:31:27,183 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 00:31:27,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1723293.3333333333, ans=0.125
2023-11-22 00:31:32,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258500
2023-11-22 00:31:35,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1723293.3333333333, ans=0.1
2023-11-22 00:31:39,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1723360.0, ans=0.125
2023-11-22 00:31:50,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1723360.0, ans=0.0
2023-11-22 00:31:57,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1723426.6666666667, ans=0.0
2023-11-22 00:31:58,331 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.255e+01 8.695e+01 9.611e+01 1.390e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 00:32:02,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1723426.6666666667, ans=0.0
2023-11-22 00:32:09,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1723493.3333333333, ans=0.0
2023-11-22 00:32:13,717 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.25 vs. limit=15.0
2023-11-22 00:32:14,260 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 00:32:20,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1723560.0, ans=0.0
2023-11-22 00:32:25,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1723560.0, ans=0.125
2023-11-22 00:32:31,173 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6050, loss[loss=0.07318, simple_loss=0.1005, pruned_loss=0.01506, audio_tagging_loss=0.007868, over 14158.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.0955, pruned_loss=0.01581, audio_tagging_loss=0.009149, over 3048044.86 frames. ], batch size: 54, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:32:36,088 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258550
2023-11-22 00:33:00,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1723760.0, ans=0.0
2023-11-22 00:33:00,989 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.81 vs. limit=15.0
2023-11-22 00:33:08,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1723826.6666666667, ans=0.125
2023-11-22 00:33:18,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1723826.6666666667, ans=0.1
2023-11-22 00:33:25,528 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.99 vs. limit=15.0
2023-11-22 00:33:32,735 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.85 vs. limit=6.0
2023-11-22 00:33:34,433 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6100, loss[loss=0.07141, simple_loss=0.09646, pruned_loss=0.01565, audio_tagging_loss=0.00753, over 15108.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09536, pruned_loss=0.01587, audio_tagging_loss=0.009185, over 3040157.41 frames. ], batch size: 58, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:33:39,400 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258600
2023-11-22 00:33:54,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1724026.6666666667, ans=0.125
2023-11-22 00:34:06,999 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.187e+01 8.838e+01 9.553e+01 1.618e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 00:34:11,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1724093.3333333333, ans=0.0
2023-11-22 00:34:23,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1724160.0, ans=0.0
2023-11-22 00:34:33,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1724226.6666666667, ans=0.125
2023-11-22 00:34:39,402 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6150, loss[loss=0.05637, simple_loss=0.07418, pruned_loss=0.008606, audio_tagging_loss=0.01068, over 15605.00 frames. ], tot_loss[loss=0.07265, simple_loss=0.095, pruned_loss=0.01582, audio_tagging_loss=0.009329, over 3045269.78 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:34:39,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1724293.3333333333, ans=10.0
2023-11-22 00:34:44,399 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258650
2023-11-22 00:35:01,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1724360.0, ans=0.05
2023-11-22 00:35:21,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1724493.3333333333, ans=0.09899494936611666
2023-11-22 00:35:26,893 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:35:37,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1724560.0, ans=0.1
2023-11-22 00:35:43,807 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6200, loss[loss=0.06432, simple_loss=0.083, pruned_loss=0.01149, audio_tagging_loss=0.01133, over 14738.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09405, pruned_loss=0.0155, audio_tagging_loss=0.009443, over 3048206.66 frames. ], batch size: 57, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:35:48,831 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258700
2023-11-22 00:35:48,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1724626.6666666667, ans=0.0
2023-11-22 00:36:02,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.90 vs. limit=10.0
2023-11-22 00:36:15,754 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.702e+01 8.055e+01 8.630e+01 9.332e+01 1.528e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 00:36:37,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1724893.3333333333, ans=0.0
2023-11-22 00:36:48,009 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6250, loss[loss=0.06698, simple_loss=0.08338, pruned_loss=0.01211, audio_tagging_loss=0.01318, over 15349.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09455, pruned_loss=0.01558, audio_tagging_loss=0.009603, over 3045331.77 frames. ], batch size: 56, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:36:53,063 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258750
2023-11-22 00:36:58,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.30 vs. limit=15.0
2023-11-22 00:37:23,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1725093.3333333333, ans=10.0
2023-11-22 00:37:26,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1725160.0, ans=0.125
2023-11-22 00:37:27,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.07 vs. limit=10.0
2023-11-22 00:37:34,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1725160.0, ans=0.0
2023-11-22 00:37:47,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1725226.6666666667, ans=0.125
2023-11-22 00:37:52,622 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6300, loss[loss=0.06044, simple_loss=0.07095, pruned_loss=0.013, audio_tagging_loss=0.01196, over 14963.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09492, pruned_loss=0.01557, audio_tagging_loss=0.009631, over 3045885.01 frames. ], batch size: 59, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:37:58,178 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258800
2023-11-22 00:38:08,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1725360.0, ans=0.125
2023-11-22 00:38:15,563 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.272e-02
2023-11-22 00:38:25,690 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.059e+01 8.458e+01 8.852e+01 9.578e+01 1.235e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 00:38:29,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1725426.6666666667, ans=0.1
2023-11-22 00:38:29,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1725426.6666666667, ans=0.125
2023-11-22 00:38:38,157 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:38:43,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1725560.0, ans=0.0
2023-11-22 00:38:52,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1725560.0, ans=0.1
2023-11-22 00:38:58,129 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6350, loss[loss=0.09395, simple_loss=0.118, pruned_loss=0.02245, audio_tagging_loss=0.01249, over 14727.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09539, pruned_loss=0.01574, audio_tagging_loss=0.009726, over 3046473.92 frames. ], batch size: 54, lr: 3.11e-03, grad_scale: 32.0
2023-11-22 00:39:00,188 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.98 vs. limit=10.0
2023-11-22 00:39:01,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1725626.6666666667, ans=0.125
2023-11-22 00:39:03,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258850
2023-11-22 00:39:08,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.18 vs. limit=10.0
2023-11-22 00:39:28,742 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.34 vs. limit=22.5
2023-11-22 00:39:32,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1725760.0, ans=0.0
2023-11-22 00:39:43,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1725826.6666666667, ans=0.0
2023-11-22 00:39:54,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1725893.3333333333, ans=0.2
2023-11-22 00:39:56,099 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.62 vs. limit=15.0
2023-11-22 00:40:01,459 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6400, loss[loss=0.06976, simple_loss=0.07968, pruned_loss=0.01556, audio_tagging_loss=0.01435, over 15834.00 frames. ], tot_loss[loss=0.07283, simple_loss=0.0951, pruned_loss=0.01556, audio_tagging_loss=0.009723, over 3047136.89 frames. ], batch size: 60, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:40:06,994 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258900
2023-11-22 00:40:07,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1725960.0, ans=0.125
2023-11-22 00:40:17,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.55 vs. limit=22.5
2023-11-22 00:40:21,487 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.38 vs. limit=12.0
2023-11-22 00:40:34,293 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.092e+01 8.951e+01 9.607e+01 2.279e+02, threshold=1.790e+02, percent-clipped=1.0
2023-11-22 00:40:34,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1726093.3333333333, ans=0.0
2023-11-22 00:40:34,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1726093.3333333333, ans=0.125
2023-11-22 00:40:35,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1726093.3333333333, ans=0.0
2023-11-22 00:41:05,263 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6450, loss[loss=0.06559, simple_loss=0.08338, pruned_loss=0.01458, audio_tagging_loss=0.009327, over 15496.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09575, pruned_loss=0.01573, audio_tagging_loss=0.009791, over 3047856.18 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:41:08,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1726293.3333333333, ans=0.125
2023-11-22 00:41:11,506 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 258950
2023-11-22 00:41:14,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1726293.3333333333, ans=0.125
2023-11-22 00:41:26,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1726360.0, ans=0.0
2023-11-22 00:41:29,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1726360.0, ans=0.1
2023-11-22 00:41:44,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1726493.3333333333, ans=0.0
2023-11-22 00:41:54,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1726493.3333333333, ans=0.1
2023-11-22 00:42:02,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1726560.0, ans=0.0
2023-11-22 00:42:09,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1726626.6666666667, ans=0.1
2023-11-22 00:42:09,590 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.47 vs. limit=15.0
2023-11-22 00:42:10,285 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6500, loss[loss=0.1023, simple_loss=0.1271, pruned_loss=0.03006, audio_tagging_loss=0.008685, over 14831.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.09653, pruned_loss=0.01599, audio_tagging_loss=0.009698, over 3051038.75 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:42:15,955 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259000
2023-11-22 00:42:42,947 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.155e+01 8.144e+01 8.732e+01 9.307e+01 1.366e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 00:42:47,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1726760.0, ans=0.125
2023-11-22 00:42:57,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1726826.6666666667, ans=0.2
2023-11-22 00:43:15,082 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6550, loss[loss=0.05912, simple_loss=0.07189, pruned_loss=0.0145, audio_tagging_loss=0.008675, over 14174.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.0954, pruned_loss=0.01598, audio_tagging_loss=0.009692, over 3042252.44 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:43:20,268 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259050
2023-11-22 00:43:28,902 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.58 vs. limit=15.0
2023-11-22 00:43:34,269 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:43:49,022 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.17 vs. limit=10.0
2023-11-22 00:44:02,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.24 vs. limit=22.5
2023-11-22 00:44:14,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1727226.6666666667, ans=0.0
2023-11-22 00:44:18,551 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6600, loss[loss=0.05391, simple_loss=0.07346, pruned_loss=0.009823, audio_tagging_loss=0.007354, over 13479.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09559, pruned_loss=0.01599, audio_tagging_loss=0.009511, over 3040327.65 frames. ], batch size: 52, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:44:24,021 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259100
2023-11-22 00:44:39,656 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.48 vs. limit=22.5
2023-11-22 00:44:51,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1727426.6666666667, ans=0.0
2023-11-22 00:44:52,438 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.795e+01 8.221e+01 8.740e+01 9.559e+01 1.333e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 00:44:54,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1727426.6666666667, ans=0.125
2023-11-22 00:45:23,536 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6650, loss[loss=0.07787, simple_loss=0.1053, pruned_loss=0.01903, audio_tagging_loss=0.006208, over 14927.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09482, pruned_loss=0.01582, audio_tagging_loss=0.009443, over 3038143.85 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:45:25,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1727626.6666666667, ans=0.125
2023-11-22 00:45:29,251 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259150
2023-11-22 00:45:37,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1727693.3333333333, ans=10.0
2023-11-22 00:45:38,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1727693.3333333333, ans=0.1
2023-11-22 00:45:44,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1727693.3333333333, ans=0.2
2023-11-22 00:45:52,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1727760.0, ans=0.125
2023-11-22 00:46:00,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1727826.6666666667, ans=0.125
2023-11-22 00:46:23,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1727893.3333333333, ans=0.0
2023-11-22 00:46:27,268 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6700, loss[loss=0.09315, simple_loss=0.1209, pruned_loss=0.02514, audio_tagging_loss=0.007568, over 14630.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09492, pruned_loss=0.01596, audio_tagging_loss=0.009361, over 3042977.00 frames. ], batch size: 53, lr: 3.10e-03, grad_scale: 8.0
2023-11-22 00:46:32,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259200
2023-11-22 00:46:43,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1728026.6666666667, ans=0.2
2023-11-22 00:46:46,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1728026.6666666667, ans=0.125
2023-11-22 00:46:48,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1728026.6666666667, ans=0.2
2023-11-22 00:47:01,894 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 7.937e+01 8.531e+01 9.262e+01 1.112e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-22 00:47:02,719 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.23 vs. limit=15.0
2023-11-22 00:47:03,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1728093.3333333333, ans=0.0
2023-11-22 00:47:09,334 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.95 vs. limit=15.0
2023-11-22 00:47:24,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1728226.6666666667, ans=0.0
2023-11-22 00:47:30,819 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6750, loss[loss=0.07888, simple_loss=0.09331, pruned_loss=0.01796, audio_tagging_loss=0.01427, over 15377.00 frames. ], tot_loss[loss=0.07319, simple_loss=0.09558, pruned_loss=0.016, audio_tagging_loss=0.009403, over 3038336.91 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 8.0
2023-11-22 00:47:32,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1728293.3333333333, ans=0.125
2023-11-22 00:47:35,709 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259250
2023-11-22 00:47:55,844 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.50 vs. limit=15.0
2023-11-22 00:47:56,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1728426.6666666667, ans=0.0
2023-11-22 00:48:05,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1728426.6666666667, ans=0.035
2023-11-22 00:48:32,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1728560.0, ans=0.0
2023-11-22 00:48:35,747 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.75 vs. limit=15.0
2023-11-22 00:48:36,237 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6800, loss[loss=0.07385, simple_loss=0.09089, pruned_loss=0.01537, audio_tagging_loss=0.01303, over 16164.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09614, pruned_loss=0.01616, audio_tagging_loss=0.009343, over 3038337.01 frames. ], batch size: 61, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:48:37,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1728626.6666666667, ans=0.125
2023-11-22 00:48:38,959 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 00:48:41,216 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259300
2023-11-22 00:48:57,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1728693.3333333333, ans=0.125
2023-11-22 00:49:04,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1728760.0, ans=0.125
2023-11-22 00:49:09,689 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.092e+01 8.012e+01 8.646e+01 9.288e+01 1.206e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 00:49:40,203 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6850, loss[loss=0.06237, simple_loss=0.07901, pruned_loss=0.01249, audio_tagging_loss=0.01037, over 14336.00 frames. ], tot_loss[loss=0.07288, simple_loss=0.09515, pruned_loss=0.01595, audio_tagging_loss=0.009364, over 3041997.26 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:49:45,203 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259350
2023-11-22 00:50:01,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1729026.6666666667, ans=0.0
2023-11-22 00:50:01,617 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.06 vs. limit=15.0
2023-11-22 00:50:07,093 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.80 vs. limit=12.0
2023-11-22 00:50:25,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1729160.0, ans=0.125
2023-11-22 00:50:28,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1729160.0, ans=0.2
2023-11-22 00:50:43,751 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6900, loss[loss=0.08373, simple_loss=0.1009, pruned_loss=0.02443, audio_tagging_loss=0.008879, over 15663.00 frames. ], tot_loss[loss=0.07378, simple_loss=0.09674, pruned_loss=0.01614, audio_tagging_loss=0.009263, over 3049698.56 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:50:48,753 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259400
2023-11-22 00:51:05,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1729360.0, ans=0.125
2023-11-22 00:51:19,570 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.107e+01 8.289e+01 8.755e+01 9.420e+01 1.385e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 00:51:21,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1729426.6666666667, ans=0.2
2023-11-22 00:51:23,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1729493.3333333333, ans=0.0
2023-11-22 00:51:27,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1729493.3333333333, ans=0.1
2023-11-22 00:51:35,569 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 00:51:41,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.21 vs. limit=8.0
2023-11-22 00:51:49,213 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 6950, loss[loss=0.09151, simple_loss=0.1145, pruned_loss=0.02544, audio_tagging_loss=0.008824, over 16250.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09559, pruned_loss=0.0159, audio_tagging_loss=0.009364, over 3050357.34 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:51:54,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259450
2023-11-22 00:52:23,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1729760.0, ans=0.0
2023-11-22 00:52:34,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1729826.6666666667, ans=0.1
2023-11-22 00:52:36,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1729826.6666666667, ans=0.125
2023-11-22 00:52:43,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=1729893.3333333333, ans=0.05
2023-11-22 00:52:54,405 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7000, loss[loss=0.0688, simple_loss=0.08344, pruned_loss=0.01538, audio_tagging_loss=0.01169, over 14731.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.0944, pruned_loss=0.01556, audio_tagging_loss=0.009407, over 3046788.14 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:52:54,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1729960.0, ans=0.125
2023-11-22 00:52:59,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259500
2023-11-22 00:53:28,623 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.568e+01 7.948e+01 8.560e+01 9.274e+01 1.120e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-22 00:53:42,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1730160.0, ans=0.125
2023-11-22 00:53:55,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1730226.6666666667, ans=0.0
2023-11-22 00:53:56,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1730226.6666666667, ans=0.0
2023-11-22 00:53:58,428 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7050, loss[loss=0.06831, simple_loss=0.08488, pruned_loss=0.01463, audio_tagging_loss=0.01124, over 15686.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09414, pruned_loss=0.01566, audio_tagging_loss=0.009499, over 3044419.71 frames. ], batch size: 60, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:54:03,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259550
2023-11-22 00:54:08,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1730293.3333333333, ans=0.125
2023-11-22 00:54:09,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1730360.0, ans=0.0
2023-11-22 00:54:32,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1730426.6666666667, ans=0.125
2023-11-22 00:54:44,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1730493.3333333333, ans=0.0
2023-11-22 00:54:52,463 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.58 vs. limit=5.0
2023-11-22 00:54:54,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1730560.0, ans=0.125
2023-11-22 00:54:59,421 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.65 vs. limit=10.0
2023-11-22 00:55:02,337 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7100, loss[loss=0.07214, simple_loss=0.09084, pruned_loss=0.01239, audio_tagging_loss=0.01433, over 15359.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.0945, pruned_loss=0.01574, audio_tagging_loss=0.009587, over 3048403.35 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:55:03,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1730626.6666666667, ans=0.125
2023-11-22 00:55:07,474 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259600
2023-11-22 00:55:30,699 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.24 vs. limit=10.0
2023-11-22 00:55:37,159 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.901e+01 8.168e+01 8.717e+01 9.423e+01 1.311e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 00:55:37,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1730760.0, ans=0.1
2023-11-22 00:55:43,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1730826.6666666667, ans=0.07
2023-11-22 00:55:52,866 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.35 vs. limit=6.0
2023-11-22 00:55:59,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1730893.3333333333, ans=0.2
2023-11-22 00:56:07,814 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7150, loss[loss=0.09146, simple_loss=0.1144, pruned_loss=0.0216, audio_tagging_loss=0.01265, over 16230.00 frames. ], tot_loss[loss=0.07329, simple_loss=0.09564, pruned_loss=0.01592, audio_tagging_loss=0.009542, over 3058601.37 frames. ], batch size: 59, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 00:56:08,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1730960.0, ans=0.0
2023-11-22 00:56:09,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.48 vs. limit=10.0
2023-11-22 00:56:12,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259650
2023-11-22 00:56:21,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1731026.6666666667, ans=0.125
2023-11-22 00:56:23,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1731026.6666666667, ans=0.125
2023-11-22 00:56:24,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1731026.6666666667, ans=0.2
2023-11-22 00:56:24,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1731026.6666666667, ans=0.125
2023-11-22 00:56:25,890 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.33 vs. limit=15.0
2023-11-22 00:56:43,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1731093.3333333333, ans=0.0
2023-11-22 00:56:43,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1731093.3333333333, ans=0.0
2023-11-22 00:56:54,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1731160.0, ans=0.125
2023-11-22 00:57:09,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1731226.6666666667, ans=0.1
2023-11-22 00:57:11,640 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7200, loss[loss=0.08166, simple_loss=0.1047, pruned_loss=0.01741, audio_tagging_loss=0.01191, over 16188.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09523, pruned_loss=0.01579, audio_tagging_loss=0.00964, over 3055174.46 frames. ], batch size: 60, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:57:16,561 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259700
2023-11-22 00:57:20,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1731293.3333333333, ans=0.125
2023-11-22 00:57:25,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1731360.0, ans=0.125
2023-11-22 00:57:29,248 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.20 vs. limit=6.0
2023-11-22 00:57:36,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1731426.6666666667, ans=0.0
2023-11-22 00:57:38,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1731426.6666666667, ans=0.2
2023-11-22 00:57:41,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1731426.6666666667, ans=0.125
2023-11-22 00:57:46,560 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.390e+01 8.081e+01 8.874e+01 9.710e+01 1.298e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 00:57:58,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=1731493.3333333333, ans=0.5
2023-11-22 00:58:15,117 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7250, loss[loss=0.04155, simple_loss=0.04532, pruned_loss=0.008695, audio_tagging_loss=0.0102, over 13302.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09442, pruned_loss=0.01565, audio_tagging_loss=0.009699, over 3046683.53 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:58:21,207 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259750
2023-11-22 00:58:45,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1731760.0, ans=0.125
2023-11-22 00:58:48,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1731760.0, ans=0.125
2023-11-22 00:58:52,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1731826.6666666667, ans=0.0
2023-11-22 00:59:12,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1731893.3333333333, ans=0.0
2023-11-22 00:59:17,268 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.50 vs. limit=12.0
2023-11-22 00:59:19,755 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7300, loss[loss=0.04993, simple_loss=0.05798, pruned_loss=0.01026, audio_tagging_loss=0.01068, over 14933.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09371, pruned_loss=0.01555, audio_tagging_loss=0.009676, over 3040197.73 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 00:59:21,833 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.81 vs. limit=15.0
2023-11-22 00:59:22,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1731960.0, ans=0.1
2023-11-22 00:59:24,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259800
2023-11-22 00:59:49,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1732093.3333333333, ans=0.09899494936611666
2023-11-22 00:59:50,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1732093.3333333333, ans=0.2
2023-11-22 00:59:53,080 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.552e+01 8.104e+01 8.588e+01 9.277e+01 1.159e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 01:00:08,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1732160.0, ans=0.1
2023-11-22 01:00:14,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1732226.6666666667, ans=0.0
2023-11-22 01:00:23,338 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7350, loss[loss=0.08313, simple_loss=0.1163, pruned_loss=0.0185, audio_tagging_loss=0.006469, over 16416.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09471, pruned_loss=0.01568, audio_tagging_loss=0.009461, over 3044882.72 frames. ], batch size: 59, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:00:25,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=1732293.3333333333, ans=15.0
2023-11-22 01:00:28,293 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259850
2023-11-22 01:00:35,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1732360.0, ans=0.1
2023-11-22 01:00:45,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1732360.0, ans=0.125
2023-11-22 01:00:46,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1732360.0, ans=0.125
2023-11-22 01:00:55,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1732426.6666666667, ans=0.035
2023-11-22 01:01:04,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1732493.3333333333, ans=0.04949747468305833
2023-11-22 01:01:25,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1732626.6666666667, ans=0.125
2023-11-22 01:01:26,429 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7400, loss[loss=0.07526, simple_loss=0.09659, pruned_loss=0.01748, audio_tagging_loss=0.009476, over 14928.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09451, pruned_loss=0.01575, audio_tagging_loss=0.009426, over 3041870.67 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:01:32,011 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259900
2023-11-22 01:01:36,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1732626.6666666667, ans=0.0
2023-11-22 01:01:41,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1732693.3333333333, ans=0.125
2023-11-22 01:01:59,696 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.91 vs. limit=15.0
2023-11-22 01:02:02,676 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.170e+01 8.751e+01 9.458e+01 1.086e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 01:02:24,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1732893.3333333333, ans=0.125
2023-11-22 01:02:30,995 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7450, loss[loss=0.06938, simple_loss=0.08971, pruned_loss=0.01666, audio_tagging_loss=0.007862, over 14077.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09325, pruned_loss=0.01556, audio_tagging_loss=0.00947, over 3046988.66 frames. ], batch size: 53, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:02:36,541 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 259950
2023-11-22 01:02:43,111 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.61 vs. limit=22.5
2023-11-22 01:02:55,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1733093.3333333333, ans=0.125
2023-11-22 01:02:59,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.77 vs. limit=15.0
2023-11-22 01:03:20,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.56 vs. limit=15.0
2023-11-22 01:03:35,333 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7500, loss[loss=0.07191, simple_loss=0.09793, pruned_loss=0.01554, audio_tagging_loss=0.007403, over 16227.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09329, pruned_loss=0.01551, audio_tagging_loss=0.009379, over 3043767.04 frames. ], batch size: 60, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:03:40,950 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260000
2023-11-22 01:03:42,433 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-260000.pt
2023-11-22 01:04:05,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1733426.6666666667, ans=0.125
2023-11-22 01:04:14,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1733426.6666666667, ans=0.0
2023-11-22 01:04:14,749 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.000e+01 8.690e+01 9.325e+01 1.229e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 01:04:15,614 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.28 vs. limit=22.5
2023-11-22 01:04:22,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1733493.3333333333, ans=0.0
2023-11-22 01:04:42,090 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7550, loss[loss=0.08161, simple_loss=0.1104, pruned_loss=0.01889, audio_tagging_loss=0.007512, over 16085.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09474, pruned_loss=0.01578, audio_tagging_loss=0.009306, over 3054406.54 frames. ], batch size: 59, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:04:47,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260050
2023-11-22 01:04:55,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.89 vs. limit=22.5
2023-11-22 01:05:14,596 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.59 vs. limit=15.0
2023-11-22 01:05:24,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1733826.6666666667, ans=0.07
2023-11-22 01:05:25,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1733826.6666666667, ans=0.2
2023-11-22 01:05:33,359 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.73 vs. limit=15.0
2023-11-22 01:05:45,951 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7600, loss[loss=0.05345, simple_loss=0.07542, pruned_loss=0.008112, audio_tagging_loss=0.007624, over 14479.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.0935, pruned_loss=0.01551, audio_tagging_loss=0.009333, over 3045452.30 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:05:46,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1733960.0, ans=0.125
2023-11-22 01:05:51,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260100
2023-11-22 01:05:52,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1733960.0, ans=0.04949747468305833
2023-11-22 01:05:55,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1733960.0, ans=0.0
2023-11-22 01:05:56,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1733960.0, ans=0.125
2023-11-22 01:06:01,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.07 vs. limit=15.0
2023-11-22 01:06:13,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1734093.3333333333, ans=0.1
2023-11-22 01:06:21,716 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.674e+01 8.289e+01 8.897e+01 9.780e+01 1.167e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 01:06:22,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1734093.3333333333, ans=0.125
2023-11-22 01:06:27,973 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.94 vs. limit=10.0
2023-11-22 01:06:36,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1734226.6666666667, ans=0.125
2023-11-22 01:06:43,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1734226.6666666667, ans=0.125
2023-11-22 01:06:49,636 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7650, loss[loss=0.06135, simple_loss=0.08653, pruned_loss=0.008255, audio_tagging_loss=0.009834, over 15033.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09401, pruned_loss=0.01562, audio_tagging_loss=0.009316, over 3032871.03 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:06:49,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1734293.3333333333, ans=0.0
2023-11-22 01:06:54,494 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260150
2023-11-22 01:07:39,308 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:07:53,001 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7700, loss[loss=0.07354, simple_loss=0.09501, pruned_loss=0.01725, audio_tagging_loss=0.008786, over 15058.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09408, pruned_loss=0.01545, audio_tagging_loss=0.009271, over 3034138.69 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:07:53,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1734626.6666666667, ans=0.04949747468305833
2023-11-22 01:07:57,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260200
2023-11-22 01:08:02,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1734626.6666666667, ans=0.5
2023-11-22 01:08:02,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1734626.6666666667, ans=0.2
2023-11-22 01:08:09,830 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.42 vs. limit=22.5
2023-11-22 01:08:14,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1734693.3333333333, ans=0.125
2023-11-22 01:08:26,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1734760.0, ans=0.2
2023-11-22 01:08:29,042 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 7.976e+01 8.890e+01 9.341e+01 1.175e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 01:08:43,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1734893.3333333333, ans=0.0
2023-11-22 01:08:44,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1734893.3333333333, ans=0.0
2023-11-22 01:08:57,775 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7750, loss[loss=0.07673, simple_loss=0.09447, pruned_loss=0.02004, audio_tagging_loss=0.00945, over 14559.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09489, pruned_loss=0.01564, audio_tagging_loss=0.009298, over 3036410.05 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:09:02,698 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260250
2023-11-22 01:09:04,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1734960.0, ans=0.1
2023-11-22 01:09:17,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1735026.6666666667, ans=0.125
2023-11-22 01:09:31,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1735093.3333333333, ans=0.0
2023-11-22 01:09:37,013 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.44 vs. limit=15.0
2023-11-22 01:09:50,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1735226.6666666667, ans=0.1
2023-11-22 01:10:01,239 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7800, loss[loss=0.06869, simple_loss=0.08784, pruned_loss=0.01487, audio_tagging_loss=0.009894, over 15005.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09542, pruned_loss=0.0157, audio_tagging_loss=0.00932, over 3039874.41 frames. ], batch size: 55, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:10:06,191 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260300
2023-11-22 01:10:12,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1735360.0, ans=0.0
2023-11-22 01:10:27,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.07 vs. limit=15.0
2023-11-22 01:10:33,610 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:10:37,014 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.748e+01 8.146e+01 9.007e+01 9.749e+01 1.228e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 01:11:04,762 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7850, loss[loss=0.06421, simple_loss=0.07796, pruned_loss=0.01184, audio_tagging_loss=0.01339, over 15181.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09626, pruned_loss=0.01586, audio_tagging_loss=0.009307, over 3049397.37 frames. ], batch size: 57, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:11:09,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260350
2023-11-22 01:11:13,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1735626.6666666667, ans=0.125
2023-11-22 01:11:14,691 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.78 vs. limit=22.5
2023-11-22 01:11:16,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1735693.3333333333, ans=0.0
2023-11-22 01:11:22,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=1735693.3333333333, ans=0.05
2023-11-22 01:11:27,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1735693.3333333333, ans=0.1
2023-11-22 01:11:37,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1735760.0, ans=0.05
2023-11-22 01:12:07,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1735893.3333333333, ans=0.125
2023-11-22 01:12:09,885 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7900, loss[loss=0.08197, simple_loss=0.1098, pruned_loss=0.01824, audio_tagging_loss=0.008812, over 16257.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09535, pruned_loss=0.01579, audio_tagging_loss=0.009406, over 3048394.93 frames. ], batch size: 61, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:12:14,987 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260400
2023-11-22 01:12:46,391 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.144e+01 8.401e+01 8.796e+01 9.438e+01 1.194e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 01:12:55,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1736160.0, ans=0.125
2023-11-22 01:13:13,833 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 7950, loss[loss=0.06029, simple_loss=0.07301, pruned_loss=0.01327, audio_tagging_loss=0.01051, over 15692.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09569, pruned_loss=0.01608, audio_tagging_loss=0.00948, over 3046921.95 frames. ], batch size: 60, lr: 3.10e-03, grad_scale: 16.0
2023-11-22 01:13:15,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1736293.3333333333, ans=0.0
2023-11-22 01:13:18,758 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260450
2023-11-22 01:13:29,888 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:14:16,702 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8000, loss[loss=0.06826, simple_loss=0.09399, pruned_loss=0.0132, audio_tagging_loss=0.008065, over 15472.00 frames. ], tot_loss[loss=0.073, simple_loss=0.09489, pruned_loss=0.01591, audio_tagging_loss=0.009645, over 3045740.82 frames. ], batch size: 58, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:14:21,664 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260500
2023-11-22 01:14:26,973 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.50 vs. limit=15.0
2023-11-22 01:14:30,857 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.46 vs. limit=12.0
2023-11-22 01:14:54,344 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 7.877e+01 8.568e+01 9.236e+01 1.291e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-22 01:14:55,159 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.78 vs. limit=15.0
2023-11-22 01:15:16,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1736893.3333333333, ans=0.0
2023-11-22 01:15:20,835 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8050, loss[loss=0.08402, simple_loss=0.1095, pruned_loss=0.02005, audio_tagging_loss=0.009208, over 15450.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.0946, pruned_loss=0.0159, audio_tagging_loss=0.00966, over 3041894.29 frames. ], batch size: 56, lr: 3.10e-03, grad_scale: 32.0
2023-11-22 01:15:23,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=13.07 vs. limit=15.0
2023-11-22 01:15:26,904 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260550
2023-11-22 01:15:28,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1736960.0, ans=0.125
2023-11-22 01:15:55,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1737093.3333333333, ans=0.125
2023-11-22 01:16:23,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1737226.6666666667, ans=0.2
2023-11-22 01:16:26,003 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8100, loss[loss=0.06124, simple_loss=0.08292, pruned_loss=0.01167, audio_tagging_loss=0.008108, over 15190.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09436, pruned_loss=0.01583, audio_tagging_loss=0.009667, over 3044719.93 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:16:26,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1737293.3333333333, ans=0.0
2023-11-22 01:16:30,910 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260600
2023-11-22 01:16:54,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1737426.6666666667, ans=0.0
2023-11-22 01:17:04,364 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.436e+01 7.880e+01 8.583e+01 9.266e+01 1.211e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-22 01:17:08,228 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.35 vs. limit=15.0
2023-11-22 01:17:19,489 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:17:27,847 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:17:30,012 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8150, loss[loss=0.06315, simple_loss=0.08803, pruned_loss=0.01215, audio_tagging_loss=0.006978, over 14861.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09535, pruned_loss=0.01599, audio_tagging_loss=0.009502, over 3044753.48 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:17:34,898 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260650
2023-11-22 01:17:39,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1737626.6666666667, ans=0.0
2023-11-22 01:17:43,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1737693.3333333333, ans=0.0
2023-11-22 01:17:44,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1737693.3333333333, ans=0.125
2023-11-22 01:18:00,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1737760.0, ans=0.125
2023-11-22 01:18:11,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1737826.6666666667, ans=0.0
2023-11-22 01:18:15,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1737826.6666666667, ans=0.125
2023-11-22 01:18:33,268 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8200, loss[loss=0.07774, simple_loss=0.09205, pruned_loss=0.02004, audio_tagging_loss=0.01168, over 14895.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09471, pruned_loss=0.01595, audio_tagging_loss=0.009469, over 3048646.75 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:18:36,339 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:18:39,497 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260700
2023-11-22 01:18:42,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1737960.0, ans=0.125
2023-11-22 01:18:45,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1738026.6666666667, ans=10.0
2023-11-22 01:18:54,542 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.33 vs. limit=12.0
2023-11-22 01:19:01,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1738093.3333333333, ans=0.125
2023-11-22 01:19:02,137 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.87 vs. limit=15.0
2023-11-22 01:19:11,979 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0
2023-11-22 01:19:12,436 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.958e+01 8.128e+01 8.787e+01 9.499e+01 1.177e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 01:19:15,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten.whitening_limit, batch_count=1738160.0, ans=15.0
2023-11-22 01:19:23,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1738226.6666666667, ans=0.0
2023-11-22 01:19:36,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1738293.3333333333, ans=0.125
2023-11-22 01:19:38,530 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8250, loss[loss=0.05777, simple_loss=0.06662, pruned_loss=0.01096, audio_tagging_loss=0.01351, over 15018.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09375, pruned_loss=0.01571, audio_tagging_loss=0.00943, over 3046126.91 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:19:43,494 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260750
2023-11-22 01:19:45,328 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.15 vs. limit=15.0
2023-11-22 01:19:53,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1738360.0, ans=0.1
2023-11-22 01:20:09,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1738426.6666666667, ans=0.5
2023-11-22 01:20:31,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1738560.0, ans=0.0
2023-11-22 01:20:41,963 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8300, loss[loss=0.06988, simple_loss=0.07623, pruned_loss=0.017, audio_tagging_loss=0.01477, over 13465.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09392, pruned_loss=0.01562, audio_tagging_loss=0.009528, over 3050457.16 frames. ], batch size: 53, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:20:46,935 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260800
2023-11-22 01:21:09,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1738760.0, ans=0.0
2023-11-22 01:21:14,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1738760.0, ans=0.05
2023-11-22 01:21:21,281 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.003e+01 8.893e+01 9.644e+01 1.813e+02, threshold=1.779e+02, percent-clipped=1.0
2023-11-22 01:21:31,791 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.87 vs. limit=22.5
2023-11-22 01:21:36,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1738893.3333333333, ans=0.1
2023-11-22 01:21:38,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1738893.3333333333, ans=0.0
2023-11-22 01:21:41,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1738893.3333333333, ans=0.125
2023-11-22 01:21:42,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1738893.3333333333, ans=0.0
2023-11-22 01:21:46,178 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8350, loss[loss=0.07242, simple_loss=0.1021, pruned_loss=0.01134, audio_tagging_loss=0.01001, over 15535.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09334, pruned_loss=0.01533, audio_tagging_loss=0.00935, over 3048922.86 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:21:50,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.65 vs. limit=12.0
2023-11-22 01:21:51,707 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260850
2023-11-22 01:21:53,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1738960.0, ans=0.125
2023-11-22 01:22:50,541 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8400, loss[loss=0.06345, simple_loss=0.07655, pruned_loss=0.01215, audio_tagging_loss=0.01303, over 15501.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09313, pruned_loss=0.01526, audio_tagging_loss=0.009427, over 3046968.37 frames. ], batch size: 61, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:22:53,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1739293.3333333333, ans=0.0
2023-11-22 01:22:53,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1739293.3333333333, ans=0.0
2023-11-22 01:22:56,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260900
2023-11-22 01:23:11,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1739360.0, ans=0.07
2023-11-22 01:23:13,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1739360.0, ans=0.2
2023-11-22 01:23:28,558 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.555e+01 7.994e+01 8.596e+01 9.193e+01 1.208e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-22 01:23:50,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1739560.0, ans=0.0
2023-11-22 01:23:54,883 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8450, loss[loss=0.07371, simple_loss=0.09387, pruned_loss=0.01591, audio_tagging_loss=0.01087, over 14956.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09385, pruned_loss=0.01547, audio_tagging_loss=0.009422, over 3042835.43 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:23:56,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1739626.6666666667, ans=0.125
2023-11-22 01:23:59,807 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 260950
2023-11-22 01:24:01,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1739626.6666666667, ans=0.125
2023-11-22 01:24:07,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1739693.3333333333, ans=0.125
2023-11-22 01:24:33,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1739826.6666666667, ans=0.125
2023-11-22 01:24:35,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1739826.6666666667, ans=0.1
2023-11-22 01:24:36,874 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.46 vs. limit=5.0
2023-11-22 01:24:40,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1739826.6666666667, ans=0.1
2023-11-22 01:24:48,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1739893.3333333333, ans=0.125
2023-11-22 01:24:58,320 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8500, loss[loss=0.07593, simple_loss=0.09277, pruned_loss=0.02006, audio_tagging_loss=0.009483, over 14608.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09389, pruned_loss=0.01549, audio_tagging_loss=0.009465, over 3039913.42 frames. ], batch size: 53, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:24:58,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1739960.0, ans=0.2
2023-11-22 01:25:03,332 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261000
2023-11-22 01:25:34,015 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.46 vs. limit=15.0
2023-11-22 01:25:36,958 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.714e+01 8.117e+01 8.646e+01 9.659e+01 1.248e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 01:26:02,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.66 vs. limit=15.0
2023-11-22 01:26:02,757 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8550, loss[loss=0.07069, simple_loss=0.09181, pruned_loss=0.01555, audio_tagging_loss=0.009233, over 15335.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09371, pruned_loss=0.01556, audio_tagging_loss=0.009508, over 3042227.79 frames. ], batch size: 59, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:26:07,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.34 vs. limit=15.0
2023-11-22 01:26:08,528 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261050
2023-11-22 01:26:27,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1740426.6666666667, ans=0.1
2023-11-22 01:26:30,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1740426.6666666667, ans=0.125
2023-11-22 01:26:40,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1740493.3333333333, ans=0.0
2023-11-22 01:27:05,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1740626.6666666667, ans=0.125
2023-11-22 01:27:06,900 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8600, loss[loss=0.05694, simple_loss=0.07565, pruned_loss=0.01154, audio_tagging_loss=0.007573, over 15151.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09489, pruned_loss=0.01561, audio_tagging_loss=0.00936, over 3034551.02 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:27:12,461 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261100
2023-11-22 01:27:17,984 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.61 vs. limit=15.0
2023-11-22 01:27:22,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1740693.3333333333, ans=0.125
2023-11-22 01:27:22,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1740693.3333333333, ans=0.125
2023-11-22 01:27:36,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1740760.0, ans=0.125
2023-11-22 01:27:40,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1740760.0, ans=0.1
2023-11-22 01:27:40,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1740760.0, ans=0.1
2023-11-22 01:27:45,977 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.851e+01 8.115e+01 8.738e+01 9.336e+01 1.223e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 01:27:56,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1740826.6666666667, ans=0.125
2023-11-22 01:28:05,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1740893.3333333333, ans=0.125
2023-11-22 01:28:06,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1740893.3333333333, ans=0.2
2023-11-22 01:28:10,849 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8650, loss[loss=0.0613, simple_loss=0.07828, pruned_loss=0.01194, audio_tagging_loss=0.01022, over 14562.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09488, pruned_loss=0.01556, audio_tagging_loss=0.009526, over 3041116.49 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:28:15,791 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261150
2023-11-22 01:28:26,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1741026.6666666667, ans=0.125
2023-11-22 01:28:33,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1741026.6666666667, ans=0.0
2023-11-22 01:29:05,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1741226.6666666667, ans=0.125
2023-11-22 01:29:15,067 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8700, loss[loss=0.08197, simple_loss=0.1109, pruned_loss=0.01796, audio_tagging_loss=0.008546, over 14982.00 frames. ], tot_loss[loss=0.07333, simple_loss=0.09607, pruned_loss=0.01581, audio_tagging_loss=0.009484, over 3039485.55 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:29:16,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1741293.3333333333, ans=0.2
2023-11-22 01:29:20,594 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261200
2023-11-22 01:29:23,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1741293.3333333333, ans=0.0
2023-11-22 01:29:24,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1741293.3333333333, ans=0.125
2023-11-22 01:29:29,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1741360.0, ans=0.0
2023-11-22 01:29:44,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1741426.6666666667, ans=0.125
2023-11-22 01:29:45,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1741426.6666666667, ans=15.0
2023-11-22 01:29:54,201 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.097e+01 8.323e+01 9.034e+01 9.944e+01 3.984e+02, threshold=1.807e+02, percent-clipped=2.0
2023-11-22 01:30:03,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1741493.3333333333, ans=0.0
2023-11-22 01:30:08,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1741560.0, ans=0.2
2023-11-22 01:30:19,271 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8750, loss[loss=0.04894, simple_loss=0.05475, pruned_loss=0.0103, audio_tagging_loss=0.01126, over 15909.00 frames. ], tot_loss[loss=0.07362, simple_loss=0.09639, pruned_loss=0.01583, audio_tagging_loss=0.009591, over 3043560.57 frames. ], batch size: 62, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:30:24,150 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261250
2023-11-22 01:30:53,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1741760.0, ans=0.125
2023-11-22 01:31:11,406 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.95 vs. limit=22.5
2023-11-22 01:31:23,444 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8800, loss[loss=0.09132, simple_loss=0.1152, pruned_loss=0.02723, audio_tagging_loss=0.006516, over 14814.00 frames. ], tot_loss[loss=0.07396, simple_loss=0.09667, pruned_loss=0.01601, audio_tagging_loss=0.009616, over 3050848.79 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:31:28,354 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261300
2023-11-22 01:32:03,194 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.567e+01 8.285e+01 8.942e+01 9.871e+01 1.431e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 01:32:28,145 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8850, loss[loss=0.05843, simple_loss=0.07597, pruned_loss=0.009985, audio_tagging_loss=0.01046, over 16107.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09598, pruned_loss=0.01603, audio_tagging_loss=0.009648, over 3051247.92 frames. ], batch size: 59, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:32:33,127 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261350
2023-11-22 01:32:37,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1742293.3333333333, ans=0.125
2023-11-22 01:32:42,100 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:32:42,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1742360.0, ans=0.2
2023-11-22 01:32:46,158 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.43 vs. limit=22.5
2023-11-22 01:33:01,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1742426.6666666667, ans=0.125
2023-11-22 01:33:12,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1742493.3333333333, ans=0.125
2023-11-22 01:33:31,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1742626.6666666667, ans=0.125
2023-11-22 01:33:31,957 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8900, loss[loss=0.05166, simple_loss=0.07501, pruned_loss=0.008011, audio_tagging_loss=0.006142, over 15410.00 frames. ], tot_loss[loss=0.07367, simple_loss=0.09636, pruned_loss=0.016, audio_tagging_loss=0.009493, over 3051010.96 frames. ], batch size: 59, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:33:36,916 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261400
2023-11-22 01:33:43,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1742693.3333333333, ans=0.1
2023-11-22 01:33:54,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1742693.3333333333, ans=0.0
2023-11-22 01:34:04,348 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.79 vs. limit=22.5
2023-11-22 01:34:08,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1742760.0, ans=0.2
2023-11-22 01:34:13,144 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.357e+01 8.809e+01 9.586e+01 1.233e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 01:34:13,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1742826.6666666667, ans=0.0
2023-11-22 01:34:23,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1742893.3333333333, ans=0.2
2023-11-22 01:34:30,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1742893.3333333333, ans=0.125
2023-11-22 01:34:33,025 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:34:35,048 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 8950, loss[loss=0.07849, simple_loss=0.09823, pruned_loss=0.02145, audio_tagging_loss=0.007927, over 15429.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.09563, pruned_loss=0.01583, audio_tagging_loss=0.009336, over 3049105.39 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:34:37,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1742960.0, ans=0.0
2023-11-22 01:34:40,104 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261450
2023-11-22 01:35:03,758 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.48 vs. limit=15.0
2023-11-22 01:35:08,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1743093.3333333333, ans=0.1
2023-11-22 01:35:12,173 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.36 vs. limit=12.0
2023-11-22 01:35:18,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1743160.0, ans=0.0
2023-11-22 01:35:20,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1743160.0, ans=0.0
2023-11-22 01:35:22,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1743160.0, ans=0.0
2023-11-22 01:35:26,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1743226.6666666667, ans=0.125
2023-11-22 01:35:36,848 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:35:39,534 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9000, loss[loss=0.05624, simple_loss=0.07199, pruned_loss=0.01028, audio_tagging_loss=0.009967, over 15443.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09481, pruned_loss=0.01564, audio_tagging_loss=0.009269, over 3046558.12 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:35:39,536 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 01:36:20,056 INFO [train_asr.py:1253] (0/4) Epoch 22, validation: loss=0.0605, simple_loss=0.05183, pruned_loss=0.005175, audio_tagging_loss=0.02941, over 4681554.00 frames. 
2023-11-22 01:36:20,057 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 01:36:23,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1743293.3333333333, ans=0.2
2023-11-22 01:36:24,862 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261500
2023-11-22 01:36:27,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1743293.3333333333, ans=0.1
2023-11-22 01:36:44,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=1743426.6666666667, ans=0.05
2023-11-22 01:37:01,166 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.583e+01 8.315e+01 8.919e+01 9.797e+01 1.182e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 01:37:13,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1743560.0, ans=0.125
2023-11-22 01:37:22,982 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9050, loss[loss=0.06713, simple_loss=0.08438, pruned_loss=0.01626, audio_tagging_loss=0.008684, over 14062.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09487, pruned_loss=0.01559, audio_tagging_loss=0.009193, over 3043893.41 frames. ], batch size: 54, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:37:24,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1743626.6666666667, ans=0.2
2023-11-22 01:37:27,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261550
2023-11-22 01:37:41,580 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.60 vs. limit=22.5
2023-11-22 01:38:07,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1743826.6666666667, ans=0.2
2023-11-22 01:38:23,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1743893.3333333333, ans=0.125
2023-11-22 01:38:27,026 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9100, loss[loss=0.06992, simple_loss=0.08665, pruned_loss=0.01598, audio_tagging_loss=0.01062, over 14573.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09469, pruned_loss=0.01549, audio_tagging_loss=0.009224, over 3046928.37 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:38:28,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1743960.0, ans=0.0
2023-11-22 01:38:31,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261600
2023-11-22 01:39:06,383 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.063e+01 8.742e+01 9.387e+01 1.203e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 01:39:13,094 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.53 vs. limit=15.0
2023-11-22 01:39:14,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1744160.0, ans=0.05
2023-11-22 01:39:26,470 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.65 vs. limit=15.0
2023-11-22 01:39:30,546 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9150, loss[loss=0.05877, simple_loss=0.07181, pruned_loss=0.01287, audio_tagging_loss=0.00999, over 13989.00 frames. ], tot_loss[loss=0.07174, simple_loss=0.0944, pruned_loss=0.01539, audio_tagging_loss=0.009138, over 3038817.75 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:39:35,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261650
2023-11-22 01:39:49,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1744360.0, ans=0.125
2023-11-22 01:39:53,094 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.27 vs. limit=12.0
2023-11-22 01:40:07,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1744493.3333333333, ans=0.1
2023-11-22 01:40:12,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1744493.3333333333, ans=0.1
2023-11-22 01:40:25,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1744560.0, ans=0.2
2023-11-22 01:40:33,201 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.23 vs. limit=15.0
2023-11-22 01:40:33,658 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9200, loss[loss=0.06827, simple_loss=0.081, pruned_loss=0.01656, audio_tagging_loss=0.01122, over 14103.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09529, pruned_loss=0.01559, audio_tagging_loss=0.009216, over 3043553.29 frames. ], batch size: 54, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:40:35,220 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:40:38,712 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261700
2023-11-22 01:40:43,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1744626.6666666667, ans=0.125
2023-11-22 01:41:01,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1744760.0, ans=0.125
2023-11-22 01:41:05,735 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.70 vs. limit=15.0
2023-11-22 01:41:14,625 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.066e+01 8.592e+01 9.287e+01 1.258e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 01:41:27,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1744893.3333333333, ans=0.2
2023-11-22 01:41:37,060 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9250, loss[loss=0.07757, simple_loss=0.1026, pruned_loss=0.01669, audio_tagging_loss=0.009569, over 16420.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09421, pruned_loss=0.0153, audio_tagging_loss=0.009283, over 3046904.86 frames. ], batch size: 61, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:41:39,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1744960.0, ans=0.125
2023-11-22 01:41:43,178 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261750
2023-11-22 01:41:47,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1744960.0, ans=0.025
2023-11-22 01:42:09,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1745093.3333333333, ans=0.125
2023-11-22 01:42:35,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1745226.6666666667, ans=0.2
2023-11-22 01:42:41,994 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9300, loss[loss=0.05333, simple_loss=0.06132, pruned_loss=0.01134, audio_tagging_loss=0.01133, over 14975.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09402, pruned_loss=0.01535, audio_tagging_loss=0.009343, over 3042141.92 frames. ], batch size: 57, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:42:46,935 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261800
2023-11-22 01:42:48,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1745293.3333333333, ans=0.95
2023-11-22 01:43:19,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1745493.3333333333, ans=0.0
2023-11-22 01:43:24,446 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.431e+01 7.861e+01 8.461e+01 9.251e+01 1.268e+02, threshold=1.692e+02, percent-clipped=0.0
2023-11-22 01:43:42,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1745560.0, ans=0.05
2023-11-22 01:43:45,726 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9350, loss[loss=0.05716, simple_loss=0.06603, pruned_loss=0.01256, audio_tagging_loss=0.01159, over 16532.00 frames. ], tot_loss[loss=0.07165, simple_loss=0.0937, pruned_loss=0.01535, audio_tagging_loss=0.009443, over 3044213.23 frames. ], batch size: 62, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:43:48,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1745626.6666666667, ans=0.0
2023-11-22 01:43:48,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1745626.6666666667, ans=0.0
2023-11-22 01:43:50,721 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261850
2023-11-22 01:43:51,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.36 vs. limit=10.0
2023-11-22 01:44:18,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1745760.0, ans=0.5
2023-11-22 01:44:49,924 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9400, loss[loss=0.1101, simple_loss=0.1412, pruned_loss=0.03429, audio_tagging_loss=0.005249, over 15349.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09457, pruned_loss=0.01548, audio_tagging_loss=0.009501, over 3051269.94 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:44:55,554 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261900
2023-11-22 01:45:02,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1746026.6666666667, ans=6.0
2023-11-22 01:45:10,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1746026.6666666667, ans=0.2
2023-11-22 01:45:16,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1746093.3333333333, ans=0.1
2023-11-22 01:45:26,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1746093.3333333333, ans=0.1
2023-11-22 01:45:32,787 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.919e+01 8.299e+01 8.911e+01 9.594e+01 2.055e+02, threshold=1.782e+02, percent-clipped=1.0
2023-11-22 01:45:32,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1746160.0, ans=0.2
2023-11-22 01:45:39,911 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:45:44,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1746226.6666666667, ans=0.125
2023-11-22 01:45:49,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1746226.6666666667, ans=0.1
2023-11-22 01:45:56,231 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9450, loss[loss=0.05269, simple_loss=0.07113, pruned_loss=0.006661, audio_tagging_loss=0.01047, over 14960.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.0944, pruned_loss=0.01544, audio_tagging_loss=0.009641, over 3050028.15 frames. ], batch size: 58, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:45:56,303 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:45:56,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1746293.3333333333, ans=0.125
2023-11-22 01:46:01,361 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 261950
2023-11-22 01:46:22,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1746426.6666666667, ans=0.125
2023-11-22 01:46:29,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1746426.6666666667, ans=0.1
2023-11-22 01:46:30,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1746426.6666666667, ans=0.1
2023-11-22 01:46:32,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1746426.6666666667, ans=0.125
2023-11-22 01:46:32,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1746426.6666666667, ans=0.125
2023-11-22 01:46:42,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1746493.3333333333, ans=0.0
2023-11-22 01:46:50,790 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.55 vs. limit=12.0
2023-11-22 01:46:59,809 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9500, loss[loss=0.05522, simple_loss=0.06434, pruned_loss=0.01266, audio_tagging_loss=0.01039, over 14175.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09302, pruned_loss=0.01526, audio_tagging_loss=0.009736, over 3039271.61 frames. ], batch size: 56, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:47:04,663 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262000
2023-11-22 01:47:12,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1746693.3333333333, ans=0.2
2023-11-22 01:47:12,935 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.95 vs. limit=15.0
2023-11-22 01:47:21,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1746693.3333333333, ans=0.2
2023-11-22 01:47:42,661 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.497e+01 8.239e+01 8.709e+01 9.390e+01 1.179e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 01:47:50,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1746893.3333333333, ans=0.125
2023-11-22 01:48:03,599 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9550, loss[loss=0.07025, simple_loss=0.08066, pruned_loss=0.01851, audio_tagging_loss=0.01141, over 14015.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09436, pruned_loss=0.01566, audio_tagging_loss=0.009704, over 3044659.79 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 16.0
2023-11-22 01:48:09,202 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262050
2023-11-22 01:48:21,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1747026.6666666667, ans=0.125
2023-11-22 01:48:29,080 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.50 vs. limit=22.5
2023-11-22 01:48:35,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1747093.3333333333, ans=0.125
2023-11-22 01:48:57,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1747226.6666666667, ans=0.125
2023-11-22 01:49:06,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1747226.6666666667, ans=0.0
2023-11-22 01:49:08,521 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9600, loss[loss=0.08507, simple_loss=0.1042, pruned_loss=0.023, audio_tagging_loss=0.009988, over 13768.00 frames. ], tot_loss[loss=0.07265, simple_loss=0.09451, pruned_loss=0.01559, audio_tagging_loss=0.009803, over 3044824.17 frames. ], batch size: 53, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:49:14,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262100
2023-11-22 01:49:20,068 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.65 vs. limit=22.5
2023-11-22 01:49:22,415 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.49 vs. limit=10.0
2023-11-22 01:49:51,166 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.503e+01 8.033e+01 8.722e+01 9.429e+01 2.117e+02, threshold=1.744e+02, percent-clipped=1.0
2023-11-22 01:49:51,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1747493.3333333333, ans=0.1
2023-11-22 01:49:57,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1747493.3333333333, ans=0.07
2023-11-22 01:50:02,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1747560.0, ans=0.0
2023-11-22 01:50:11,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1747560.0, ans=0.2
2023-11-22 01:50:13,333 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9650, loss[loss=0.06128, simple_loss=0.07628, pruned_loss=0.0147, audio_tagging_loss=0.008436, over 16408.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09428, pruned_loss=0.01573, audio_tagging_loss=0.009748, over 3041057.67 frames. ], batch size: 62, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:50:18,204 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262150
2023-11-22 01:50:23,405 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:50:39,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1747760.0, ans=0.2
2023-11-22 01:51:06,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1747893.3333333333, ans=0.07
2023-11-22 01:51:16,684 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9700, loss[loss=0.065, simple_loss=0.09227, pruned_loss=0.01286, audio_tagging_loss=0.005997, over 14870.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09367, pruned_loss=0.01556, audio_tagging_loss=0.009648, over 3040081.51 frames. ], batch size: 55, lr: 3.09e-03, grad_scale: 32.0
2023-11-22 01:51:21,570 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262200
2023-11-22 01:51:57,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1748160.0, ans=0.125
2023-11-22 01:52:00,821 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.573e+01 8.169e+01 8.750e+01 9.625e+01 1.305e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 01:52:15,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1748226.6666666667, ans=0.0
2023-11-22 01:52:19,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1748226.6666666667, ans=0.0
2023-11-22 01:52:21,634 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9750, loss[loss=0.08445, simple_loss=0.1188, pruned_loss=0.01971, audio_tagging_loss=0.00536, over 15988.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09311, pruned_loss=0.0154, audio_tagging_loss=0.009569, over 3039454.57 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:52:27,330 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262250
2023-11-22 01:52:37,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1748360.0, ans=0.0
2023-11-22 01:52:50,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1748426.6666666667, ans=0.125
2023-11-22 01:52:53,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1748426.6666666667, ans=0.09899494936611666
2023-11-22 01:53:16,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1748560.0, ans=0.2
2023-11-22 01:53:26,050 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9800, loss[loss=0.05844, simple_loss=0.0768, pruned_loss=0.01054, audio_tagging_loss=0.009503, over 15087.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09366, pruned_loss=0.01536, audio_tagging_loss=0.009526, over 3040777.72 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:53:31,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262300
2023-11-22 01:53:59,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1748760.0, ans=0.0
2023-11-22 01:53:59,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1748760.0, ans=0.0
2023-11-22 01:54:10,085 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.594e+01 8.436e+01 9.164e+01 9.957e+01 1.259e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-22 01:54:11,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1748826.6666666667, ans=0.1
2023-11-22 01:54:17,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1748893.3333333333, ans=0.1
2023-11-22 01:54:26,528 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 01:54:28,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1748893.3333333333, ans=0.125
2023-11-22 01:54:30,331 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9850, loss[loss=0.09578, simple_loss=0.1331, pruned_loss=0.02078, audio_tagging_loss=0.008447, over 15119.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09339, pruned_loss=0.01531, audio_tagging_loss=0.009517, over 3046309.45 frames. ], batch size: 53, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:54:31,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1748960.0, ans=0.0
2023-11-22 01:54:35,285 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262350
2023-11-22 01:55:00,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1749093.3333333333, ans=0.125
2023-11-22 01:55:06,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1749093.3333333333, ans=0.0
2023-11-22 01:55:07,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1749160.0, ans=0.0
2023-11-22 01:55:19,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1749160.0, ans=0.0
2023-11-22 01:55:25,720 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 01:55:26,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1749226.6666666667, ans=0.125
2023-11-22 01:55:35,352 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9900, loss[loss=0.06857, simple_loss=0.07974, pruned_loss=0.01521, audio_tagging_loss=0.01349, over 14869.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09366, pruned_loss=0.01537, audio_tagging_loss=0.00953, over 3044430.25 frames. ], batch size: 59, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:55:38,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1749293.3333333333, ans=0.0
2023-11-22 01:55:40,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262400
2023-11-22 01:55:50,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1749360.0, ans=0.125
2023-11-22 01:55:51,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1749360.0, ans=0.1
2023-11-22 01:55:57,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1749360.0, ans=0.2
2023-11-22 01:56:04,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1749426.6666666667, ans=0.125
2023-11-22 01:56:10,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1749426.6666666667, ans=0.125
2023-11-22 01:56:13,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1749493.3333333333, ans=0.125
2023-11-22 01:56:19,003 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.153e+01 9.023e+01 9.668e+01 1.710e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-22 01:56:23,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1749493.3333333333, ans=0.125
2023-11-22 01:56:28,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1749560.0, ans=0.0
2023-11-22 01:56:35,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1749560.0, ans=0.125
2023-11-22 01:56:39,847 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 9950, loss[loss=0.07513, simple_loss=0.09912, pruned_loss=0.01569, audio_tagging_loss=0.009876, over 14899.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.094, pruned_loss=0.01546, audio_tagging_loss=0.009442, over 3054727.31 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:56:42,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1749626.6666666667, ans=0.1
2023-11-22 01:56:43,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1749626.6666666667, ans=0.0
2023-11-22 01:56:44,746 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262450
2023-11-22 01:56:49,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1749626.6666666667, ans=0.0
2023-11-22 01:56:56,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1749693.3333333333, ans=0.0
2023-11-22 01:57:05,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.63 vs. limit=12.0
2023-11-22 01:57:23,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1749826.6666666667, ans=0.0
2023-11-22 01:57:23,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1749826.6666666667, ans=0.1
2023-11-22 01:57:31,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1749893.3333333333, ans=0.125
2023-11-22 01:57:36,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.76 vs. limit=15.0
2023-11-22 01:57:43,625 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10000, loss[loss=0.05999, simple_loss=0.07971, pruned_loss=0.01199, audio_tagging_loss=0.008142, over 15163.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09351, pruned_loss=0.01555, audio_tagging_loss=0.009479, over 3057323.38 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 01:57:48,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262500
2023-11-22 01:58:08,711 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.44 vs. limit=15.0
2023-11-22 01:58:17,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1750093.3333333333, ans=0.0
2023-11-22 01:58:28,336 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.434e+01 8.075e+01 8.690e+01 9.364e+01 1.327e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 01:58:47,718 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10050, loss[loss=0.07601, simple_loss=0.09946, pruned_loss=0.01652, audio_tagging_loss=0.009761, over 14700.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09356, pruned_loss=0.01569, audio_tagging_loss=0.009469, over 3058785.24 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:58:53,387 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262550
2023-11-22 01:59:20,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1750426.6666666667, ans=0.1
2023-11-22 01:59:23,533 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.27 vs. limit=15.0
2023-11-22 01:59:24,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1750426.6666666667, ans=0.125
2023-11-22 01:59:52,858 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10100, loss[loss=0.06467, simple_loss=0.077, pruned_loss=0.01381, audio_tagging_loss=0.01237, over 15913.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09305, pruned_loss=0.01568, audio_tagging_loss=0.009575, over 3062791.84 frames. ], batch size: 63, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 01:59:57,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262600
2023-11-22 02:00:04,884 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.11 vs. limit=6.0
2023-11-22 02:00:13,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1750693.3333333333, ans=0.0
2023-11-22 02:00:24,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1750760.0, ans=0.2
2023-11-22 02:00:29,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1750760.0, ans=0.1
2023-11-22 02:00:33,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1750826.6666666667, ans=0.1
2023-11-22 02:00:38,612 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.583e+01 8.193e+01 8.828e+01 9.579e+01 1.146e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 02:00:47,436 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:00:57,216 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10150, loss[loss=0.07634, simple_loss=0.1076, pruned_loss=0.0152, audio_tagging_loss=0.007321, over 15720.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09322, pruned_loss=0.01552, audio_tagging_loss=0.009625, over 3051128.46 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:01:01,975 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262650
2023-11-22 02:01:03,379 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.72 vs. limit=22.5
2023-11-22 02:01:04,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1750960.0, ans=0.125
2023-11-22 02:01:30,311 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:01:42,799 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:02:01,523 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10200, loss[loss=0.07811, simple_loss=0.09677, pruned_loss=0.02022, audio_tagging_loss=0.009504, over 16243.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09378, pruned_loss=0.01573, audio_tagging_loss=0.009686, over 3055647.78 frames. ], batch size: 62, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:02:07,065 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262700
2023-11-22 02:02:16,882 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.47 vs. limit=15.0
2023-11-22 02:02:27,342 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:02:41,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1751493.3333333333, ans=0.2
2023-11-22 02:02:44,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1751493.3333333333, ans=0.1
2023-11-22 02:02:46,379 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.977e+01 7.988e+01 8.675e+01 9.636e+01 1.224e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 02:03:06,724 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10250, loss[loss=0.06388, simple_loss=0.08591, pruned_loss=0.01269, audio_tagging_loss=0.008242, over 15484.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09358, pruned_loss=0.01566, audio_tagging_loss=0.009652, over 3058601.29 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 8.0
2023-11-22 02:03:11,667 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262750
2023-11-22 02:03:14,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1751626.6666666667, ans=0.0
2023-11-22 02:03:25,467 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.91 vs. limit=15.0
2023-11-22 02:04:09,759 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10300, loss[loss=0.07474, simple_loss=0.1076, pruned_loss=0.01372, audio_tagging_loss=0.007212, over 14909.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.0936, pruned_loss=0.0156, audio_tagging_loss=0.009752, over 3057359.25 frames. ], batch size: 54, lr: 3.08e-03, grad_scale: 8.0
2023-11-22 02:04:14,690 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262800
2023-11-22 02:04:17,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1751960.0, ans=0.0
2023-11-22 02:04:27,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1752026.6666666667, ans=0.125
2023-11-22 02:04:34,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1752026.6666666667, ans=0.0
2023-11-22 02:04:44,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1752093.3333333333, ans=0.1
2023-11-22 02:04:56,373 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.259e+01 8.792e+01 9.429e+01 1.253e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 02:05:07,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1752226.6666666667, ans=0.1
2023-11-22 02:05:11,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1752226.6666666667, ans=0.05
2023-11-22 02:05:14,237 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10350, loss[loss=0.08263, simple_loss=0.09755, pruned_loss=0.02186, audio_tagging_loss=0.012, over 14940.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09427, pruned_loss=0.01567, audio_tagging_loss=0.009757, over 3059741.74 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 8.0
2023-11-22 02:05:17,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1752293.3333333333, ans=0.07
2023-11-22 02:05:20,461 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262850
2023-11-22 02:05:41,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1752426.6666666667, ans=0.125
2023-11-22 02:06:19,258 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10400, loss[loss=0.08232, simple_loss=0.1068, pruned_loss=0.02135, audio_tagging_loss=0.007555, over 15241.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09354, pruned_loss=0.01545, audio_tagging_loss=0.009853, over 3053437.82 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:06:24,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262900
2023-11-22 02:06:42,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1752760.0, ans=0.125
2023-11-22 02:06:59,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1752826.6666666667, ans=0.1
2023-11-22 02:07:02,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1752826.6666666667, ans=0.2
2023-11-22 02:07:04,970 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 8.068e+01 8.726e+01 9.431e+01 1.284e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 02:07:13,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1752893.3333333333, ans=0.125
2023-11-22 02:07:14,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1752893.3333333333, ans=0.125
2023-11-22 02:07:17,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1752893.3333333333, ans=0.125
2023-11-22 02:07:22,693 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10450, loss[loss=0.08819, simple_loss=0.11, pruned_loss=0.02629, audio_tagging_loss=0.006923, over 14941.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09301, pruned_loss=0.01546, audio_tagging_loss=0.009858, over 3049773.33 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:07:27,721 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 262950
2023-11-22 02:07:29,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1752960.0, ans=10.0
2023-11-22 02:07:32,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1752960.0, ans=0.125
2023-11-22 02:07:50,619 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.72 vs. limit=10.0
2023-11-22 02:07:55,426 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.68 vs. limit=15.0
2023-11-22 02:08:03,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1753160.0, ans=0.035
2023-11-22 02:08:25,953 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10500, loss[loss=0.07696, simple_loss=0.1059, pruned_loss=0.0139, audio_tagging_loss=0.0101, over 16403.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09436, pruned_loss=0.01566, audio_tagging_loss=0.009678, over 3052618.16 frames. ], batch size: 60, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:08:31,495 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263000
2023-11-22 02:09:11,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1753493.3333333333, ans=0.0
2023-11-22 02:09:12,585 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.420e+01 8.257e+01 8.868e+01 9.626e+01 1.177e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 02:09:20,987 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=2.516e-03
2023-11-22 02:09:31,577 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10550, loss[loss=0.05695, simple_loss=0.07512, pruned_loss=0.01058, audio_tagging_loss=0.00881, over 14004.00 frames. ], tot_loss[loss=0.072, simple_loss=0.0939, pruned_loss=0.01556, audio_tagging_loss=0.009493, over 3054245.86 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:09:37,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263050
2023-11-22 02:10:01,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1753760.0, ans=0.0
2023-11-22 02:10:06,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1753760.0, ans=0.0
2023-11-22 02:10:21,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1753893.3333333333, ans=0.0
2023-11-22 02:10:21,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1753893.3333333333, ans=0.0
2023-11-22 02:10:35,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1753960.0, ans=0.0
2023-11-22 02:10:35,937 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10600, loss[loss=0.07798, simple_loss=0.1079, pruned_loss=0.01673, audio_tagging_loss=0.007283, over 15153.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09273, pruned_loss=0.01525, audio_tagging_loss=0.00956, over 3045882.75 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:10:40,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263100
2023-11-22 02:10:42,464 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.27 vs. limit=22.5
2023-11-22 02:10:55,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1754026.6666666667, ans=0.5
2023-11-22 02:11:01,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1754093.3333333333, ans=0.125
2023-11-22 02:11:17,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1754160.0, ans=0.09899494936611666
2023-11-22 02:11:21,770 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.379e+01 8.070e+01 8.614e+01 9.337e+01 1.249e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-22 02:11:24,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff3.min_abs, batch_count=1754160.0, ans=0.2
2023-11-22 02:11:30,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1754226.6666666667, ans=0.035
2023-11-22 02:11:30,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1754226.6666666667, ans=0.0
2023-11-22 02:11:37,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1754293.3333333333, ans=0.0
2023-11-22 02:11:38,652 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10650, loss[loss=0.08454, simple_loss=0.1105, pruned_loss=0.01869, audio_tagging_loss=0.01061, over 15526.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09296, pruned_loss=0.01529, audio_tagging_loss=0.009464, over 3042786.79 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:11:43,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263150
2023-11-22 02:11:47,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1754293.3333333333, ans=0.025
2023-11-22 02:12:07,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1754426.6666666667, ans=0.125
2023-11-22 02:12:14,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1754426.6666666667, ans=0.0
2023-11-22 02:12:42,113 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10700, loss[loss=0.06384, simple_loss=0.07678, pruned_loss=0.01644, audio_tagging_loss=0.009018, over 15302.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09299, pruned_loss=0.01526, audio_tagging_loss=0.009422, over 3039884.13 frames. ], batch size: 60, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:12:42,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1754626.6666666667, ans=0.0
2023-11-22 02:12:47,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263200
2023-11-22 02:12:56,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1754693.3333333333, ans=0.2
2023-11-22 02:13:19,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1754826.6666666667, ans=0.125
2023-11-22 02:13:29,043 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.759e+01 7.890e+01 8.587e+01 9.339e+01 1.239e+02, threshold=1.717e+02, percent-clipped=0.0
2023-11-22 02:13:45,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.64 vs. limit=15.0
2023-11-22 02:13:45,393 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.52 vs. limit=22.5
2023-11-22 02:13:47,137 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10750, loss[loss=0.05675, simple_loss=0.0665, pruned_loss=0.01136, audio_tagging_loss=0.01214, over 14638.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09293, pruned_loss=0.01526, audio_tagging_loss=0.009429, over 3045311.90 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:13:52,202 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263250
2023-11-22 02:14:08,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1755026.6666666667, ans=0.125
2023-11-22 02:14:14,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten.whitening_limit, batch_count=1755093.3333333333, ans=15.0
2023-11-22 02:14:49,800 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10800, loss[loss=0.05533, simple_loss=0.06728, pruned_loss=0.01126, audio_tagging_loss=0.01043, over 14500.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09275, pruned_loss=0.01538, audio_tagging_loss=0.00945, over 3051036.44 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:14:54,759 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263300
2023-11-22 02:15:00,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1755293.3333333333, ans=0.1
2023-11-22 02:15:37,523 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.483e+01 8.311e+01 8.848e+01 9.855e+01 1.832e+02, threshold=1.770e+02, percent-clipped=2.0
2023-11-22 02:15:54,662 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10850, loss[loss=0.07125, simple_loss=0.1034, pruned_loss=0.01317, audio_tagging_loss=0.006394, over 16802.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09315, pruned_loss=0.0155, audio_tagging_loss=0.009366, over 3056413.26 frames. ], batch size: 60, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:15:57,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1755626.6666666667, ans=0.0
2023-11-22 02:15:59,584 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263350
2023-11-22 02:15:59,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1755626.6666666667, ans=0.125
2023-11-22 02:16:08,134 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0
2023-11-22 02:16:38,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1755826.6666666667, ans=0.1
2023-11-22 02:16:54,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.87 vs. limit=15.0
2023-11-22 02:16:57,478 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:16:58,693 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10900, loss[loss=0.05312, simple_loss=0.06235, pruned_loss=0.006995, audio_tagging_loss=0.01496, over 14579.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09455, pruned_loss=0.01575, audio_tagging_loss=0.00932, over 3057267.76 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:17:04,294 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263400
2023-11-22 02:17:11,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1756026.6666666667, ans=0.125
2023-11-22 02:17:38,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1756160.0, ans=0.0
2023-11-22 02:17:39,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1756160.0, ans=0.125
2023-11-22 02:17:43,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1756160.0, ans=0.0
2023-11-22 02:17:45,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1756160.0, ans=0.025
2023-11-22 02:17:46,988 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.462e+01 8.172e+01 8.770e+01 9.474e+01 2.328e+02, threshold=1.754e+02, percent-clipped=1.0
2023-11-22 02:18:01,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1756226.6666666667, ans=0.0
2023-11-22 02:18:03,364 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.92 vs. limit=15.0
2023-11-22 02:18:03,849 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 10950, loss[loss=0.05802, simple_loss=0.07181, pruned_loss=0.01058, audio_tagging_loss=0.01153, over 14818.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.0934, pruned_loss=0.01548, audio_tagging_loss=0.009455, over 3052876.59 frames. ], batch size: 55, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:18:08,977 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263450
2023-11-22 02:18:17,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1756360.0, ans=0.125
2023-11-22 02:18:19,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1756360.0, ans=0.2
2023-11-22 02:18:27,699 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.24 vs. limit=15.0
2023-11-22 02:18:37,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1756426.6666666667, ans=0.0
2023-11-22 02:18:40,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1756426.6666666667, ans=0.04949747468305833
2023-11-22 02:18:44,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1756493.3333333333, ans=0.0
2023-11-22 02:19:07,473 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11000, loss[loss=0.04445, simple_loss=0.0501, pruned_loss=0.005564, audio_tagging_loss=0.01384, over 16147.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09382, pruned_loss=0.01555, audio_tagging_loss=0.009441, over 3055337.79 frames. ], batch size: 63, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:19:13,183 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263500
2023-11-22 02:19:21,168 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:19:23,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1756693.3333333333, ans=0.125
2023-11-22 02:19:37,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1756760.0, ans=0.125
2023-11-22 02:19:46,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1756826.6666666667, ans=0.125
2023-11-22 02:19:55,170 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.467e+01 8.140e+01 8.725e+01 9.449e+01 1.514e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 02:19:56,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1756826.6666666667, ans=0.125
2023-11-22 02:20:12,229 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11050, loss[loss=0.07096, simple_loss=0.09435, pruned_loss=0.01591, audio_tagging_loss=0.007879, over 15832.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09346, pruned_loss=0.01537, audio_tagging_loss=0.009508, over 3052256.62 frames. ], batch size: 59, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:20:17,123 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263550
2023-11-22 02:21:10,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1757226.6666666667, ans=0.125
2023-11-22 02:21:16,263 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11100, loss[loss=0.07296, simple_loss=0.1001, pruned_loss=0.01338, audio_tagging_loss=0.009508, over 15529.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09374, pruned_loss=0.01549, audio_tagging_loss=0.009528, over 3054753.10 frames. ], batch size: 58, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:21:20,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1757293.3333333333, ans=0.0
2023-11-22 02:21:21,248 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263600
2023-11-22 02:21:24,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1757293.3333333333, ans=0.1
2023-11-22 02:21:26,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1757293.3333333333, ans=0.0
2023-11-22 02:21:32,287 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:21:39,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1757360.0, ans=0.0
2023-11-22 02:21:41,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1757426.6666666667, ans=0.0
2023-11-22 02:21:42,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1757426.6666666667, ans=0.1
2023-11-22 02:21:51,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1757426.6666666667, ans=0.0
2023-11-22 02:22:03,459 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.908e+01 8.113e+01 8.819e+01 9.747e+01 1.374e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 02:22:07,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1757560.0, ans=0.1
2023-11-22 02:22:10,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1757560.0, ans=0.125
2023-11-22 02:22:20,505 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11150, loss[loss=0.08691, simple_loss=0.1149, pruned_loss=0.01934, audio_tagging_loss=0.0101, over 14589.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09378, pruned_loss=0.01543, audio_tagging_loss=0.009718, over 3048163.56 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 16.0
2023-11-22 02:22:22,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.19 vs. limit=15.0
2023-11-22 02:22:25,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263650
2023-11-22 02:22:28,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1757626.6666666667, ans=0.1
2023-11-22 02:22:36,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1757693.3333333333, ans=0.0
2023-11-22 02:22:59,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.88 vs. limit=15.0
2023-11-22 02:23:16,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0
2023-11-22 02:23:25,254 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11200, loss[loss=0.07625, simple_loss=0.1062, pruned_loss=0.0157, audio_tagging_loss=0.007426, over 15166.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.0941, pruned_loss=0.0153, audio_tagging_loss=0.009721, over 3047013.64 frames. ], batch size: 56, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:23:27,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1757960.0, ans=0.125
2023-11-22 02:23:27,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1757960.0, ans=0.125
2023-11-22 02:23:30,072 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263700
2023-11-22 02:23:52,475 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:24:11,343 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 8.183e+01 8.908e+01 9.614e+01 1.304e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 02:24:11,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1758160.0, ans=0.2
2023-11-22 02:24:14,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1758226.6666666667, ans=0.0
2023-11-22 02:24:27,701 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11250, loss[loss=0.05797, simple_loss=0.07129, pruned_loss=0.01304, audio_tagging_loss=0.009286, over 15503.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09361, pruned_loss=0.0152, audio_tagging_loss=0.009692, over 3047632.16 frames. ], batch size: 60, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:24:33,400 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263750
2023-11-22 02:24:40,066 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.42 vs. limit=6.0
2023-11-22 02:24:47,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1758360.0, ans=0.125
2023-11-22 02:25:15,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1758493.3333333333, ans=0.0
2023-11-22 02:25:31,823 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11300, loss[loss=0.0755, simple_loss=0.1088, pruned_loss=0.01537, audio_tagging_loss=0.005738, over 15644.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09453, pruned_loss=0.01544, audio_tagging_loss=0.009475, over 3047778.67 frames. ], batch size: 59, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:25:36,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263800
2023-11-22 02:25:44,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1758693.3333333333, ans=0.5
2023-11-22 02:25:54,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.83 vs. limit=6.0
2023-11-22 02:26:00,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1758760.0, ans=0.05
2023-11-22 02:26:12,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1758826.6666666667, ans=0.2
2023-11-22 02:26:19,792 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.464e+01 8.064e+01 8.666e+01 9.719e+01 1.287e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 02:26:33,343 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.65 vs. limit=15.0
2023-11-22 02:26:36,265 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11350, loss[loss=0.0754, simple_loss=0.09925, pruned_loss=0.01492, audio_tagging_loss=0.01086, over 16579.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09454, pruned_loss=0.01555, audio_tagging_loss=0.009466, over 3048978.03 frames. ], batch size: 60, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:26:39,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1758960.0, ans=0.07
2023-11-22 02:26:41,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263850
2023-11-22 02:26:44,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1758960.0, ans=0.125
2023-11-22 02:26:50,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1759026.6666666667, ans=0.1
2023-11-22 02:27:17,529 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.41 vs. limit=15.0
2023-11-22 02:27:20,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1759160.0, ans=0.125
2023-11-22 02:27:30,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1759226.6666666667, ans=0.0
2023-11-22 02:27:31,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1759226.6666666667, ans=0.125
2023-11-22 02:27:39,811 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11400, loss[loss=0.08353, simple_loss=0.1165, pruned_loss=0.01775, audio_tagging_loss=0.007513, over 15401.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09452, pruned_loss=0.01562, audio_tagging_loss=0.009367, over 3038672.70 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:27:44,849 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263900
2023-11-22 02:27:46,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1759293.3333333333, ans=0.0
2023-11-22 02:28:11,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1759426.6666666667, ans=0.125
2023-11-22 02:28:18,312 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.65 vs. limit=12.0
2023-11-22 02:28:26,717 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.50 vs. limit=10.0
2023-11-22 02:28:27,241 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.669e+01 8.049e+01 8.601e+01 9.494e+01 1.206e+02, threshold=1.720e+02, percent-clipped=0.0
2023-11-22 02:28:43,100 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11450, loss[loss=0.09146, simple_loss=0.1268, pruned_loss=0.02149, audio_tagging_loss=0.006594, over 16440.00 frames. ], tot_loss[loss=0.07199, simple_loss=0.09412, pruned_loss=0.01552, audio_tagging_loss=0.009407, over 3041702.48 frames. ], batch size: 57, lr: 3.08e-03, grad_scale: 32.0
2023-11-22 02:28:48,603 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 263950
2023-11-22 02:29:13,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1759760.0, ans=0.125
2023-11-22 02:29:34,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1759893.3333333333, ans=0.125
2023-11-22 02:29:40,505 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.93 vs. limit=10.0
2023-11-22 02:29:41,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1759893.3333333333, ans=0.125
2023-11-22 02:29:47,461 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11500, loss[loss=0.0685, simple_loss=0.09367, pruned_loss=0.01408, audio_tagging_loss=0.007585, over 14952.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.094, pruned_loss=0.01561, audio_tagging_loss=0.009468, over 3036894.52 frames. ], batch size: 54, lr: 3.07e-03, grad_scale: 32.0
2023-11-22 02:29:53,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264000
2023-11-22 02:29:54,598 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-264000.pt
2023-11-22 02:30:09,161 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.15 vs. limit=6.0
2023-11-22 02:30:09,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1760026.6666666667, ans=0.125
2023-11-22 02:30:29,749 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0
2023-11-22 02:30:38,030 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.114e+01 8.860e+01 9.560e+01 1.133e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 02:30:54,560 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11550, loss[loss=0.06787, simple_loss=0.08855, pruned_loss=0.01566, audio_tagging_loss=0.007937, over 14872.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.0945, pruned_loss=0.01574, audio_tagging_loss=0.009437, over 3040304.47 frames. ], batch size: 57, lr: 3.07e-03, grad_scale: 32.0
2023-11-22 02:30:59,463 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264050
2023-11-22 02:31:37,434 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 02:31:37,943 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.53 vs. limit=22.5
2023-11-22 02:31:38,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1760493.3333333333, ans=0.025
2023-11-22 02:31:48,739 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.89 vs. limit=12.0
2023-11-22 02:31:57,972 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11600, loss[loss=0.07421, simple_loss=0.1036, pruned_loss=0.01463, audio_tagging_loss=0.007793, over 14285.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09496, pruned_loss=0.0156, audio_tagging_loss=0.009355, over 3042959.13 frames. ], batch size: 55, lr: 3.07e-03, grad_scale: 32.0
2023-11-22 02:32:00,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1760626.6666666667, ans=0.125
2023-11-22 02:32:03,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264100
2023-11-22 02:32:26,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1760760.0, ans=0.0
2023-11-22 02:32:29,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1760760.0, ans=0.0
2023-11-22 02:32:39,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1760826.6666666667, ans=10.0
2023-11-22 02:32:39,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1760826.6666666667, ans=0.125
2023-11-22 02:32:47,755 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.841e+01 8.153e+01 8.705e+01 9.370e+01 1.172e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 02:32:53,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1760893.3333333333, ans=0.125
2023-11-22 02:33:02,416 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11650, loss[loss=0.08651, simple_loss=0.1246, pruned_loss=0.01765, audio_tagging_loss=0.006562, over 14553.00 frames. ], tot_loss[loss=0.07298, simple_loss=0.09591, pruned_loss=0.01569, audio_tagging_loss=0.009335, over 3045130.57 frames. ], batch size: 56, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:33:07,463 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264150
2023-11-22 02:33:28,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1761093.3333333333, ans=0.05
2023-11-22 02:33:49,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1761160.0, ans=0.125
2023-11-22 02:34:06,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1761293.3333333333, ans=0.0
2023-11-22 02:34:07,585 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11700, loss[loss=0.06242, simple_loss=0.07802, pruned_loss=0.01434, audio_tagging_loss=0.009067, over 15203.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.0949, pruned_loss=0.0156, audio_tagging_loss=0.009405, over 3043458.69 frames. ], batch size: 57, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:34:12,569 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264200
2023-11-22 02:34:12,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1761293.3333333333, ans=0.125
2023-11-22 02:34:12,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1761293.3333333333, ans=0.1
2023-11-22 02:34:18,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1761293.3333333333, ans=0.0
2023-11-22 02:34:24,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.60 vs. limit=6.0
2023-11-22 02:34:38,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1761426.6666666667, ans=0.125
2023-11-22 02:34:44,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1761426.6666666667, ans=0.125
2023-11-22 02:34:57,767 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.071e+01 8.694e+01 9.444e+01 1.226e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 02:35:07,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1761560.0, ans=0.125
2023-11-22 02:35:11,133 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11750, loss[loss=0.08737, simple_loss=0.1174, pruned_loss=0.02004, audio_tagging_loss=0.008644, over 16050.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09397, pruned_loss=0.01555, audio_tagging_loss=0.00954, over 3045518.55 frames. ], batch size: 58, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:35:14,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1761626.6666666667, ans=0.125
2023-11-22 02:35:15,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1761626.6666666667, ans=0.2
2023-11-22 02:35:16,074 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264250
2023-11-22 02:35:17,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1761626.6666666667, ans=0.1
2023-11-22 02:35:36,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1761760.0, ans=0.0
2023-11-22 02:35:46,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1761760.0, ans=0.125
2023-11-22 02:35:52,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1761826.6666666667, ans=0.125
2023-11-22 02:35:52,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1761826.6666666667, ans=0.0
2023-11-22 02:35:58,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1761826.6666666667, ans=0.2
2023-11-22 02:36:15,728 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11800, loss[loss=0.08032, simple_loss=0.1048, pruned_loss=0.0189, audio_tagging_loss=0.008993, over 15028.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09408, pruned_loss=0.01564, audio_tagging_loss=0.009538, over 3047759.04 frames. ], batch size: 56, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:36:21,271 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264300
2023-11-22 02:36:28,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1762026.6666666667, ans=0.1
2023-11-22 02:36:31,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1762026.6666666667, ans=0.0
2023-11-22 02:36:37,490 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.83 vs. limit=10.0
2023-11-22 02:36:41,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1762093.3333333333, ans=0.1
2023-11-22 02:37:06,208 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 8.285e+01 8.657e+01 9.300e+01 1.154e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-22 02:37:16,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1762226.6666666667, ans=0.1
2023-11-22 02:37:21,178 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11850, loss[loss=0.04402, simple_loss=0.0525, pruned_loss=0.007064, audio_tagging_loss=0.01071, over 15429.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09587, pruned_loss=0.01596, audio_tagging_loss=0.009472, over 3049754.27 frames. ], batch size: 60, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:37:26,066 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264350
2023-11-22 02:37:34,014 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.88 vs. limit=22.5
2023-11-22 02:37:40,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1762360.0, ans=0.05
2023-11-22 02:37:51,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1762426.6666666667, ans=0.1
2023-11-22 02:37:52,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1762426.6666666667, ans=0.0
2023-11-22 02:38:04,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.43 vs. limit=22.5
2023-11-22 02:38:23,851 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11900, loss[loss=0.09862, simple_loss=0.139, pruned_loss=0.02218, audio_tagging_loss=0.006952, over 15466.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09463, pruned_loss=0.01575, audio_tagging_loss=0.009625, over 3049182.48 frames. ], batch size: 54, lr: 3.07e-03, grad_scale: 8.0
2023-11-22 02:38:28,936 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264400
2023-11-22 02:38:42,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1762693.3333333333, ans=0.125
2023-11-22 02:38:47,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1762693.3333333333, ans=0.2
2023-11-22 02:38:50,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.28 vs. limit=6.0
2023-11-22 02:39:00,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1762760.0, ans=0.0
2023-11-22 02:39:04,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1762826.6666666667, ans=0.2
2023-11-22 02:39:06,819 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 02:39:08,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1762826.6666666667, ans=0.125
2023-11-22 02:39:13,247 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.21 vs. limit=15.0
2023-11-22 02:39:14,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1762893.3333333333, ans=0.125
2023-11-22 02:39:15,104 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.906e+01 8.188e+01 8.826e+01 9.362e+01 1.336e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 02:39:28,024 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 11950, loss[loss=0.07461, simple_loss=0.09161, pruned_loss=0.01749, audio_tagging_loss=0.01132, over 16457.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09448, pruned_loss=0.01567, audio_tagging_loss=0.009673, over 3060899.98 frames. ], batch size: 65, lr: 3.07e-03, grad_scale: 8.0
2023-11-22 02:39:33,675 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264450
2023-11-22 02:39:51,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1763026.6666666667, ans=0.1
2023-11-22 02:39:57,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1763093.3333333333, ans=0.125
2023-11-22 02:40:11,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=1763160.0, ans=0.05
2023-11-22 02:40:14,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1763160.0, ans=0.1
2023-11-22 02:40:15,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1763160.0, ans=0.125
2023-11-22 02:40:23,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1763226.6666666667, ans=0.125
2023-11-22 02:40:24,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1763226.6666666667, ans=0.125
2023-11-22 02:40:31,119 INFO [train_asr.py:1221] (0/4) Epoch 22, batch 12000, loss[loss=0.06386, simple_loss=0.07842, pruned_loss=0.01207, audio_tagging_loss=0.01258, over 15806.00 frames. ], tot_loss[loss=0.07353, simple_loss=0.09591, pruned_loss=0.01587, audio_tagging_loss=0.009704, over 3059981.17 frames. ], batch size: 59, lr: 3.07e-03, grad_scale: 16.0
2023-11-22 02:40:31,122 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 02:41:05,112 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([4.4823, 3.8268, 4.3142, 3.5201], device='cuda:0')
2023-11-22 02:41:14,449 INFO [train_asr.py:1253] (0/4) Epoch 22, validation: loss=0.05922, simple_loss=0.05191, pruned_loss=0.005254, audio_tagging_loss=0.02801, over 4681554.00 frames. 
2023-11-22 02:41:14,450 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 02:41:19,277 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264500
2023-11-22 02:41:25,723 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.77 vs. limit=12.0
2023-11-22 02:41:46,788 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-22.pt
2023-11-22 02:42:19,292 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.67 vs. limit=15.0
2023-11-22 02:42:19,937 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 0, loss[loss=0.08302, simple_loss=0.08618, pruned_loss=0.01524, audio_tagging_loss=0.02469, over 15772.00 frames. ], tot_loss[loss=0.08302, simple_loss=0.08618, pruned_loss=0.01524, audio_tagging_loss=0.02469, over 15772.00 frames. ], batch size: 62, lr: 3.00e-03, grad_scale: 32.0
2023-11-22 02:42:19,943 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 02:42:55,354 INFO [train_asr.py:1253] (0/4) Epoch 23, validation: loss=0.05874, simple_loss=0.05183, pruned_loss=0.005194, audio_tagging_loss=0.02763, over 4681554.00 frames. 
2023-11-22 02:42:55,355 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 02:43:01,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1763473.3333333333, ans=0.125
2023-11-22 02:43:13,348 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.157e+01 8.370e+01 9.150e+01 9.856e+01 1.621e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-22 02:43:30,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264550
2023-11-22 02:43:36,944 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.88 vs. limit=15.0
2023-11-22 02:43:37,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.49 vs. limit=22.5
2023-11-22 02:43:59,711 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 50, loss[loss=0.1006, simple_loss=0.1223, pruned_loss=0.02396, audio_tagging_loss=0.01548, over 15313.00 frames. ], tot_loss[loss=0.08265, simple_loss=0.0979, pruned_loss=0.01592, audio_tagging_loss=0.01778, over 685201.93 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:44:12,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1763873.3333333333, ans=0.1
2023-11-22 02:44:35,223 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264600
2023-11-22 02:44:38,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1764006.6666666667, ans=0.125
2023-11-22 02:45:01,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1764073.3333333333, ans=0.125
2023-11-22 02:45:02,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1764073.3333333333, ans=0.125
2023-11-22 02:45:05,753 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 100, loss[loss=0.08331, simple_loss=0.1058, pruned_loss=0.01708, audio_tagging_loss=0.01332, over 15078.00 frames. ], tot_loss[loss=0.08089, simple_loss=0.09618, pruned_loss=0.0155, audio_tagging_loss=0.0173, over 1209787.80 frames. ], batch size: 55, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:45:23,744 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.380e+01 9.017e+01 9.520e+01 1.030e+02 1.259e+02, threshold=1.904e+02, percent-clipped=0.0
2023-11-22 02:45:40,552 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264650
2023-11-22 02:45:50,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1764340.0, ans=0.0
2023-11-22 02:46:03,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1764406.6666666667, ans=10.0
2023-11-22 02:46:11,383 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 150, loss[loss=0.06988, simple_loss=0.09256, pruned_loss=0.01414, audio_tagging_loss=0.009468, over 15679.00 frames. ], tot_loss[loss=0.079, simple_loss=0.09664, pruned_loss=0.0155, audio_tagging_loss=0.01518, over 1618717.79 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:46:40,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1764606.6666666667, ans=0.0
2023-11-22 02:46:40,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1764606.6666666667, ans=0.125
2023-11-22 02:46:46,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264700
2023-11-22 02:46:53,855 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.63 vs. limit=10.0
2023-11-22 02:47:07,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1764740.0, ans=0.0
2023-11-22 02:47:16,071 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 200, loss[loss=0.07998, simple_loss=0.09558, pruned_loss=0.02113, audio_tagging_loss=0.01106, over 14842.00 frames. ], tot_loss[loss=0.07774, simple_loss=0.09729, pruned_loss=0.01566, audio_tagging_loss=0.01343, over 1934609.94 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:47:33,568 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.877e+01 8.445e+01 8.941e+01 9.618e+01 1.553e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 02:47:38,585 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.29 vs. limit=12.0
2023-11-22 02:47:44,624 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.20 vs. limit=15.0
2023-11-22 02:47:50,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264750
2023-11-22 02:47:54,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1765006.6666666667, ans=0.125
2023-11-22 02:48:12,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1765073.3333333333, ans=0.0
2023-11-22 02:48:20,500 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 250, loss[loss=0.08053, simple_loss=0.09761, pruned_loss=0.01934, audio_tagging_loss=0.01239, over 14413.00 frames. ], tot_loss[loss=0.07622, simple_loss=0.09679, pruned_loss=0.01561, audio_tagging_loss=0.01221, over 2182807.64 frames. ], batch size: 54, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:48:22,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1765140.0, ans=0.0
2023-11-22 02:48:23,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1765140.0, ans=0.0
2023-11-22 02:48:40,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1765206.6666666667, ans=0.0
2023-11-22 02:48:44,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1765206.6666666667, ans=0.1
2023-11-22 02:48:47,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1765273.3333333333, ans=0.0
2023-11-22 02:48:54,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1765273.3333333333, ans=0.1
2023-11-22 02:48:55,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264800
2023-11-22 02:49:09,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1765340.0, ans=0.125
2023-11-22 02:49:26,628 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 300, loss[loss=0.07454, simple_loss=0.09848, pruned_loss=0.01386, audio_tagging_loss=0.01144, over 15189.00 frames. ], tot_loss[loss=0.07586, simple_loss=0.09714, pruned_loss=0.01591, audio_tagging_loss=0.01138, over 2373920.22 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:49:27,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.80 vs. limit=15.0
2023-11-22 02:49:44,446 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.292e+01 9.030e+01 9.528e+01 1.689e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-22 02:49:46,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1765540.0, ans=0.1
2023-11-22 02:49:48,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1765540.0, ans=0.125
2023-11-22 02:50:00,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1765606.6666666667, ans=0.125
2023-11-22 02:50:01,675 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264850
2023-11-22 02:50:01,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1765606.6666666667, ans=0.125
2023-11-22 02:50:11,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1765673.3333333333, ans=0.125
2023-11-22 02:50:12,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1765673.3333333333, ans=0.125
2023-11-22 02:50:27,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1765740.0, ans=0.125
2023-11-22 02:50:31,831 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 350, loss[loss=0.07187, simple_loss=0.09765, pruned_loss=0.01452, audio_tagging_loss=0.008525, over 14128.00 frames. ], tot_loss[loss=0.07546, simple_loss=0.09708, pruned_loss=0.01601, audio_tagging_loss=0.01092, over 2527043.45 frames. ], batch size: 54, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:50:37,363 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.16 vs. limit=10.0
2023-11-22 02:50:53,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1765873.3333333333, ans=0.1
2023-11-22 02:50:57,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1765940.0, ans=0.1
2023-11-22 02:50:59,613 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.63 vs. limit=15.0
2023-11-22 02:51:07,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264900
2023-11-22 02:51:23,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1766073.3333333333, ans=0.2
2023-11-22 02:51:36,548 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 400, loss[loss=0.06767, simple_loss=0.08533, pruned_loss=0.01403, audio_tagging_loss=0.01098, over 15433.00 frames. ], tot_loss[loss=0.07465, simple_loss=0.09667, pruned_loss=0.01585, audio_tagging_loss=0.01047, over 2646998.15 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 32.0
2023-11-22 02:51:39,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1766140.0, ans=0.0
2023-11-22 02:51:55,101 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.193e+01 8.089e+01 8.847e+01 9.636e+01 1.262e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 02:52:08,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1766273.3333333333, ans=0.0
2023-11-22 02:52:11,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 264950
2023-11-22 02:52:41,686 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 450, loss[loss=0.05901, simple_loss=0.08133, pruned_loss=0.01182, audio_tagging_loss=0.006521, over 14722.00 frames. ], tot_loss[loss=0.07388, simple_loss=0.09602, pruned_loss=0.01572, audio_tagging_loss=0.01016, over 2731790.80 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:52:44,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1766473.3333333333, ans=0.125
2023-11-22 02:52:46,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1766473.3333333333, ans=0.04949747468305833
2023-11-22 02:53:11,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1766606.6666666667, ans=0.125
2023-11-22 02:53:11,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1766606.6666666667, ans=0.0
2023-11-22 02:53:17,301 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265000
2023-11-22 02:53:19,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1766606.6666666667, ans=0.125
2023-11-22 02:53:20,724 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.20 vs. limit=15.0
2023-11-22 02:53:35,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.62 vs. limit=15.0
2023-11-22 02:53:46,792 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 500, loss[loss=0.089, simple_loss=0.1148, pruned_loss=0.02354, audio_tagging_loss=0.008079, over 16082.00 frames. ], tot_loss[loss=0.07373, simple_loss=0.09611, pruned_loss=0.01576, audio_tagging_loss=0.009914, over 2801618.17 frames. ], batch size: 60, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:54:06,404 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.266e+01 9.127e+01 9.856e+01 1.336e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-22 02:54:08,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1766873.3333333333, ans=0.1
2023-11-22 02:54:21,497 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265050
2023-11-22 02:54:22,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1766940.0, ans=0.0
2023-11-22 02:54:28,878 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.76 vs. limit=22.5
2023-11-22 02:54:38,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1767073.3333333333, ans=0.2
2023-11-22 02:54:51,568 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 550, loss[loss=0.07045, simple_loss=0.08449, pruned_loss=0.01547, audio_tagging_loss=0.01273, over 15852.00 frames. ], tot_loss[loss=0.0736, simple_loss=0.09596, pruned_loss=0.01583, audio_tagging_loss=0.009791, over 2856679.76 frames. ], batch size: 61, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:54:55,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1767140.0, ans=0.0
2023-11-22 02:55:00,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1767140.0, ans=0.0
2023-11-22 02:55:00,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1767140.0, ans=0.0
2023-11-22 02:55:03,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1767206.6666666667, ans=0.2
2023-11-22 02:55:03,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1767206.6666666667, ans=0.125
2023-11-22 02:55:04,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1767206.6666666667, ans=0.1
2023-11-22 02:55:26,304 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265100
2023-11-22 02:55:34,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1767340.0, ans=0.1
2023-11-22 02:55:36,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1767340.0, ans=0.1
2023-11-22 02:55:41,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1767340.0, ans=0.125
2023-11-22 02:55:47,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1767406.6666666667, ans=0.1
2023-11-22 02:55:55,627 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 600, loss[loss=0.07074, simple_loss=0.09506, pruned_loss=0.01351, audio_tagging_loss=0.009694, over 14901.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09466, pruned_loss=0.01551, audio_tagging_loss=0.009679, over 2896891.01 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:56:15,835 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.111e+01 8.755e+01 9.380e+01 1.139e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 02:56:22,651 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.94 vs. limit=22.5
2023-11-22 02:56:24,917 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.31 vs. limit=15.0
2023-11-22 02:56:31,561 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265150
2023-11-22 02:57:01,423 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 650, loss[loss=0.07521, simple_loss=0.1049, pruned_loss=0.01487, audio_tagging_loss=0.007871, over 17318.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09447, pruned_loss=0.01544, audio_tagging_loss=0.009708, over 2926879.60 frames. ], batch size: 63, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:57:03,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.40 vs. limit=10.0
2023-11-22 02:57:15,149 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.33 vs. limit=15.0
2023-11-22 02:57:35,972 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265200
2023-11-22 02:57:37,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1767940.0, ans=0.125
2023-11-22 02:57:53,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1768073.3333333333, ans=0.1
2023-11-22 02:58:06,616 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 700, loss[loss=0.07209, simple_loss=0.09769, pruned_loss=0.01614, audio_tagging_loss=0.007109, over 15041.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09467, pruned_loss=0.01559, audio_tagging_loss=0.009635, over 2952853.10 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 02:58:26,070 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.743e+01 8.011e+01 8.675e+01 9.413e+01 1.282e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 02:58:26,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1768206.6666666667, ans=0.125
2023-11-22 02:58:26,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1768206.6666666667, ans=0.125
2023-11-22 02:58:42,171 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265250
2023-11-22 02:58:49,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1768340.0, ans=0.1
2023-11-22 02:59:11,960 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 750, loss[loss=0.07662, simple_loss=0.102, pruned_loss=0.01401, audio_tagging_loss=0.01162, over 16077.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09445, pruned_loss=0.0154, audio_tagging_loss=0.009681, over 2980751.24 frames. ], batch size: 61, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 02:59:14,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1768473.3333333333, ans=0.0
2023-11-22 02:59:47,586 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265300
2023-11-22 02:59:49,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1768606.6666666667, ans=0.0
2023-11-22 02:59:51,509 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=10.19 vs. limit=10.0
2023-11-22 03:00:07,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1768740.0, ans=0.0
2023-11-22 03:00:07,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1768740.0, ans=0.07
2023-11-22 03:00:17,348 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 800, loss[loss=0.05922, simple_loss=0.07542, pruned_loss=0.01154, audio_tagging_loss=0.009976, over 15913.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09592, pruned_loss=0.01573, audio_tagging_loss=0.009613, over 2997217.31 frames. ], batch size: 61, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:00:18,016 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.14 vs. limit=10.0
2023-11-22 03:00:32,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1768873.3333333333, ans=0.125
2023-11-22 03:00:39,219 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.565e+01 9.192e+01 1.025e+02 1.525e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-22 03:00:51,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1768940.0, ans=0.125
2023-11-22 03:00:53,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265350
2023-11-22 03:00:53,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1768940.0, ans=0.125
2023-11-22 03:00:58,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1769006.6666666667, ans=0.0
2023-11-22 03:01:01,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1769006.6666666667, ans=0.125
2023-11-22 03:01:09,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1769073.3333333333, ans=0.125
2023-11-22 03:01:11,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1769073.3333333333, ans=0.125
2023-11-22 03:01:17,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1769073.3333333333, ans=0.125
2023-11-22 03:01:23,374 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 850, loss[loss=0.07391, simple_loss=0.1003, pruned_loss=0.01498, audio_tagging_loss=0.008785, over 15161.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09571, pruned_loss=0.0157, audio_tagging_loss=0.009672, over 3009066.90 frames. ], batch size: 56, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:01:46,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1769206.6666666667, ans=0.125
2023-11-22 03:01:58,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265400
2023-11-22 03:02:09,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=1769340.0, ans=15.0
2023-11-22 03:02:18,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1769406.6666666667, ans=0.035
2023-11-22 03:02:28,374 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 900, loss[loss=0.07669, simple_loss=0.1015, pruned_loss=0.01645, audio_tagging_loss=0.009482, over 15472.00 frames. ], tot_loss[loss=0.07354, simple_loss=0.09605, pruned_loss=0.01576, audio_tagging_loss=0.00975, over 3017803.76 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:02:32,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1769473.3333333333, ans=0.1
2023-11-22 03:02:50,266 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.531e+01 8.073e+01 8.740e+01 9.361e+01 1.142e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 03:02:51,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1769540.0, ans=0.125
2023-11-22 03:02:55,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1769606.6666666667, ans=0.2
2023-11-22 03:03:04,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265450
2023-11-22 03:03:33,881 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 950, loss[loss=0.06803, simple_loss=0.09011, pruned_loss=0.01479, audio_tagging_loss=0.008183, over 14916.00 frames. ], tot_loss[loss=0.07304, simple_loss=0.09584, pruned_loss=0.01551, audio_tagging_loss=0.009611, over 3024994.75 frames. ], batch size: 55, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:04:05,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1769940.0, ans=0.2
2023-11-22 03:04:07,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1769940.0, ans=0.125
2023-11-22 03:04:08,681 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265500
2023-11-22 03:04:14,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.07 vs. limit=12.0
2023-11-22 03:04:25,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1770073.3333333333, ans=0.05
2023-11-22 03:04:28,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1770073.3333333333, ans=0.125
2023-11-22 03:04:39,187 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1000, loss[loss=0.08199, simple_loss=0.1065, pruned_loss=0.0212, audio_tagging_loss=0.007536, over 14543.00 frames. ], tot_loss[loss=0.07262, simple_loss=0.09523, pruned_loss=0.01557, audio_tagging_loss=0.009442, over 3032230.47 frames. ], batch size: 55, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:04:58,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1770206.6666666667, ans=0.125
2023-11-22 03:05:00,671 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.104e+01 8.710e+01 9.492e+01 1.362e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 03:05:05,591 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:05:07,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1770273.3333333333, ans=0.125
2023-11-22 03:05:13,765 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265550
2023-11-22 03:05:24,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1770340.0, ans=0.125
2023-11-22 03:05:43,705 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1050, loss[loss=0.06194, simple_loss=0.0833, pruned_loss=0.01215, audio_tagging_loss=0.008129, over 14348.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09513, pruned_loss=0.01545, audio_tagging_loss=0.009411, over 3038588.06 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:06:08,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1770606.6666666667, ans=0.125
2023-11-22 03:06:19,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265600
2023-11-22 03:06:33,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1770673.3333333333, ans=0.07
2023-11-22 03:06:38,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1770740.0, ans=0.1
2023-11-22 03:06:41,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1770740.0, ans=0.1
2023-11-22 03:06:41,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1770740.0, ans=0.0
2023-11-22 03:06:49,315 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1100, loss[loss=0.07217, simple_loss=0.0958, pruned_loss=0.01341, audio_tagging_loss=0.01086, over 15384.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09498, pruned_loss=0.01547, audio_tagging_loss=0.009337, over 3047253.68 frames. ], batch size: 58, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:06:49,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1770806.6666666667, ans=0.125
2023-11-22 03:06:51,773 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:07:11,626 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.139e+01 7.983e+01 8.697e+01 9.396e+01 1.258e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 03:07:18,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1770940.0, ans=0.125
2023-11-22 03:07:20,474 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.13 vs. limit=22.5
2023-11-22 03:07:21,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1770940.0, ans=0.04949747468305833
2023-11-22 03:07:24,849 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265650
2023-11-22 03:07:42,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1771073.3333333333, ans=0.025
2023-11-22 03:07:54,452 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1150, loss[loss=0.1064, simple_loss=0.1491, pruned_loss=0.02599, audio_tagging_loss=0.005862, over 14915.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09542, pruned_loss=0.01564, audio_tagging_loss=0.009324, over 3042654.69 frames. ], batch size: 52, lr: 3.00e-03, grad_scale: 8.0
2023-11-22 03:07:54,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1771140.0, ans=0.1
2023-11-22 03:08:05,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1771140.0, ans=0.125
2023-11-22 03:08:06,152 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.33 vs. limit=10.0
2023-11-22 03:08:29,849 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265700
2023-11-22 03:08:31,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1771273.3333333333, ans=0.125
2023-11-22 03:08:33,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1771340.0, ans=0.0
2023-11-22 03:08:47,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1771406.6666666667, ans=0.0
2023-11-22 03:08:59,913 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1200, loss[loss=0.08356, simple_loss=0.1144, pruned_loss=0.0184, audio_tagging_loss=0.007965, over 15538.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09611, pruned_loss=0.01579, audio_tagging_loss=0.009206, over 3045283.32 frames. ], batch size: 59, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:09:21,603 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.059e+01 8.626e+01 9.376e+01 1.132e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 03:09:23,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1771540.0, ans=0.0
2023-11-22 03:09:25,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1771606.6666666667, ans=0.05
2023-11-22 03:09:34,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265750
2023-11-22 03:09:37,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1771673.3333333333, ans=0.2
2023-11-22 03:09:42,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1771673.3333333333, ans=0.0
2023-11-22 03:09:45,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1771673.3333333333, ans=0.125
2023-11-22 03:09:56,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1771740.0, ans=0.125
2023-11-22 03:10:04,582 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1250, loss[loss=0.07517, simple_loss=0.1044, pruned_loss=0.01284, audio_tagging_loss=0.01011, over 14856.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.09556, pruned_loss=0.01588, audio_tagging_loss=0.009179, over 3043041.23 frames. ], batch size: 54, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:10:11,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1771806.6666666667, ans=0.1
2023-11-22 03:10:21,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1771873.3333333333, ans=0.2
2023-11-22 03:10:25,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1771873.3333333333, ans=0.0
2023-11-22 03:10:28,231 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.32 vs. limit=15.0
2023-11-22 03:10:29,243 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.18 vs. limit=15.0
2023-11-22 03:10:30,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1771940.0, ans=0.0
2023-11-22 03:10:39,928 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265800
2023-11-22 03:10:56,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1772073.3333333333, ans=0.125
2023-11-22 03:11:10,109 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1300, loss[loss=0.06141, simple_loss=0.08393, pruned_loss=0.01271, audio_tagging_loss=0.00673, over 15584.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09524, pruned_loss=0.01571, audio_tagging_loss=0.009184, over 3045445.54 frames. ], batch size: 59, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:11:10,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1772140.0, ans=0.1
2023-11-22 03:11:32,789 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.081e+01 8.746e+01 9.574e+01 1.188e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 03:11:40,866 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.33 vs. limit=12.0
2023-11-22 03:11:41,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1772273.3333333333, ans=0.95
2023-11-22 03:11:45,784 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265850
2023-11-22 03:11:50,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1772340.0, ans=0.09899494936611666
2023-11-22 03:11:51,110 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.38 vs. limit=15.0
2023-11-22 03:11:53,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1772340.0, ans=0.125
2023-11-22 03:12:04,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1772406.6666666667, ans=0.5
2023-11-22 03:12:14,908 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1350, loss[loss=0.09087, simple_loss=0.1222, pruned_loss=0.02096, audio_tagging_loss=0.00879, over 15921.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09484, pruned_loss=0.01561, audio_tagging_loss=0.009233, over 3049310.46 frames. ], batch size: 57, lr: 3.00e-03, grad_scale: 16.0
2023-11-22 03:12:24,632 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=15.0
2023-11-22 03:12:26,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1772540.0, ans=0.04949747468305833
2023-11-22 03:12:49,933 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265900
2023-11-22 03:12:55,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1772673.3333333333, ans=0.09899494936611666
2023-11-22 03:12:58,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1772673.3333333333, ans=0.2
2023-11-22 03:13:01,458 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:13:05,520 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:13:11,501 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.65 vs. limit=15.0
2023-11-22 03:13:12,410 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.22 vs. limit=15.0
2023-11-22 03:13:19,329 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1400, loss[loss=0.0785, simple_loss=0.09631, pruned_loss=0.02047, audio_tagging_loss=0.009882, over 15283.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09437, pruned_loss=0.01563, audio_tagging_loss=0.00928, over 3038767.44 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:13:40,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1772873.3333333333, ans=0.125
2023-11-22 03:13:41,399 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.909e+01 8.105e+01 8.866e+01 9.820e+01 1.190e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 03:13:47,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1772940.0, ans=0.125
2023-11-22 03:13:54,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 265950
2023-11-22 03:13:55,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1772940.0, ans=0.05
2023-11-22 03:13:56,311 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.57 vs. limit=6.0
2023-11-22 03:14:14,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1773073.3333333333, ans=0.125
2023-11-22 03:14:16,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1773073.3333333333, ans=0.1
2023-11-22 03:14:23,827 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1450, loss[loss=0.06692, simple_loss=0.08379, pruned_loss=0.01617, audio_tagging_loss=0.008847, over 14917.00 frames. ], tot_loss[loss=0.07251, simple_loss=0.0947, pruned_loss=0.0158, audio_tagging_loss=0.009365, over 3038334.65 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:14:25,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1773140.0, ans=0.1
2023-11-22 03:14:28,724 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.95 vs. limit=22.5
2023-11-22 03:14:31,959 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:14:35,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1773206.6666666667, ans=0.0
2023-11-22 03:14:51,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1773273.3333333333, ans=0.015
2023-11-22 03:14:51,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1773273.3333333333, ans=0.125
2023-11-22 03:14:58,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266000
2023-11-22 03:15:03,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1773340.0, ans=0.125
2023-11-22 03:15:08,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1773340.0, ans=0.07
2023-11-22 03:15:20,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1773406.6666666667, ans=0.125
2023-11-22 03:15:28,996 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1500, loss[loss=0.07359, simple_loss=0.09443, pruned_loss=0.01502, audio_tagging_loss=0.01135, over 14249.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.09448, pruned_loss=0.01583, audio_tagging_loss=0.009596, over 3034984.54 frames. ], batch size: 54, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:15:32,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.66 vs. limit=10.0
2023-11-22 03:15:33,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1773473.3333333333, ans=0.2
2023-11-22 03:15:51,116 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.300e+01 8.986e+01 9.912e+01 1.319e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 03:15:56,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1773606.6666666667, ans=0.2
2023-11-22 03:16:04,212 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266050
2023-11-22 03:16:15,768 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.35 vs. limit=6.0
2023-11-22 03:16:16,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1773673.3333333333, ans=0.0
2023-11-22 03:16:34,338 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1550, loss[loss=0.08764, simple_loss=0.1203, pruned_loss=0.02025, audio_tagging_loss=0.00725, over 14662.00 frames. ], tot_loss[loss=0.07252, simple_loss=0.09412, pruned_loss=0.01573, audio_tagging_loss=0.009725, over 3030823.51 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:16:54,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=1773873.3333333333, ans=0.05
2023-11-22 03:16:58,114 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.02 vs. limit=12.0
2023-11-22 03:17:09,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266100
2023-11-22 03:17:19,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1774006.6666666667, ans=0.125
2023-11-22 03:17:38,575 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1600, loss[loss=0.07446, simple_loss=0.09478, pruned_loss=0.01682, audio_tagging_loss=0.01025, over 15164.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09562, pruned_loss=0.01607, audio_tagging_loss=0.00967, over 3041788.55 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:18:00,641 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.210e+01 8.853e+01 9.521e+01 1.233e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 03:18:02,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1774206.6666666667, ans=0.125
2023-11-22 03:18:02,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1774206.6666666667, ans=0.125
2023-11-22 03:18:07,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.57 vs. limit=22.5
2023-11-22 03:18:13,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266150
2023-11-22 03:18:19,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1774340.0, ans=0.1
2023-11-22 03:18:31,346 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.51 vs. limit=22.5
2023-11-22 03:18:43,453 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1650, loss[loss=0.09654, simple_loss=0.1293, pruned_loss=0.02323, audio_tagging_loss=0.008667, over 15915.00 frames. ], tot_loss[loss=0.07372, simple_loss=0.09579, pruned_loss=0.01612, audio_tagging_loss=0.009702, over 3043758.05 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:18:49,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1774473.3333333333, ans=0.125
2023-11-22 03:19:18,925 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266200
2023-11-22 03:19:19,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1774606.6666666667, ans=0.125
2023-11-22 03:19:22,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1774673.3333333333, ans=0.0
2023-11-22 03:19:48,672 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1700, loss[loss=0.08238, simple_loss=0.1152, pruned_loss=0.01792, audio_tagging_loss=0.006875, over 16701.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09513, pruned_loss=0.01589, audio_tagging_loss=0.009769, over 3050621.63 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:19:58,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1774806.6666666667, ans=0.125
2023-11-22 03:20:05,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1774873.3333333333, ans=0.0
2023-11-22 03:20:05,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1774873.3333333333, ans=0.1
2023-11-22 03:20:11,471 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.074e+01 8.655e+01 9.245e+01 1.270e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-22 03:20:21,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1774940.0, ans=0.1
2023-11-22 03:20:24,688 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266250
2023-11-22 03:20:25,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1774940.0, ans=0.125
2023-11-22 03:20:28,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1775006.6666666667, ans=0.0
2023-11-22 03:20:41,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1775073.3333333333, ans=0.2
2023-11-22 03:20:43,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.79 vs. limit=15.0
2023-11-22 03:20:54,033 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1750, loss[loss=0.07908, simple_loss=0.1147, pruned_loss=0.01196, audio_tagging_loss=0.009748, over 16313.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09543, pruned_loss=0.01585, audio_tagging_loss=0.009616, over 3052276.48 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:21:00,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1775140.0, ans=0.2
2023-11-22 03:21:10,409 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.96 vs. limit=22.5
2023-11-22 03:21:12,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1775206.6666666667, ans=0.125
2023-11-22 03:21:28,421 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.81 vs. limit=15.0
2023-11-22 03:21:28,895 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266300
2023-11-22 03:21:29,204 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:21:36,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1775340.0, ans=0.125
2023-11-22 03:21:55,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1775406.6666666667, ans=0.0
2023-11-22 03:21:57,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=15.78 vs. limit=15.0
2023-11-22 03:21:58,585 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1800, loss[loss=0.06141, simple_loss=0.08653, pruned_loss=0.01054, audio_tagging_loss=0.007601, over 14821.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09507, pruned_loss=0.01582, audio_tagging_loss=0.00945, over 3045147.40 frames. ], batch size: 54, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:22:05,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1775473.3333333333, ans=0.1
2023-11-22 03:22:21,025 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.483e+01 7.973e+01 8.651e+01 9.483e+01 1.357e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 03:22:29,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1775606.6666666667, ans=0.125
2023-11-22 03:22:33,248 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266350
2023-11-22 03:22:35,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1775673.3333333333, ans=0.0
2023-11-22 03:22:44,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1775673.3333333333, ans=0.1
2023-11-22 03:22:49,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1775740.0, ans=15.0
2023-11-22 03:23:00,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1775740.0, ans=0.2
2023-11-22 03:23:02,239 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1850, loss[loss=0.05532, simple_loss=0.06934, pruned_loss=0.01041, audio_tagging_loss=0.01024, over 14334.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.09608, pruned_loss=0.0159, audio_tagging_loss=0.009424, over 3049798.02 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:23:19,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1775873.3333333333, ans=0.1
2023-11-22 03:23:37,158 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266400
2023-11-22 03:23:52,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1776073.3333333333, ans=0.125
2023-11-22 03:23:55,555 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.52 vs. limit=15.0
2023-11-22 03:23:59,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.69 vs. limit=8.0
2023-11-22 03:24:06,611 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1900, loss[loss=0.06963, simple_loss=0.09539, pruned_loss=0.01214, audio_tagging_loss=0.009798, over 15491.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.0954, pruned_loss=0.01578, audio_tagging_loss=0.009371, over 3046853.96 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:24:27,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1776206.6666666667, ans=0.1
2023-11-22 03:24:30,454 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.182e+01 8.329e+01 9.047e+01 9.606e+01 1.236e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-22 03:24:40,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1776273.3333333333, ans=0.1
2023-11-22 03:24:40,711 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:24:41,893 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266450
2023-11-22 03:24:53,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1776340.0, ans=0.125
2023-11-22 03:24:56,566 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.37 vs. limit=15.0
2023-11-22 03:25:08,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1776406.6666666667, ans=0.0
2023-11-22 03:25:11,705 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 1950, loss[loss=0.06235, simple_loss=0.07685, pruned_loss=0.01187, audio_tagging_loss=0.01205, over 13831.00 frames. ], tot_loss[loss=0.07272, simple_loss=0.09519, pruned_loss=0.01575, audio_tagging_loss=0.00938, over 3052585.55 frames. ], batch size: 53, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:25:15,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1776473.3333333333, ans=0.125
2023-11-22 03:25:33,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1776540.0, ans=0.125
2023-11-22 03:25:37,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1776606.6666666667, ans=0.05
2023-11-22 03:25:41,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.47 vs. limit=6.0
2023-11-22 03:25:42,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1776606.6666666667, ans=0.0
2023-11-22 03:25:46,324 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266500
2023-11-22 03:25:53,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=1776673.3333333333, ans=10.0
2023-11-22 03:26:01,791 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.31 vs. limit=22.5
2023-11-22 03:26:16,494 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2000, loss[loss=0.07367, simple_loss=0.1093, pruned_loss=0.01247, audio_tagging_loss=0.006574, over 16264.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09416, pruned_loss=0.01561, audio_tagging_loss=0.009521, over 3041151.95 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:26:39,145 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.529e+01 7.854e+01 8.461e+01 9.076e+01 1.159e+02, threshold=1.692e+02, percent-clipped=0.0
2023-11-22 03:26:51,394 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266550
2023-11-22 03:26:59,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1777006.6666666667, ans=0.125
2023-11-22 03:27:21,308 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2050, loss[loss=0.06324, simple_loss=0.07688, pruned_loss=0.01503, audio_tagging_loss=0.009774, over 14751.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09418, pruned_loss=0.01566, audio_tagging_loss=0.00953, over 3036766.52 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:27:37,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1777206.6666666667, ans=0.125
2023-11-22 03:27:39,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1777206.6666666667, ans=0.125
2023-11-22 03:27:56,182 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266600
2023-11-22 03:28:08,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1777340.0, ans=0.1
2023-11-22 03:28:18,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1777406.6666666667, ans=0.07
2023-11-22 03:28:27,068 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2100, loss[loss=0.08791, simple_loss=0.1149, pruned_loss=0.02059, audio_tagging_loss=0.009877, over 15130.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.09456, pruned_loss=0.01577, audio_tagging_loss=0.009536, over 3031807.37 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:28:31,472 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.22 vs. limit=22.5
2023-11-22 03:28:47,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1777540.0, ans=0.125
2023-11-22 03:28:49,898 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.317e+01 8.958e+01 9.611e+01 1.215e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 03:29:01,613 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266650
2023-11-22 03:29:06,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1777673.3333333333, ans=0.0
2023-11-22 03:29:16,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1777673.3333333333, ans=0.0
2023-11-22 03:29:17,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=1777740.0, ans=0.5
2023-11-22 03:29:31,464 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2150, loss[loss=0.08811, simple_loss=0.1131, pruned_loss=0.02236, audio_tagging_loss=0.009206, over 14422.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09489, pruned_loss=0.0158, audio_tagging_loss=0.00949, over 3038349.27 frames. ], batch size: 53, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:30:07,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266700
2023-11-22 03:30:10,777 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:30:22,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1778073.3333333333, ans=0.125
2023-11-22 03:30:31,632 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.08 vs. limit=22.5
2023-11-22 03:30:36,536 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2200, loss[loss=0.08312, simple_loss=0.1042, pruned_loss=0.01971, audio_tagging_loss=0.0113, over 15562.00 frames. ], tot_loss[loss=0.07331, simple_loss=0.09585, pruned_loss=0.01592, audio_tagging_loss=0.009471, over 3041209.16 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:30:41,049 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.74 vs. limit=6.0
2023-11-22 03:30:43,580 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.76 vs. limit=15.0
2023-11-22 03:31:01,207 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.015e+01 8.446e+01 9.352e+01 1.016e+02 2.574e+02, threshold=1.870e+02, percent-clipped=1.0
2023-11-22 03:31:06,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1778273.3333333333, ans=0.125
2023-11-22 03:31:08,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1778273.3333333333, ans=0.0
2023-11-22 03:31:12,183 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266750
2023-11-22 03:31:16,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1778340.0, ans=0.125
2023-11-22 03:31:27,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1778406.6666666667, ans=0.0
2023-11-22 03:31:39,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1778406.6666666667, ans=0.125
2023-11-22 03:31:41,872 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2250, loss[loss=0.07242, simple_loss=0.09016, pruned_loss=0.01506, audio_tagging_loss=0.01229, over 14860.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09569, pruned_loss=0.01592, audio_tagging_loss=0.00944, over 3042448.81 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:31:43,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1778473.3333333333, ans=0.2
2023-11-22 03:31:43,830 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=13.10 vs. limit=15.0
2023-11-22 03:31:59,706 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.22 vs. limit=15.0
2023-11-22 03:32:14,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1778606.6666666667, ans=0.125
2023-11-22 03:32:17,310 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266800
2023-11-22 03:32:25,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1778673.3333333333, ans=0.2
2023-11-22 03:32:28,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1778673.3333333333, ans=0.0
2023-11-22 03:32:47,674 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2300, loss[loss=0.05169, simple_loss=0.06091, pruned_loss=0.0124, audio_tagging_loss=0.008833, over 13579.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.09399, pruned_loss=0.01551, audio_tagging_loss=0.009608, over 3035538.25 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:32:59,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1778873.3333333333, ans=0.0
2023-11-22 03:32:59,538 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.99 vs. limit=15.0
2023-11-22 03:33:05,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1778873.3333333333, ans=0.125
2023-11-22 03:33:11,799 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.550e+01 8.208e+01 8.733e+01 9.600e+01 1.238e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 03:33:21,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1778940.0, ans=0.015
2023-11-22 03:33:22,328 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266850
2023-11-22 03:33:43,831 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:33:52,705 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2350, loss[loss=0.07029, simple_loss=0.09195, pruned_loss=0.01527, audio_tagging_loss=0.00904, over 14876.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09489, pruned_loss=0.01573, audio_tagging_loss=0.009594, over 3037851.66 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:33:59,034 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.69 vs. limit=6.0
2023-11-22 03:34:02,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1779140.0, ans=0.125
2023-11-22 03:34:28,364 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266900
2023-11-22 03:34:45,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1779406.6666666667, ans=0.0
2023-11-22 03:34:57,447 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2400, loss[loss=0.07571, simple_loss=0.09312, pruned_loss=0.01842, audio_tagging_loss=0.01073, over 15874.00 frames. ], tot_loss[loss=0.07318, simple_loss=0.09571, pruned_loss=0.01576, audio_tagging_loss=0.009559, over 3043328.49 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:35:00,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1779473.3333333333, ans=0.0
2023-11-22 03:35:07,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1779473.3333333333, ans=0.2
2023-11-22 03:35:09,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1779540.0, ans=0.0
2023-11-22 03:35:22,842 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 7.925e+01 8.589e+01 9.306e+01 1.063e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 03:35:33,466 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 266950
2023-11-22 03:35:38,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1779673.3333333333, ans=0.0
2023-11-22 03:35:41,888 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.24 vs. limit=15.0
2023-11-22 03:36:03,418 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2450, loss[loss=0.09319, simple_loss=0.1286, pruned_loss=0.02255, audio_tagging_loss=0.006359, over 15473.00 frames. ], tot_loss[loss=0.07341, simple_loss=0.09603, pruned_loss=0.01578, audio_tagging_loss=0.009617, over 3043850.02 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:36:09,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1779806.6666666667, ans=0.125
2023-11-22 03:36:09,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1779806.6666666667, ans=0.2
2023-11-22 03:36:17,253 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.33 vs. limit=15.0
2023-11-22 03:36:26,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1779873.3333333333, ans=0.2
2023-11-22 03:36:39,249 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267000
2023-11-22 03:36:50,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1780006.6666666667, ans=0.0
2023-11-22 03:36:54,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1780006.6666666667, ans=0.0
2023-11-22 03:36:54,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1780006.6666666667, ans=0.0
2023-11-22 03:37:09,371 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2500, loss[loss=0.06379, simple_loss=0.07863, pruned_loss=0.01272, audio_tagging_loss=0.01176, over 14887.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.095, pruned_loss=0.01576, audio_tagging_loss=0.009791, over 3048010.61 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:37:34,058 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.102e+01 8.757e+01 9.446e+01 1.143e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 03:37:37,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1780273.3333333333, ans=0.125
2023-11-22 03:37:45,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267050
2023-11-22 03:37:48,894 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.00 vs. limit=12.0
2023-11-22 03:38:14,917 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2550, loss[loss=0.06379, simple_loss=0.08107, pruned_loss=0.01498, audio_tagging_loss=0.008274, over 14124.00 frames. ], tot_loss[loss=0.07266, simple_loss=0.0946, pruned_loss=0.01564, audio_tagging_loss=0.009721, over 3042351.23 frames. ], batch size: 55, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:38:36,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1780540.0, ans=0.1
2023-11-22 03:38:50,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267100
2023-11-22 03:38:52,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1780606.6666666667, ans=0.0
2023-11-22 03:38:57,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1780673.3333333333, ans=0.05
2023-11-22 03:38:57,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1780673.3333333333, ans=0.0
2023-11-22 03:39:14,697 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.23 vs. limit=15.0
2023-11-22 03:39:20,358 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2600, loss[loss=0.06772, simple_loss=0.0929, pruned_loss=0.01152, audio_tagging_loss=0.009743, over 14354.00 frames. ], tot_loss[loss=0.07177, simple_loss=0.09364, pruned_loss=0.01541, audio_tagging_loss=0.009541, over 3036694.60 frames. ], batch size: 54, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:39:20,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1780806.6666666667, ans=0.125
2023-11-22 03:39:37,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1780873.3333333333, ans=0.125
2023-11-22 03:39:38,039 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.88 vs. limit=15.0
2023-11-22 03:39:45,352 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.768e+01 8.100e+01 8.795e+01 9.749e+01 1.306e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 03:39:55,257 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267150
2023-11-22 03:40:19,519 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.63 vs. limit=15.0
2023-11-22 03:40:25,449 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2650, loss[loss=0.05208, simple_loss=0.07157, pruned_loss=0.01125, audio_tagging_loss=0.00505, over 15293.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09305, pruned_loss=0.01532, audio_tagging_loss=0.009344, over 3039729.29 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:40:41,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1781206.6666666667, ans=0.125
2023-11-22 03:40:57,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1781273.3333333333, ans=0.125
2023-11-22 03:41:00,648 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267200
2023-11-22 03:41:20,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1781406.6666666667, ans=0.0
2023-11-22 03:41:24,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1781406.6666666667, ans=0.2
2023-11-22 03:41:30,833 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2700, loss[loss=0.04954, simple_loss=0.0582, pruned_loss=0.007978, audio_tagging_loss=0.01246, over 15684.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09357, pruned_loss=0.0153, audio_tagging_loss=0.009451, over 3040313.84 frames. ], batch size: 58, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:41:33,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1781473.3333333333, ans=0.1
2023-11-22 03:41:39,654 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.82 vs. limit=15.0
2023-11-22 03:41:42,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1781540.0, ans=0.125
2023-11-22 03:41:52,493 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:41:57,230 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.172e+01 8.019e+01 8.519e+01 9.203e+01 1.202e+02, threshold=1.704e+02, percent-clipped=0.0
2023-11-22 03:42:06,770 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267250
2023-11-22 03:42:30,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1781740.0, ans=0.125
2023-11-22 03:42:36,518 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2750, loss[loss=0.07703, simple_loss=0.1006, pruned_loss=0.01754, audio_tagging_loss=0.009176, over 14122.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09401, pruned_loss=0.01532, audio_tagging_loss=0.009356, over 3039906.76 frames. ], batch size: 54, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:42:50,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1781873.3333333333, ans=0.125
2023-11-22 03:43:02,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1781940.0, ans=0.025
2023-11-22 03:43:09,777 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.03 vs. limit=10.0
2023-11-22 03:43:11,830 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267300
2023-11-22 03:43:25,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1782006.6666666667, ans=0.0
2023-11-22 03:43:25,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.15 vs. limit=10.0
2023-11-22 03:43:32,659 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:43:34,469 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=15.0
2023-11-22 03:43:41,235 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2800, loss[loss=0.08538, simple_loss=0.112, pruned_loss=0.02054, audio_tagging_loss=0.008839, over 14693.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09419, pruned_loss=0.01542, audio_tagging_loss=0.009244, over 3040294.85 frames. ], batch size: 53, lr: 2.99e-03, grad_scale: 32.0
2023-11-22 03:44:06,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1782273.3333333333, ans=0.1
2023-11-22 03:44:07,787 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.687e+01 8.173e+01 8.676e+01 9.379e+01 1.430e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 03:44:13,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1782273.3333333333, ans=0.2
2023-11-22 03:44:16,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267350
2023-11-22 03:44:35,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.49 vs. limit=10.0
2023-11-22 03:44:47,361 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2850, loss[loss=0.07864, simple_loss=0.1105, pruned_loss=0.01614, audio_tagging_loss=0.00725, over 15493.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09429, pruned_loss=0.01553, audio_tagging_loss=0.009224, over 3043128.79 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:45:00,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1782540.0, ans=0.125
2023-11-22 03:45:22,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267400
2023-11-22 03:45:23,281 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.28 vs. limit=15.0
2023-11-22 03:45:42,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1782740.0, ans=0.0
2023-11-22 03:45:43,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1782740.0, ans=0.09899494936611666
2023-11-22 03:45:52,365 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2900, loss[loss=0.0485, simple_loss=0.05029, pruned_loss=0.0106, audio_tagging_loss=0.01276, over 15191.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.0946, pruned_loss=0.01569, audio_tagging_loss=0.009305, over 3044222.92 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:46:14,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=13.00 vs. limit=15.0
2023-11-22 03:46:19,365 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.173e+01 8.778e+01 9.450e+01 1.328e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 03:46:27,407 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267450
2023-11-22 03:46:31,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1783006.6666666667, ans=0.0
2023-11-22 03:46:35,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1783006.6666666667, ans=0.2
2023-11-22 03:46:37,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1783006.6666666667, ans=0.0
2023-11-22 03:46:40,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1783006.6666666667, ans=0.0
2023-11-22 03:46:44,624 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.91 vs. limit=15.0
2023-11-22 03:46:45,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1783073.3333333333, ans=0.125
2023-11-22 03:46:56,574 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 2950, loss[loss=0.09542, simple_loss=0.129, pruned_loss=0.02257, audio_tagging_loss=0.008334, over 15656.00 frames. ], tot_loss[loss=0.07284, simple_loss=0.0952, pruned_loss=0.01594, audio_tagging_loss=0.009298, over 3044949.95 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:46:59,889 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.66 vs. limit=15.0
2023-11-22 03:47:00,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1783140.0, ans=0.125
2023-11-22 03:47:08,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1783206.6666666667, ans=0.1
2023-11-22 03:47:15,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1783206.6666666667, ans=0.125
2023-11-22 03:47:24,381 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.28 vs. limit=15.0
2023-11-22 03:47:31,589 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267500
2023-11-22 03:47:50,043 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.23 vs. limit=22.5
2023-11-22 03:48:01,457 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3000, loss[loss=0.08603, simple_loss=0.1176, pruned_loss=0.02029, audio_tagging_loss=0.006942, over 15687.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.09564, pruned_loss=0.01605, audio_tagging_loss=0.009347, over 3053712.56 frames. ], batch size: 56, lr: 2.99e-03, grad_scale: 16.0
2023-11-22 03:48:01,460 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 03:48:36,343 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.7807, 1.3683, 3.5622, 3.1671, 2.9936, 3.3216, 2.8890, 3.3162],
       device='cuda:0')
2023-11-22 03:48:39,909 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.0287, 1.9627, 3.3783, 2.8626, 3.7465, 3.6632, 3.2013, 3.2100],
       device='cuda:0')
2023-11-22 03:48:41,400 INFO [train_asr.py:1253] (0/4) Epoch 23, validation: loss=0.05946, simple_loss=0.05181, pruned_loss=0.005129, audio_tagging_loss=0.02843, over 4681554.00 frames. 
2023-11-22 03:48:41,400 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 03:48:51,826 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.58 vs. limit=22.5
2023-11-22 03:49:07,733 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.414e+01 8.252e+01 8.878e+01 9.473e+01 1.263e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-22 03:49:15,810 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267550
2023-11-22 03:49:19,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1783673.3333333333, ans=0.1
2023-11-22 03:49:25,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1783673.3333333333, ans=0.125
2023-11-22 03:49:33,024 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:49:44,976 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3050, loss[loss=0.05621, simple_loss=0.06703, pruned_loss=0.009823, audio_tagging_loss=0.01287, over 15253.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.0959, pruned_loss=0.016, audio_tagging_loss=0.0094, over 3053192.86 frames. ], batch size: 59, lr: 2.99e-03, grad_scale: 8.0
2023-11-22 03:49:47,811 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:49:49,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.92 vs. limit=15.0
2023-11-22 03:50:09,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1783940.0, ans=0.125
2023-11-22 03:50:09,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1783940.0, ans=0.125
2023-11-22 03:50:20,640 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267600
2023-11-22 03:50:23,299 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:50:40,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1784073.3333333333, ans=0.1
2023-11-22 03:50:44,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1784073.3333333333, ans=0.1
2023-11-22 03:50:49,188 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3100, loss[loss=0.06624, simple_loss=0.09421, pruned_loss=0.01092, audio_tagging_loss=0.008219, over 14906.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.09593, pruned_loss=0.01578, audio_tagging_loss=0.009405, over 3056012.28 frames. ], batch size: 57, lr: 2.99e-03, grad_scale: 8.0
2023-11-22 03:51:17,604 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.039e+01 8.127e+01 8.791e+01 9.439e+01 1.386e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 03:51:23,768 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267650
2023-11-22 03:51:44,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1784406.6666666667, ans=0.125
2023-11-22 03:51:53,230 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3150, loss[loss=0.07823, simple_loss=0.09597, pruned_loss=0.01768, audio_tagging_loss=0.01256, over 16238.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09618, pruned_loss=0.01592, audio_tagging_loss=0.009587, over 3053807.84 frames. ], batch size: 60, lr: 2.99e-03, grad_scale: 8.0
2023-11-22 03:52:04,230 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.72 vs. limit=22.5
2023-11-22 03:52:23,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1784606.6666666667, ans=0.1
2023-11-22 03:52:27,543 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267700
2023-11-22 03:52:33,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1784673.3333333333, ans=0.0
2023-11-22 03:52:41,418 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:52:46,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1784740.0, ans=0.1
2023-11-22 03:52:50,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1784740.0, ans=0.125
2023-11-22 03:52:57,567 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3200, loss[loss=0.07976, simple_loss=0.108, pruned_loss=0.01742, audio_tagging_loss=0.008325, over 15336.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09596, pruned_loss=0.01581, audio_tagging_loss=0.009582, over 3047518.38 frames. ], batch size: 59, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:53:01,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1784806.6666666667, ans=0.125
2023-11-22 03:53:06,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.79 vs. limit=15.0
2023-11-22 03:53:10,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1784873.3333333333, ans=0.125
2023-11-22 03:53:23,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1784940.0, ans=0.125
2023-11-22 03:53:26,395 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.278e+01 8.640e+01 9.610e+01 1.233e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-22 03:53:33,308 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267750
2023-11-22 03:53:50,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1785073.3333333333, ans=0.125
2023-11-22 03:54:01,751 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3250, loss[loss=0.06199, simple_loss=0.08106, pruned_loss=0.01028, audio_tagging_loss=0.01119, over 14761.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09511, pruned_loss=0.01566, audio_tagging_loss=0.009771, over 3050553.70 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:54:18,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1785206.6666666667, ans=0.0
2023-11-22 03:54:26,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1785273.3333333333, ans=0.125
2023-11-22 03:54:27,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1785273.3333333333, ans=0.125
2023-11-22 03:54:35,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267800
2023-11-22 03:55:06,139 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3300, loss[loss=0.05437, simple_loss=0.07355, pruned_loss=0.008177, audio_tagging_loss=0.009423, over 15580.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09492, pruned_loss=0.01561, audio_tagging_loss=0.009837, over 3057716.06 frames. ], batch size: 60, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:55:21,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1785540.0, ans=0.1
2023-11-22 03:55:25,875 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.85 vs. limit=15.0
2023-11-22 03:55:26,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1785540.0, ans=0.125
2023-11-22 03:55:32,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1785606.6666666667, ans=10.0
2023-11-22 03:55:34,171 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.246e+01 8.906e+01 9.569e+01 1.172e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-22 03:55:35,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1785606.6666666667, ans=0.0
2023-11-22 03:55:40,367 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267850
2023-11-22 03:55:49,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1785673.3333333333, ans=0.125
2023-11-22 03:55:51,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1785673.3333333333, ans=10.0
2023-11-22 03:55:55,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1785673.3333333333, ans=0.125
2023-11-22 03:55:55,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1785673.3333333333, ans=0.2
2023-11-22 03:56:09,861 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3350, loss[loss=0.1016, simple_loss=0.1246, pruned_loss=0.02743, audio_tagging_loss=0.01181, over 16111.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09462, pruned_loss=0.01557, audio_tagging_loss=0.009695, over 3059697.82 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:56:11,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1785806.6666666667, ans=0.125
2023-11-22 03:56:11,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1785806.6666666667, ans=0.0
2023-11-22 03:56:21,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1785873.3333333333, ans=0.2
2023-11-22 03:56:22,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1785873.3333333333, ans=0.2
2023-11-22 03:56:44,849 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267900
2023-11-22 03:56:48,049 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 03:57:01,687 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.21 vs. limit=15.0
2023-11-22 03:57:08,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1786073.3333333333, ans=0.0
2023-11-22 03:57:11,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1786073.3333333333, ans=0.0
2023-11-22 03:57:13,702 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3400, loss[loss=0.07498, simple_loss=0.1002, pruned_loss=0.01672, audio_tagging_loss=0.00817, over 16032.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09533, pruned_loss=0.01577, audio_tagging_loss=0.009516, over 3060944.91 frames. ], batch size: 61, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 03:57:35,029 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.39 vs. limit=15.0
2023-11-22 03:57:42,474 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.335e+01 8.871e+01 9.703e+01 1.398e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 03:57:48,710 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 267950
2023-11-22 03:57:51,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1786340.0, ans=0.1
2023-11-22 03:57:53,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1786340.0, ans=0.1
2023-11-22 03:57:56,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1786340.0, ans=0.125
2023-11-22 03:58:07,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1786406.6666666667, ans=0.0
2023-11-22 03:58:18,138 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3450, loss[loss=0.1061, simple_loss=0.1473, pruned_loss=0.02516, audio_tagging_loss=0.0073, over 16743.00 frames. ], tot_loss[loss=0.07299, simple_loss=0.09554, pruned_loss=0.01582, audio_tagging_loss=0.009402, over 3051617.94 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 03:58:18,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1786473.3333333333, ans=0.0
2023-11-22 03:58:19,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff2.min_abs, batch_count=1786473.3333333333, ans=0.1
2023-11-22 03:58:28,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.98 vs. limit=15.0
2023-11-22 03:58:29,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1786473.3333333333, ans=0.2
2023-11-22 03:58:31,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1786540.0, ans=0.125
2023-11-22 03:58:39,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1786540.0, ans=0.125
2023-11-22 03:58:41,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1786540.0, ans=0.125
2023-11-22 03:58:53,190 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268000
2023-11-22 03:58:54,762 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-268000.pt
2023-11-22 03:58:59,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1786673.3333333333, ans=0.125
2023-11-22 03:59:18,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1786740.0, ans=0.0
2023-11-22 03:59:21,513 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.43 vs. limit=15.0
2023-11-22 03:59:25,611 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3500, loss[loss=0.06644, simple_loss=0.08098, pruned_loss=0.01556, audio_tagging_loss=0.01039, over 15277.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09412, pruned_loss=0.01548, audio_tagging_loss=0.009388, over 3053394.45 frames. ], batch size: 59, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 03:59:33,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1786806.6666666667, ans=0.0
2023-11-22 03:59:34,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1786806.6666666667, ans=0.0
2023-11-22 03:59:36,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1786873.3333333333, ans=10.0
2023-11-22 03:59:37,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1786873.3333333333, ans=0.125
2023-11-22 03:59:54,069 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.643e+01 7.936e+01 8.648e+01 9.422e+01 1.502e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 03:59:58,417 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 03:59:59,697 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268050
2023-11-22 04:00:08,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-22 04:00:09,937 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.43 vs. limit=15.0
2023-11-22 04:00:14,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1787006.6666666667, ans=0.0
2023-11-22 04:00:19,152 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:00:29,143 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3550, loss[loss=0.07105, simple_loss=0.1021, pruned_loss=0.01119, audio_tagging_loss=0.008817, over 15111.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09446, pruned_loss=0.01544, audio_tagging_loss=0.009277, over 3053710.07 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:00:32,401 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.62 vs. limit=15.0
2023-11-22 04:00:33,393 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.46 vs. limit=6.0
2023-11-22 04:00:48,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1787206.6666666667, ans=0.1
2023-11-22 04:01:01,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1787273.3333333333, ans=0.125
2023-11-22 04:01:03,895 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268100
2023-11-22 04:01:11,747 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.19 vs. limit=15.0
2023-11-22 04:01:20,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=1787406.6666666667, ans=0.1
2023-11-22 04:01:20,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=1787406.6666666667, ans=0.2
2023-11-22 04:01:21,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1787406.6666666667, ans=0.125
2023-11-22 04:01:23,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1787406.6666666667, ans=0.125
2023-11-22 04:01:31,969 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3600, loss[loss=0.05771, simple_loss=0.07139, pruned_loss=0.01121, audio_tagging_loss=0.01081, over 16090.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09413, pruned_loss=0.01551, audio_tagging_loss=0.009358, over 3058713.80 frames. ], batch size: 63, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:01:32,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1787473.3333333333, ans=0.1
2023-11-22 04:01:43,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1787473.3333333333, ans=0.0
2023-11-22 04:01:47,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1787540.0, ans=0.125
2023-11-22 04:02:01,924 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.327e+01 8.799e+01 9.625e+01 1.533e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 04:02:07,511 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268150
2023-11-22 04:02:13,132 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.84 vs. limit=15.0
2023-11-22 04:02:15,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1787673.3333333333, ans=0.125
2023-11-22 04:02:37,505 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3650, loss[loss=0.06412, simple_loss=0.08309, pruned_loss=0.01305, audio_tagging_loss=0.009527, over 14275.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09407, pruned_loss=0.01555, audio_tagging_loss=0.009296, over 3052175.12 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:02:50,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1787873.3333333333, ans=0.1
2023-11-22 04:03:05,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1787940.0, ans=0.125
2023-11-22 04:03:09,147 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.53 vs. limit=22.5
2023-11-22 04:03:10,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268200
2023-11-22 04:03:15,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1788006.6666666667, ans=0.125
2023-11-22 04:03:40,418 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3700, loss[loss=0.06407, simple_loss=0.08783, pruned_loss=0.01356, audio_tagging_loss=0.006595, over 15385.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09429, pruned_loss=0.01562, audio_tagging_loss=0.009192, over 3058246.06 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:03:46,559 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.49 vs. limit=12.0
2023-11-22 04:03:48,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.68 vs. limit=22.5
2023-11-22 04:03:58,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1788206.6666666667, ans=0.0
2023-11-22 04:03:59,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1788206.6666666667, ans=0.125
2023-11-22 04:04:09,839 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.401e+01 8.835e+01 9.477e+01 1.225e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 04:04:15,642 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268250
2023-11-22 04:04:33,465 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.79 vs. limit=15.0
2023-11-22 04:04:33,728 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.15 vs. limit=12.0
2023-11-22 04:04:43,992 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3750, loss[loss=0.08524, simple_loss=0.1071, pruned_loss=0.02079, audio_tagging_loss=0.0109, over 15894.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09469, pruned_loss=0.01585, audio_tagging_loss=0.009269, over 3059429.22 frames. ], batch size: 60, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:04:44,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1788473.3333333333, ans=0.0
2023-11-22 04:04:57,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1788540.0, ans=0.07
2023-11-22 04:05:18,524 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268300
2023-11-22 04:05:18,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1788606.6666666667, ans=0.1
2023-11-22 04:05:21,858 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.78 vs. limit=15.0
2023-11-22 04:05:27,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1788673.3333333333, ans=0.1
2023-11-22 04:05:28,625 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:05:48,209 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3800, loss[loss=0.07119, simple_loss=0.09256, pruned_loss=0.01438, audio_tagging_loss=0.01053, over 14423.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09539, pruned_loss=0.01585, audio_tagging_loss=0.009361, over 3056040.20 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:05:52,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1788806.6666666667, ans=0.1
2023-11-22 04:06:10,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1788873.3333333333, ans=0.125
2023-11-22 04:06:18,647 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.152e+01 8.764e+01 9.512e+01 1.350e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 04:06:20,634 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.82 vs. limit=12.0
2023-11-22 04:06:22,472 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268350
2023-11-22 04:06:34,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1789006.6666666667, ans=0.125
2023-11-22 04:06:52,263 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3850, loss[loss=0.06607, simple_loss=0.09217, pruned_loss=0.01007, audio_tagging_loss=0.00992, over 16180.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.0956, pruned_loss=0.01575, audio_tagging_loss=0.00939, over 3057063.64 frames. ], batch size: 61, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:07:00,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1789140.0, ans=0.125
2023-11-22 04:07:03,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1789140.0, ans=0.125
2023-11-22 04:07:05,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1789206.6666666667, ans=0.125
2023-11-22 04:07:18,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1789273.3333333333, ans=0.2
2023-11-22 04:07:27,387 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268400
2023-11-22 04:07:31,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1789340.0, ans=0.125
2023-11-22 04:07:33,336 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.59 vs. limit=22.5
2023-11-22 04:07:42,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.73 vs. limit=22.5
2023-11-22 04:07:43,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1789406.6666666667, ans=0.2
2023-11-22 04:07:47,627 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:07:54,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1789406.6666666667, ans=0.015
2023-11-22 04:07:57,335 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3900, loss[loss=0.06508, simple_loss=0.09026, pruned_loss=0.01058, audio_tagging_loss=0.009377, over 15561.00 frames. ], tot_loss[loss=0.07321, simple_loss=0.0959, pruned_loss=0.01581, audio_tagging_loss=0.009446, over 3052438.83 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:08:17,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1789540.0, ans=0.0
2023-11-22 04:08:28,733 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.625e+01 8.044e+01 8.875e+01 9.539e+01 1.176e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 04:08:32,537 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268450
2023-11-22 04:08:35,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1789673.3333333333, ans=0.2
2023-11-22 04:08:41,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1789673.3333333333, ans=0.125
2023-11-22 04:08:43,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=1789673.3333333333, ans=15.0
2023-11-22 04:08:47,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1789740.0, ans=0.0
2023-11-22 04:08:48,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1789740.0, ans=0.125
2023-11-22 04:08:49,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1789740.0, ans=0.125
2023-11-22 04:08:58,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1789740.0, ans=0.125
2023-11-22 04:09:01,975 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 3950, loss[loss=0.0748, simple_loss=0.1022, pruned_loss=0.01544, audio_tagging_loss=0.008252, over 15201.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09503, pruned_loss=0.01564, audio_tagging_loss=0.009522, over 3058198.53 frames. ], batch size: 59, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:09:05,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1789806.6666666667, ans=0.1
2023-11-22 04:09:30,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.81 vs. limit=12.0
2023-11-22 04:09:32,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1789940.0, ans=10.0
2023-11-22 04:09:35,908 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268500
2023-11-22 04:09:41,557 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.01 vs. limit=15.0
2023-11-22 04:09:45,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1790006.6666666667, ans=0.125
2023-11-22 04:09:45,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1790006.6666666667, ans=0.125
2023-11-22 04:09:46,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1790006.6666666667, ans=0.1
2023-11-22 04:10:04,737 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4000, loss[loss=0.07386, simple_loss=0.0933, pruned_loss=0.01762, audio_tagging_loss=0.00959, over 14712.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09468, pruned_loss=0.01567, audio_tagging_loss=0.009622, over 3062179.05 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:10:14,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1790140.0, ans=0.125
2023-11-22 04:10:23,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1790206.6666666667, ans=0.125
2023-11-22 04:10:32,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1790273.3333333333, ans=0.2
2023-11-22 04:10:36,043 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.201e+01 8.416e+01 8.758e+01 9.509e+01 1.185e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 04:10:37,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1790273.3333333333, ans=0.125
2023-11-22 04:10:40,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268550
2023-11-22 04:10:44,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1790340.0, ans=0.2
2023-11-22 04:10:50,757 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.16 vs. limit=15.0
2023-11-22 04:11:01,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1790406.6666666667, ans=0.0
2023-11-22 04:11:09,605 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4050, loss[loss=0.08535, simple_loss=0.1205, pruned_loss=0.01638, audio_tagging_loss=0.008719, over 14381.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09524, pruned_loss=0.01564, audio_tagging_loss=0.009665, over 3059859.52 frames. ], batch size: 54, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:11:12,184 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:11:20,155 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.84 vs. limit=22.5
2023-11-22 04:11:37,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1790606.6666666667, ans=10.0
2023-11-22 04:11:43,678 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268600
2023-11-22 04:11:55,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.93 vs. limit=15.0
2023-11-22 04:11:59,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1790740.0, ans=0.2
2023-11-22 04:12:13,010 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.37 vs. limit=22.5
2023-11-22 04:12:13,286 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4100, loss[loss=0.05489, simple_loss=0.06338, pruned_loss=0.008592, audio_tagging_loss=0.01461, over 16052.00 frames. ], tot_loss[loss=0.07319, simple_loss=0.09545, pruned_loss=0.01577, audio_tagging_loss=0.009688, over 3063386.58 frames. ], batch size: 63, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:12:17,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1790806.6666666667, ans=0.1
2023-11-22 04:12:44,093 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.196e+01 8.304e+01 9.033e+01 9.578e+01 1.202e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-22 04:12:47,858 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268650
2023-11-22 04:12:48,092 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:13:17,309 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4150, loss[loss=0.1095, simple_loss=0.1446, pruned_loss=0.03128, audio_tagging_loss=0.005891, over 15705.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.09523, pruned_loss=0.0156, audio_tagging_loss=0.009522, over 3059251.64 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:13:27,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1791140.0, ans=0.125
2023-11-22 04:13:36,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1791206.6666666667, ans=0.125
2023-11-22 04:13:52,789 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268700
2023-11-22 04:14:03,570 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:14:10,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1791406.6666666667, ans=0.0
2023-11-22 04:14:13,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1791406.6666666667, ans=0.125
2023-11-22 04:14:21,880 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4200, loss[loss=0.07114, simple_loss=0.08788, pruned_loss=0.01693, audio_tagging_loss=0.01027, over 14069.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09451, pruned_loss=0.01549, audio_tagging_loss=0.009439, over 3054516.96 frames. ], batch size: 53, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:14:27,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1791473.3333333333, ans=0.125
2023-11-22 04:14:41,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1791540.0, ans=0.0
2023-11-22 04:14:44,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1791540.0, ans=0.125
2023-11-22 04:14:50,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.94 vs. limit=6.0
2023-11-22 04:14:51,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.07 vs. limit=15.0
2023-11-22 04:14:52,155 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.143e+01 8.798e+01 9.723e+01 1.732e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 04:14:55,947 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268750
2023-11-22 04:15:02,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1791673.3333333333, ans=0.1
2023-11-22 04:15:12,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1791740.0, ans=0.2
2023-11-22 04:15:16,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1791740.0, ans=0.035
2023-11-22 04:15:22,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1791740.0, ans=0.1
2023-11-22 04:15:25,758 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4250, loss[loss=0.09559, simple_loss=0.1295, pruned_loss=0.02172, audio_tagging_loss=0.00912, over 15411.00 frames. ], tot_loss[loss=0.07311, simple_loss=0.09601, pruned_loss=0.0158, audio_tagging_loss=0.009306, over 3054989.28 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:15:36,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1791806.6666666667, ans=0.1
2023-11-22 04:15:44,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.47 vs. limit=15.0
2023-11-22 04:15:59,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1791940.0, ans=0.125
2023-11-22 04:16:00,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268800
2023-11-22 04:16:04,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1792006.6666666667, ans=0.125
2023-11-22 04:16:16,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1792073.3333333333, ans=0.125
2023-11-22 04:16:29,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1792140.0, ans=0.1
2023-11-22 04:16:30,661 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4300, loss[loss=0.06812, simple_loss=0.09547, pruned_loss=0.01279, audio_tagging_loss=0.00759, over 15459.00 frames. ], tot_loss[loss=0.07377, simple_loss=0.09688, pruned_loss=0.01605, audio_tagging_loss=0.009279, over 3058463.37 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:17:01,429 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.380e+01 9.006e+01 9.846e+01 1.214e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 04:17:03,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.16 vs. limit=12.0
2023-11-22 04:17:05,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.28 vs. limit=22.5
2023-11-22 04:17:05,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268850
2023-11-22 04:17:14,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1792340.0, ans=0.125
2023-11-22 04:17:35,230 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4350, loss[loss=0.06429, simple_loss=0.08038, pruned_loss=0.01531, audio_tagging_loss=0.008792, over 15693.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.0964, pruned_loss=0.01605, audio_tagging_loss=0.009243, over 3057929.53 frames. ], batch size: 61, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:17:51,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1792540.0, ans=0.025
2023-11-22 04:18:00,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1792606.6666666667, ans=0.0
2023-11-22 04:18:04,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.whiten.whitening_limit, batch_count=1792606.6666666667, ans=12.0
2023-11-22 04:18:09,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1792606.6666666667, ans=0.0
2023-11-22 04:18:10,630 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268900
2023-11-22 04:18:13,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1792673.3333333333, ans=0.125
2023-11-22 04:18:39,826 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4400, loss[loss=0.06974, simple_loss=0.09667, pruned_loss=0.012, audio_tagging_loss=0.009407, over 15209.00 frames. ], tot_loss[loss=0.07309, simple_loss=0.09599, pruned_loss=0.01589, audio_tagging_loss=0.009211, over 3048375.51 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 32.0
2023-11-22 04:19:00,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1792873.3333333333, ans=0.125
2023-11-22 04:19:04,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1792940.0, ans=0.035
2023-11-22 04:19:10,794 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.895e+01 7.984e+01 8.605e+01 9.395e+01 1.319e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-22 04:19:14,623 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 268950
2023-11-22 04:19:38,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1793073.3333333333, ans=0.0
2023-11-22 04:19:44,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1793140.0, ans=0.125
2023-11-22 04:19:45,071 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4450, loss[loss=0.07318, simple_loss=0.0876, pruned_loss=0.01601, audio_tagging_loss=0.01337, over 16472.00 frames. ], tot_loss[loss=0.07359, simple_loss=0.09678, pruned_loss=0.01602, audio_tagging_loss=0.009175, over 3056895.05 frames. ], batch size: 63, lr: 2.98e-03, grad_scale: 32.0
2023-11-22 04:19:45,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1793140.0, ans=0.0
2023-11-22 04:20:00,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1793206.6666666667, ans=0.125
2023-11-22 04:20:05,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1793206.6666666667, ans=0.2
2023-11-22 04:20:06,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1793206.6666666667, ans=0.025
2023-11-22 04:20:19,918 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269000
2023-11-22 04:20:31,150 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.75 vs. limit=10.0
2023-11-22 04:20:37,668 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.33 vs. limit=22.5
2023-11-22 04:20:40,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1793406.6666666667, ans=0.2
2023-11-22 04:20:49,467 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4500, loss[loss=0.08784, simple_loss=0.118, pruned_loss=0.0214, audio_tagging_loss=0.007421, over 15393.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.09593, pruned_loss=0.01588, audio_tagging_loss=0.009214, over 3054495.83 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:20:54,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1793473.3333333333, ans=0.5
2023-11-22 04:21:12,109 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:21:12,343 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.26 vs. limit=15.0
2023-11-22 04:21:13,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1793540.0, ans=0.125
2023-11-22 04:21:19,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1793606.6666666667, ans=0.125
2023-11-22 04:21:20,565 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.85 vs. limit=8.0
2023-11-22 04:21:23,194 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.138e+01 8.864e+01 9.705e+01 1.342e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 04:21:24,552 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269050
2023-11-22 04:21:27,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1793673.3333333333, ans=0.125
2023-11-22 04:21:27,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1793673.3333333333, ans=0.1
2023-11-22 04:21:41,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1793740.0, ans=0.125
2023-11-22 04:21:42,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1793740.0, ans=0.1
2023-11-22 04:21:52,983 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4550, loss[loss=0.08242, simple_loss=0.1036, pruned_loss=0.02106, audio_tagging_loss=0.00958, over 14490.00 frames. ], tot_loss[loss=0.07281, simple_loss=0.09539, pruned_loss=0.0158, audio_tagging_loss=0.009313, over 3053412.62 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:21:53,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1793806.6666666667, ans=0.125
2023-11-22 04:22:06,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1793873.3333333333, ans=0.1
2023-11-22 04:22:28,048 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269100
2023-11-22 04:22:35,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1794006.6666666667, ans=0.125
2023-11-22 04:22:39,952 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.87 vs. limit=15.0
2023-11-22 04:22:39,984 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=16.39 vs. limit=22.5
2023-11-22 04:22:41,468 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:22:44,391 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.33 vs. limit=15.0
2023-11-22 04:22:49,245 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.52 vs. limit=10.0
2023-11-22 04:22:53,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1794073.3333333333, ans=0.125
2023-11-22 04:22:57,688 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4600, loss[loss=0.06284, simple_loss=0.08513, pruned_loss=0.01178, audio_tagging_loss=0.008495, over 15572.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09551, pruned_loss=0.01588, audio_tagging_loss=0.009303, over 3050456.96 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:22:59,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1794140.0, ans=0.0
2023-11-22 04:23:01,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1794140.0, ans=0.0
2023-11-22 04:23:27,375 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.51 vs. limit=22.5
2023-11-22 04:23:30,453 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.306e+01 8.737e+01 9.544e+01 1.284e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 04:23:31,789 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269150
2023-11-22 04:23:47,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1794340.0, ans=0.125
2023-11-22 04:23:54,961 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:23:59,139 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.53 vs. limit=15.0
2023-11-22 04:24:02,057 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4650, loss[loss=0.09689, simple_loss=0.1336, pruned_loss=0.02276, audio_tagging_loss=0.007321, over 15391.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.09546, pruned_loss=0.01573, audio_tagging_loss=0.009325, over 3056893.54 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:24:10,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1794473.3333333333, ans=0.125
2023-11-22 04:24:21,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1794540.0, ans=0.1
2023-11-22 04:24:32,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1794606.6666666667, ans=0.05
2023-11-22 04:24:37,459 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269200
2023-11-22 04:24:37,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1794606.6666666667, ans=0.125
2023-11-22 04:24:45,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1794673.3333333333, ans=0.0
2023-11-22 04:24:46,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1794673.3333333333, ans=0.1
2023-11-22 04:25:00,633 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.45 vs. limit=15.0
2023-11-22 04:25:01,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=1794740.0, ans=10.0
2023-11-22 04:25:05,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1794806.6666666667, ans=0.125
2023-11-22 04:25:06,244 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4700, loss[loss=0.07099, simple_loss=0.08329, pruned_loss=0.01876, audio_tagging_loss=0.01059, over 14649.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09529, pruned_loss=0.01559, audio_tagging_loss=0.009391, over 3058380.53 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:25:11,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1794806.6666666667, ans=0.125
2023-11-22 04:25:18,955 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.93 vs. limit=8.0
2023-11-22 04:25:39,202 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.243e+01 8.677e+01 9.667e+01 1.211e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 04:25:39,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1794940.0, ans=0.0
2023-11-22 04:25:40,544 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269250
2023-11-22 04:26:11,056 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4750, loss[loss=0.08452, simple_loss=0.1105, pruned_loss=0.01976, audio_tagging_loss=0.0095, over 15750.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09528, pruned_loss=0.01566, audio_tagging_loss=0.009468, over 3055425.91 frames. ], batch size: 57, lr: 2.98e-03, grad_scale: 8.0
2023-11-22 04:26:12,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1795140.0, ans=0.125
2023-11-22 04:26:20,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1795140.0, ans=0.125
2023-11-22 04:26:21,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1795140.0, ans=0.0
2023-11-22 04:26:34,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1795273.3333333333, ans=0.015
2023-11-22 04:26:44,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269300
2023-11-22 04:26:48,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1795340.0, ans=0.125
2023-11-22 04:26:56,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1795340.0, ans=0.0
2023-11-22 04:26:56,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1795340.0, ans=0.125
2023-11-22 04:27:03,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1795406.6666666667, ans=0.1
2023-11-22 04:27:06,145 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:27:07,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1795406.6666666667, ans=0.025
2023-11-22 04:27:14,499 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4800, loss[loss=0.06442, simple_loss=0.08787, pruned_loss=0.01065, audio_tagging_loss=0.009836, over 15718.00 frames. ], tot_loss[loss=0.07292, simple_loss=0.09517, pruned_loss=0.01573, audio_tagging_loss=0.009599, over 3046788.17 frames. ], batch size: 58, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:27:24,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff3.min_abs, batch_count=1795473.3333333333, ans=0.2
2023-11-22 04:27:27,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1795540.0, ans=0.125
2023-11-22 04:27:29,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1795540.0, ans=0.125
2023-11-22 04:27:34,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1795540.0, ans=0.2
2023-11-22 04:27:43,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1795606.6666666667, ans=0.125
2023-11-22 04:27:47,742 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.253e+01 9.179e+01 1.004e+02 1.459e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-22 04:27:49,052 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269350
2023-11-22 04:28:04,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1795740.0, ans=0.2
2023-11-22 04:28:17,849 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4850, loss[loss=0.09866, simple_loss=0.1283, pruned_loss=0.02535, audio_tagging_loss=0.009163, over 14671.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09574, pruned_loss=0.01577, audio_tagging_loss=0.009766, over 3048643.43 frames. ], batch size: 56, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:28:18,590 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.46 vs. limit=22.5
2023-11-22 04:28:24,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1795806.6666666667, ans=0.5
2023-11-22 04:28:40,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1795873.3333333333, ans=0.125
2023-11-22 04:28:40,909 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.35 vs. limit=22.5
2023-11-22 04:28:46,862 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.78 vs. limit=15.0
2023-11-22 04:28:47,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1795940.0, ans=0.0
2023-11-22 04:28:52,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269400
2023-11-22 04:29:21,670 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4900, loss[loss=0.07801, simple_loss=0.11, pruned_loss=0.01376, audio_tagging_loss=0.009242, over 15053.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09587, pruned_loss=0.01583, audio_tagging_loss=0.009619, over 3041749.19 frames. ], batch size: 55, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:29:30,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.40 vs. limit=12.0
2023-11-22 04:29:38,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1796206.6666666667, ans=0.1
2023-11-22 04:29:44,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1796206.6666666667, ans=0.125
2023-11-22 04:29:55,269 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.708e+01 7.981e+01 8.746e+01 9.394e+01 1.160e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 04:29:56,619 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269450
2023-11-22 04:30:26,659 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 4950, loss[loss=0.05532, simple_loss=0.07494, pruned_loss=0.007993, audio_tagging_loss=0.009859, over 14791.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.0959, pruned_loss=0.01568, audio_tagging_loss=0.009432, over 3042524.89 frames. ], batch size: 59, lr: 2.98e-03, grad_scale: 16.0
2023-11-22 04:31:01,499 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269500
2023-11-22 04:31:27,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1796740.0, ans=0.125
2023-11-22 04:31:30,703 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5000, loss[loss=0.06466, simple_loss=0.09121, pruned_loss=0.0092, audio_tagging_loss=0.009853, over 14623.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09371, pruned_loss=0.01529, audio_tagging_loss=0.009414, over 3037144.62 frames. ], batch size: 54, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:32:04,497 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.014e+01 8.549e+01 9.215e+01 1.274e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 04:32:05,853 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269550
2023-11-22 04:32:21,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1797073.3333333333, ans=0.0
2023-11-22 04:32:34,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1797140.0, ans=0.125
2023-11-22 04:32:35,488 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5050, loss[loss=0.07538, simple_loss=0.1023, pruned_loss=0.01658, audio_tagging_loss=0.007643, over 14488.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.0946, pruned_loss=0.01547, audio_tagging_loss=0.009331, over 3049879.30 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:32:38,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1797140.0, ans=0.0
2023-11-22 04:32:41,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1797140.0, ans=0.0
2023-11-22 04:32:57,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1797206.6666666667, ans=0.125
2023-11-22 04:33:08,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1797273.3333333333, ans=0.125
2023-11-22 04:33:10,989 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269600
2023-11-22 04:33:14,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1797340.0, ans=0.125
2023-11-22 04:33:25,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1797340.0, ans=0.0
2023-11-22 04:33:32,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1797406.6666666667, ans=0.0
2023-11-22 04:33:40,607 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5100, loss[loss=0.07074, simple_loss=0.1031, pruned_loss=0.01139, audio_tagging_loss=0.007789, over 14858.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09529, pruned_loss=0.01556, audio_tagging_loss=0.009239, over 3043947.55 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:33:43,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1797473.3333333333, ans=0.125
2023-11-22 04:33:56,178 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.16 vs. limit=12.0
2023-11-22 04:34:08,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=22.5
2023-11-22 04:34:14,417 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.728e+01 7.951e+01 8.483e+01 9.214e+01 1.214e+02, threshold=1.697e+02, percent-clipped=0.0
2023-11-22 04:34:15,801 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269650
2023-11-22 04:34:18,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1797673.3333333333, ans=0.2
2023-11-22 04:34:21,999 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.63 vs. limit=22.5
2023-11-22 04:34:27,439 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:34:30,316 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.58 vs. limit=22.5
2023-11-22 04:34:45,965 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5150, loss[loss=0.08381, simple_loss=0.1088, pruned_loss=0.02118, audio_tagging_loss=0.008211, over 16608.00 frames. ], tot_loss[loss=0.07301, simple_loss=0.09579, pruned_loss=0.01587, audio_tagging_loss=0.009249, over 3038931.15 frames. ], batch size: 62, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:35:20,659 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269700
2023-11-22 04:35:24,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1798006.6666666667, ans=0.5
2023-11-22 04:35:28,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1798006.6666666667, ans=0.125
2023-11-22 04:35:32,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1798006.6666666667, ans=0.125
2023-11-22 04:35:37,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1798073.3333333333, ans=0.2
2023-11-22 04:35:45,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1798073.3333333333, ans=0.125
2023-11-22 04:35:51,091 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5200, loss[loss=0.08414, simple_loss=0.1067, pruned_loss=0.01616, audio_tagging_loss=0.01462, over 14702.00 frames. ], tot_loss[loss=0.07271, simple_loss=0.09539, pruned_loss=0.01574, audio_tagging_loss=0.009267, over 3040096.78 frames. ], batch size: 53, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 04:35:55,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1798140.0, ans=0.125
2023-11-22 04:35:58,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1798140.0, ans=0.1
2023-11-22 04:36:03,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1798206.6666666667, ans=0.0
2023-11-22 04:36:11,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1798206.6666666667, ans=0.1
2023-11-22 04:36:24,910 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.234e+01 8.647e+01 9.394e+01 1.220e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 04:36:26,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269750
2023-11-22 04:36:42,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1798406.6666666667, ans=0.125
2023-11-22 04:36:47,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1798406.6666666667, ans=0.125
2023-11-22 04:36:51,873 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.73 vs. limit=12.0
2023-11-22 04:36:55,997 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5250, loss[loss=0.08295, simple_loss=0.1129, pruned_loss=0.01791, audio_tagging_loss=0.008585, over 15100.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09513, pruned_loss=0.01571, audio_tagging_loss=0.00923, over 3043544.20 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 04:37:10,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1798540.0, ans=0.125
2023-11-22 04:37:30,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269800
2023-11-22 04:37:33,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1798673.3333333333, ans=0.125
2023-11-22 04:37:56,711 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.95 vs. limit=15.0
2023-11-22 04:37:57,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1798740.0, ans=0.1
2023-11-22 04:38:00,465 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5300, loss[loss=0.05891, simple_loss=0.07004, pruned_loss=0.01162, audio_tagging_loss=0.01227, over 14870.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09535, pruned_loss=0.01574, audio_tagging_loss=0.00918, over 3043418.19 frames. ], batch size: 59, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 04:38:02,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1798806.6666666667, ans=0.125
2023-11-22 04:38:04,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten.whitening_limit, batch_count=1798806.6666666667, ans=15.0
2023-11-22 04:38:08,360 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.06 vs. limit=15.0
2023-11-22 04:38:24,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1798940.0, ans=0.125
2023-11-22 04:38:34,780 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.362e+01 8.825e+01 9.313e+01 1.444e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 04:38:34,960 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269850
2023-11-22 04:38:57,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1799073.3333333333, ans=0.125
2023-11-22 04:39:02,319 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.68 vs. limit=10.0
2023-11-22 04:39:04,231 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5350, loss[loss=0.08712, simple_loss=0.115, pruned_loss=0.02103, audio_tagging_loss=0.008604, over 14654.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09621, pruned_loss=0.01598, audio_tagging_loss=0.009178, over 3039848.88 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:39:28,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0
2023-11-22 04:39:39,527 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269900
2023-11-22 04:39:57,155 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0
2023-11-22 04:40:09,601 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5400, loss[loss=0.07791, simple_loss=0.09942, pruned_loss=0.01926, audio_tagging_loss=0.008935, over 15699.00 frames. ], tot_loss[loss=0.07344, simple_loss=0.09633, pruned_loss=0.01604, audio_tagging_loss=0.009243, over 3045966.53 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:40:16,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1799473.3333333333, ans=0.07
2023-11-22 04:40:22,508 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.66 vs. limit=15.0
2023-11-22 04:40:37,463 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.45 vs. limit=15.0
2023-11-22 04:40:43,902 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.520e+01 8.148e+01 8.900e+01 9.649e+01 1.296e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-22 04:40:44,069 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 269950
2023-11-22 04:40:51,453 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.11 vs. limit=15.0
2023-11-22 04:41:05,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1799740.0, ans=0.2
2023-11-22 04:41:13,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1799806.6666666667, ans=0.125
2023-11-22 04:41:13,920 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5450, loss[loss=0.06011, simple_loss=0.07912, pruned_loss=0.009775, audio_tagging_loss=0.01077, over 14794.00 frames. ], tot_loss[loss=0.07287, simple_loss=0.09565, pruned_loss=0.01573, audio_tagging_loss=0.00931, over 3044170.85 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:41:18,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1799806.6666666667, ans=0.1
2023-11-22 04:41:21,208 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.32 vs. limit=15.0
2023-11-22 04:41:34,476 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:41:37,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.73 vs. limit=22.5
2023-11-22 04:41:49,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270000
2023-11-22 04:42:04,093 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.83 vs. limit=15.0
2023-11-22 04:42:09,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1800073.3333333333, ans=0.125
2023-11-22 04:42:19,709 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5500, loss[loss=0.05838, simple_loss=0.07988, pruned_loss=0.01054, audio_tagging_loss=0.007904, over 14968.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09586, pruned_loss=0.01565, audio_tagging_loss=0.009372, over 3048149.39 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:42:21,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1800140.0, ans=0.0
2023-11-22 04:42:25,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1800140.0, ans=0.0
2023-11-22 04:42:33,135 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.85 vs. limit=15.0
2023-11-22 04:42:45,844 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.56 vs. limit=22.5
2023-11-22 04:42:48,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1800273.3333333333, ans=0.125
2023-11-22 04:42:53,711 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.241e+01 8.879e+01 9.729e+01 1.307e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-22 04:42:53,854 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270050
2023-11-22 04:42:58,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1800340.0, ans=0.2
2023-11-22 04:43:24,386 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5550, loss[loss=0.07055, simple_loss=0.1034, pruned_loss=0.01094, audio_tagging_loss=0.007922, over 14557.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09534, pruned_loss=0.0156, audio_tagging_loss=0.009588, over 3040767.94 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:43:28,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1800473.3333333333, ans=0.2
2023-11-22 04:43:51,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1800606.6666666667, ans=0.04949747468305833
2023-11-22 04:43:58,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.68 vs. limit=6.0
2023-11-22 04:43:59,190 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270100
2023-11-22 04:43:59,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1800606.6666666667, ans=0.125
2023-11-22 04:44:28,580 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5600, loss[loss=0.06181, simple_loss=0.07857, pruned_loss=0.01026, audio_tagging_loss=0.01226, over 16391.00 frames. ], tot_loss[loss=0.07276, simple_loss=0.09578, pruned_loss=0.01534, audio_tagging_loss=0.00954, over 3048305.00 frames. ], batch size: 61, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 04:44:37,478 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.18 vs. limit=15.0
2023-11-22 04:44:50,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1800873.3333333333, ans=0.1
2023-11-22 04:45:04,397 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.061e+01 8.801e+01 9.492e+01 1.565e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 04:45:04,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270150
2023-11-22 04:45:07,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.34 vs. limit=12.0
2023-11-22 04:45:15,356 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:45:33,028 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5650, loss[loss=0.06571, simple_loss=0.08732, pruned_loss=0.01293, audio_tagging_loss=0.009126, over 15526.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09429, pruned_loss=0.01516, audio_tagging_loss=0.009696, over 3051722.35 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:45:56,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1801206.6666666667, ans=0.125
2023-11-22 04:46:07,682 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270200
2023-11-22 04:46:07,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1801273.3333333333, ans=0.1
2023-11-22 04:46:13,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.01 vs. limit=22.5
2023-11-22 04:46:26,791 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.16 vs. limit=15.0
2023-11-22 04:46:37,708 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5700, loss[loss=0.08731, simple_loss=0.108, pruned_loss=0.02333, audio_tagging_loss=0.009997, over 15380.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09483, pruned_loss=0.01536, audio_tagging_loss=0.009594, over 3051039.05 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:46:39,829 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.84 vs. limit=12.0
2023-11-22 04:46:45,151 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.17 vs. limit=15.0
2023-11-22 04:46:48,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1801473.3333333333, ans=0.125
2023-11-22 04:46:59,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1801540.0, ans=0.2
2023-11-22 04:47:12,157 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270250
2023-11-22 04:47:13,237 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.286e+01 8.958e+01 9.545e+01 1.493e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 04:47:16,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1801673.3333333333, ans=0.125
2023-11-22 04:47:16,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1801673.3333333333, ans=0.2
2023-11-22 04:47:26,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1801673.3333333333, ans=0.1
2023-11-22 04:47:41,476 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5750, loss[loss=0.08033, simple_loss=0.1085, pruned_loss=0.0186, audio_tagging_loss=0.007471, over 15506.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.09343, pruned_loss=0.01522, audio_tagging_loss=0.009691, over 3050986.18 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:47:41,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1801806.6666666667, ans=0.125
2023-11-22 04:47:41,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1801806.6666666667, ans=0.125
2023-11-22 04:47:42,246 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.35 vs. limit=22.5
2023-11-22 04:47:49,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1801806.6666666667, ans=0.1
2023-11-22 04:47:53,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1801873.3333333333, ans=0.125
2023-11-22 04:48:00,854 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:48:12,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1801940.0, ans=0.1
2023-11-22 04:48:16,447 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270300
2023-11-22 04:48:18,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1801940.0, ans=15.0
2023-11-22 04:48:23,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1802006.6666666667, ans=0.0
2023-11-22 04:48:30,799 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:48:38,701 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:48:45,863 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5800, loss[loss=0.06483, simple_loss=0.08422, pruned_loss=0.01462, audio_tagging_loss=0.008102, over 15444.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09353, pruned_loss=0.01527, audio_tagging_loss=0.009534, over 3054404.81 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 8.0
2023-11-22 04:48:55,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1802140.0, ans=0.0
2023-11-22 04:49:18,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1802273.3333333333, ans=0.2
2023-11-22 04:49:19,291 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.50 vs. limit=22.5
2023-11-22 04:49:21,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270350
2023-11-22 04:49:21,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1802273.3333333333, ans=0.0
2023-11-22 04:49:23,354 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.122e+01 8.117e+01 8.783e+01 9.431e+01 1.385e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 04:49:23,987 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.05 vs. limit=15.0
2023-11-22 04:49:24,169 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.86 vs. limit=15.0
2023-11-22 04:49:31,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1802340.0, ans=0.125
2023-11-22 04:49:38,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1802406.6666666667, ans=0.125
2023-11-22 04:49:50,748 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5850, loss[loss=0.07247, simple_loss=0.09316, pruned_loss=0.01507, audio_tagging_loss=0.01082, over 15553.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09375, pruned_loss=0.01521, audio_tagging_loss=0.00957, over 3051161.99 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 8.0
2023-11-22 04:49:54,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1802473.3333333333, ans=0.0
2023-11-22 04:49:54,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1802473.3333333333, ans=0.125
2023-11-22 04:49:56,792 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.91 vs. limit=15.0
2023-11-22 04:50:09,275 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.11 vs. limit=12.0
2023-11-22 04:50:09,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1802540.0, ans=0.125
2023-11-22 04:50:18,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1802606.6666666667, ans=0.07
2023-11-22 04:50:19,847 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.83 vs. limit=15.0
2023-11-22 04:50:25,363 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270400
2023-11-22 04:50:28,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1802673.3333333333, ans=0.1
2023-11-22 04:50:29,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1802673.3333333333, ans=0.0
2023-11-22 04:50:55,579 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5900, loss[loss=0.0727, simple_loss=0.09941, pruned_loss=0.01479, audio_tagging_loss=0.008208, over 15909.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09422, pruned_loss=0.01516, audio_tagging_loss=0.009588, over 3054790.24 frames. ], batch size: 60, lr: 2.97e-03, grad_scale: 8.0
2023-11-22 04:51:21,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1802940.0, ans=0.2
2023-11-22 04:51:25,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1802940.0, ans=0.125
2023-11-22 04:51:30,465 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270450
2023-11-22 04:51:32,856 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.407e+01 8.248e+01 8.937e+01 9.600e+01 1.150e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 04:51:33,319 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:51:59,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1803140.0, ans=0.125
2023-11-22 04:52:00,270 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 5950, loss[loss=0.06205, simple_loss=0.07046, pruned_loss=0.01351, audio_tagging_loss=0.01331, over 15171.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.0942, pruned_loss=0.01521, audio_tagging_loss=0.009523, over 3057526.23 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 8.0
2023-11-22 04:52:10,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1803140.0, ans=0.07
2023-11-22 04:52:10,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1803140.0, ans=0.2
2023-11-22 04:52:16,417 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.44 vs. limit=15.0
2023-11-22 04:52:36,194 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270500
2023-11-22 04:52:42,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1803340.0, ans=0.125
2023-11-22 04:52:55,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1803406.6666666667, ans=0.0
2023-11-22 04:52:58,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1803406.6666666667, ans=0.2
2023-11-22 04:52:59,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1803406.6666666667, ans=0.125
2023-11-22 04:53:04,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1803473.3333333333, ans=0.0
2023-11-22 04:53:05,924 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6000, loss[loss=0.06506, simple_loss=0.0733, pruned_loss=0.01407, audio_tagging_loss=0.01434, over 13893.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09389, pruned_loss=0.01534, audio_tagging_loss=0.009593, over 3047733.88 frames. ], batch size: 53, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:53:05,927 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 04:53:38,280 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9380, 3.6823, 4.9045, 4.4823], device='cuda:0')
2023-11-22 04:53:45,801 INFO [train_asr.py:1253] (0/4) Epoch 23, validation: loss=0.05955, simple_loss=0.05175, pruned_loss=0.005139, audio_tagging_loss=0.02853, over 4681554.00 frames. 
2023-11-22 04:53:45,802 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 04:53:47,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.31 vs. limit=15.0
2023-11-22 04:53:51,568 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.06 vs. limit=15.0
2023-11-22 04:53:55,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1803473.3333333333, ans=0.1
2023-11-22 04:54:12,688 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.10 vs. limit=15.0
2023-11-22 04:54:20,425 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270550
2023-11-22 04:54:23,986 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.322e+01 8.050e+01 8.627e+01 9.368e+01 1.674e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 04:54:29,538 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.78 vs. limit=15.0
2023-11-22 04:54:32,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1803673.3333333333, ans=0.125
2023-11-22 04:54:33,809 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 04:54:45,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1803740.0, ans=0.125
2023-11-22 04:54:50,498 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6050, loss[loss=0.06106, simple_loss=0.07703, pruned_loss=0.01144, audio_tagging_loss=0.0111, over 15125.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09447, pruned_loss=0.01553, audio_tagging_loss=0.009365, over 3042913.90 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:54:58,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1803806.6666666667, ans=0.125
2023-11-22 04:55:02,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1803873.3333333333, ans=0.125
2023-11-22 04:55:21,532 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.68 vs. limit=15.0
2023-11-22 04:55:24,556 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270600
2023-11-22 04:55:26,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1803940.0, ans=0.0
2023-11-22 04:55:28,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1804006.6666666667, ans=0.1
2023-11-22 04:55:29,250 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.04 vs. limit=22.5
2023-11-22 04:55:54,019 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6100, loss[loss=0.08554, simple_loss=0.1106, pruned_loss=0.02071, audio_tagging_loss=0.009503, over 16020.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09472, pruned_loss=0.01561, audio_tagging_loss=0.009355, over 3050419.18 frames. ], batch size: 59, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:56:03,321 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.22 vs. limit=10.0
2023-11-22 04:56:06,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1804206.6666666667, ans=0.125
2023-11-22 04:56:29,380 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270650
2023-11-22 04:56:31,739 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.256e+01 8.259e+01 8.763e+01 9.618e+01 1.406e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 04:56:44,138 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.09 vs. limit=12.0
2023-11-22 04:56:56,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1804406.6666666667, ans=0.125
2023-11-22 04:56:59,494 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6150, loss[loss=0.07854, simple_loss=0.1056, pruned_loss=0.01621, audio_tagging_loss=0.009527, over 14521.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09415, pruned_loss=0.01549, audio_tagging_loss=0.009353, over 3042214.31 frames. ], batch size: 53, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:57:10,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1804473.3333333333, ans=0.0
2023-11-22 04:57:12,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1804540.0, ans=0.125
2023-11-22 04:57:14,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1804540.0, ans=0.125
2023-11-22 04:57:31,123 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.33 vs. limit=22.5
2023-11-22 04:57:34,178 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270700
2023-11-22 04:57:56,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1804740.0, ans=0.2
2023-11-22 04:58:03,975 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6200, loss[loss=0.07716, simple_loss=0.09909, pruned_loss=0.01617, audio_tagging_loss=0.01145, over 15377.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09392, pruned_loss=0.01547, audio_tagging_loss=0.009503, over 3040034.83 frames. ], batch size: 60, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:58:38,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270750
2023-11-22 04:58:40,813 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.162e+01 9.001e+01 9.547e+01 1.182e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 04:58:47,807 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 04:59:07,555 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6250, loss[loss=0.07312, simple_loss=0.09186, pruned_loss=0.01883, audio_tagging_loss=0.008362, over 14623.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09386, pruned_loss=0.01534, audio_tagging_loss=0.009529, over 3039509.01 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 04:59:13,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1805140.0, ans=0.2
2023-11-22 04:59:14,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.86 vs. limit=15.0
2023-11-22 04:59:19,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.71 vs. limit=15.0
2023-11-22 04:59:30,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1805206.6666666667, ans=0.0
2023-11-22 04:59:33,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1805273.3333333333, ans=0.2
2023-11-22 04:59:38,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1805273.3333333333, ans=0.125
2023-11-22 04:59:42,734 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270800
2023-11-22 04:59:57,565 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.94 vs. limit=15.0
2023-11-22 05:00:03,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1805406.6666666667, ans=0.0
2023-11-22 05:00:11,912 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6300, loss[loss=0.06872, simple_loss=0.08946, pruned_loss=0.01434, audio_tagging_loss=0.009651, over 14807.00 frames. ], tot_loss[loss=0.07171, simple_loss=0.09379, pruned_loss=0.0152, audio_tagging_loss=0.009618, over 3042998.43 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:00:46,743 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270850
2023-11-22 05:00:49,020 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.509e+01 8.345e+01 8.951e+01 9.677e+01 1.248e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 05:00:50,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1805673.3333333333, ans=0.125
2023-11-22 05:00:53,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1805673.3333333333, ans=0.2
2023-11-22 05:00:58,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1805673.3333333333, ans=0.125
2023-11-22 05:01:15,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1805806.6666666667, ans=0.015
2023-11-22 05:01:16,584 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6350, loss[loss=0.06699, simple_loss=0.07683, pruned_loss=0.01626, audio_tagging_loss=0.01232, over 16917.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09463, pruned_loss=0.01546, audio_tagging_loss=0.009678, over 3047047.91 frames. ], batch size: 65, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:01:27,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1805806.6666666667, ans=0.125
2023-11-22 05:01:29,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=1805873.3333333333, ans=0.1
2023-11-22 05:01:38,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1805873.3333333333, ans=0.0
2023-11-22 05:01:45,403 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.88 vs. limit=15.0
2023-11-22 05:01:50,979 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270900
2023-11-22 05:01:55,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1806006.6666666667, ans=0.125
2023-11-22 05:01:56,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.50 vs. limit=15.0
2023-11-22 05:02:13,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1806073.3333333333, ans=0.0
2023-11-22 05:02:14,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1806073.3333333333, ans=0.125
2023-11-22 05:02:17,611 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.70 vs. limit=22.5
2023-11-22 05:02:20,587 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6400, loss[loss=0.06646, simple_loss=0.08194, pruned_loss=0.0124, audio_tagging_loss=0.0131, over 13978.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09423, pruned_loss=0.01545, audio_tagging_loss=0.009749, over 3038773.02 frames. ], batch size: 53, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:02:23,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1806140.0, ans=0.0
2023-11-22 05:02:23,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1806140.0, ans=0.1
2023-11-22 05:02:51,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1806273.3333333333, ans=0.125
2023-11-22 05:02:54,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 270950
2023-11-22 05:02:57,219 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.214e+01 8.231e+01 8.800e+01 9.655e+01 1.370e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 05:02:59,012 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=22.5
2023-11-22 05:03:23,828 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6450, loss[loss=0.06252, simple_loss=0.07099, pruned_loss=0.01599, audio_tagging_loss=0.01103, over 15808.00 frames. ], tot_loss[loss=0.07296, simple_loss=0.09496, pruned_loss=0.01571, audio_tagging_loss=0.00977, over 3043995.84 frames. ], batch size: 60, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:03:36,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1806540.0, ans=0.2
2023-11-22 05:03:50,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1806606.6666666667, ans=0.0
2023-11-22 05:03:58,960 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271000
2023-11-22 05:04:28,837 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6500, loss[loss=0.06844, simple_loss=0.09399, pruned_loss=0.0158, audio_tagging_loss=0.005641, over 15879.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09467, pruned_loss=0.01563, audio_tagging_loss=0.00974, over 3048622.28 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:04:29,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1806806.6666666667, ans=0.0
2023-11-22 05:04:57,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.73 vs. limit=15.0
2023-11-22 05:05:02,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271050
2023-11-22 05:05:05,194 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.277e+01 8.344e+01 8.837e+01 9.717e+01 1.139e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 05:05:31,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1807140.0, ans=0.05
2023-11-22 05:05:32,607 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6550, loss[loss=0.07725, simple_loss=0.1072, pruned_loss=0.01678, audio_tagging_loss=0.006856, over 15417.00 frames. ], tot_loss[loss=0.07262, simple_loss=0.09454, pruned_loss=0.01571, audio_tagging_loss=0.009641, over 3048544.76 frames. ], batch size: 55, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:05:37,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1807140.0, ans=0.0
2023-11-22 05:05:45,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1807206.6666666667, ans=0.0
2023-11-22 05:05:47,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1807206.6666666667, ans=0.125
2023-11-22 05:05:50,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1807206.6666666667, ans=0.125
2023-11-22 05:06:07,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271100
2023-11-22 05:06:09,427 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.85 vs. limit=15.0
2023-11-22 05:06:16,835 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.64 vs. limit=12.0
2023-11-22 05:06:36,255 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6600, loss[loss=0.05841, simple_loss=0.07274, pruned_loss=0.01036, audio_tagging_loss=0.01168, over 14415.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.0947, pruned_loss=0.01563, audio_tagging_loss=0.009515, over 3046328.21 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:06:41,146 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.76 vs. limit=15.0
2023-11-22 05:07:11,740 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271150
2023-11-22 05:07:15,164 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.298e+01 8.931e+01 9.754e+01 1.164e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 05:07:28,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1807740.0, ans=0.2
2023-11-22 05:07:31,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1807740.0, ans=0.125
2023-11-22 05:07:40,508 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6650, loss[loss=0.07515, simple_loss=0.09754, pruned_loss=0.01656, audio_tagging_loss=0.009825, over 15257.00 frames. ], tot_loss[loss=0.07265, simple_loss=0.09482, pruned_loss=0.01573, audio_tagging_loss=0.009511, over 3039807.90 frames. ], batch size: 56, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:07:43,512 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.39 vs. limit=22.5
2023-11-22 05:07:48,017 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.97 vs. limit=22.5
2023-11-22 05:08:00,696 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.81 vs. limit=15.0
2023-11-22 05:08:15,390 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271200
2023-11-22 05:08:19,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1808006.6666666667, ans=0.125
2023-11-22 05:08:36,531 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.07 vs. limit=15.0
2023-11-22 05:08:45,703 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6700, loss[loss=0.04084, simple_loss=0.04218, pruned_loss=0.007214, audio_tagging_loss=0.01254, over 14438.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09524, pruned_loss=0.01573, audio_tagging_loss=0.009417, over 3041775.87 frames. ], batch size: 58, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:08:46,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1808140.0, ans=0.2
2023-11-22 05:09:16,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1808273.3333333333, ans=0.0
2023-11-22 05:09:20,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271250
2023-11-22 05:09:24,049 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.573e+01 8.261e+01 8.925e+01 9.693e+01 1.242e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 05:09:50,123 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6750, loss[loss=0.06006, simple_loss=0.07793, pruned_loss=0.01232, audio_tagging_loss=0.008766, over 15912.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09391, pruned_loss=0.0154, audio_tagging_loss=0.00935, over 3038834.08 frames. ], batch size: 61, lr: 2.97e-03, grad_scale: 16.0
2023-11-22 05:09:56,099 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.91 vs. limit=12.0
2023-11-22 05:09:57,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=1808473.3333333333, ans=22.5
2023-11-22 05:10:11,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1808540.0, ans=0.0
2023-11-22 05:10:18,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1808606.6666666667, ans=0.125
2023-11-22 05:10:25,994 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271300
2023-11-22 05:10:27,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1808606.6666666667, ans=0.125
2023-11-22 05:10:28,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1808673.3333333333, ans=0.1
2023-11-22 05:10:38,571 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:10:38,895 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.01 vs. limit=15.0
2023-11-22 05:10:42,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1808740.0, ans=0.1
2023-11-22 05:10:54,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.51 vs. limit=15.0
2023-11-22 05:10:54,839 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6800, loss[loss=0.05987, simple_loss=0.08192, pruned_loss=0.01105, audio_tagging_loss=0.007856, over 15363.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.0932, pruned_loss=0.01534, audio_tagging_loss=0.009447, over 3036174.33 frames. ], batch size: 57, lr: 2.97e-03, grad_scale: 32.0
2023-11-22 05:10:55,414 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.35 vs. limit=15.0
2023-11-22 05:11:08,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1808873.3333333333, ans=0.0
2023-11-22 05:11:30,434 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271350
2023-11-22 05:11:33,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1809006.6666666667, ans=0.125
2023-11-22 05:11:34,844 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.372e+01 8.044e+01 8.828e+01 9.747e+01 1.352e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 05:11:37,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1809006.6666666667, ans=10.0
2023-11-22 05:11:42,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1809006.6666666667, ans=0.2
2023-11-22 05:11:45,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1809006.6666666667, ans=0.125
2023-11-22 05:12:00,286 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6850, loss[loss=0.05024, simple_loss=0.06394, pruned_loss=0.006511, audio_tagging_loss=0.01176, over 14914.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09334, pruned_loss=0.01524, audio_tagging_loss=0.009348, over 3034434.45 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:12:08,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1809140.0, ans=0.0
2023-11-22 05:12:25,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1809273.3333333333, ans=0.2
2023-11-22 05:12:35,952 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271400
2023-11-22 05:12:36,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1809273.3333333333, ans=0.125
2023-11-22 05:12:46,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1809340.0, ans=0.125
2023-11-22 05:12:46,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1809340.0, ans=0.0
2023-11-22 05:12:58,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1809406.6666666667, ans=0.125
2023-11-22 05:13:06,061 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6900, loss[loss=0.07197, simple_loss=0.08709, pruned_loss=0.01742, audio_tagging_loss=0.011, over 14523.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09366, pruned_loss=0.01526, audio_tagging_loss=0.009261, over 3032629.00 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:13:39,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1809606.6666666667, ans=0.125
2023-11-22 05:13:41,015 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271450
2023-11-22 05:13:46,999 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.045e+01 8.623e+01 9.194e+01 1.221e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 05:13:54,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1809673.3333333333, ans=0.2
2023-11-22 05:13:55,813 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 05:14:11,290 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 6950, loss[loss=0.07775, simple_loss=0.104, pruned_loss=0.0149, audio_tagging_loss=0.01086, over 16465.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09424, pruned_loss=0.01529, audio_tagging_loss=0.009333, over 3033873.86 frames. ], batch size: 60, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:14:11,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1809806.6666666667, ans=0.2
2023-11-22 05:14:16,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1809806.6666666667, ans=0.0
2023-11-22 05:14:24,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1809873.3333333333, ans=0.2
2023-11-22 05:14:38,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1809940.0, ans=0.125
2023-11-22 05:14:46,478 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271500
2023-11-22 05:15:04,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1810073.3333333333, ans=0.125
2023-11-22 05:15:05,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1810073.3333333333, ans=10.0
2023-11-22 05:15:05,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1810073.3333333333, ans=0.07
2023-11-22 05:15:13,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.08 vs. limit=12.0
2023-11-22 05:15:15,680 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7000, loss[loss=0.08539, simple_loss=0.1052, pruned_loss=0.02309, audio_tagging_loss=0.009679, over 14774.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09319, pruned_loss=0.01515, audio_tagging_loss=0.009423, over 3036630.88 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:15:30,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff2.min_abs, batch_count=1810206.6666666667, ans=0.1
2023-11-22 05:15:35,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1810206.6666666667, ans=0.2
2023-11-22 05:15:42,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1810273.3333333333, ans=0.0
2023-11-22 05:15:50,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271550
2023-11-22 05:15:54,499 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:15:54,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.99 vs. limit=22.5
2023-11-22 05:15:55,926 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.160e+01 8.872e+01 9.665e+01 1.181e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 05:16:03,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1810340.0, ans=0.2
2023-11-22 05:16:09,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.87 vs. limit=12.0
2023-11-22 05:16:16,040 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0
2023-11-22 05:16:20,960 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7050, loss[loss=0.07108, simple_loss=0.08944, pruned_loss=0.01602, audio_tagging_loss=0.01034, over 14735.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09254, pruned_loss=0.01504, audio_tagging_loss=0.009513, over 3039883.47 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:16:24,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1810473.3333333333, ans=0.125
2023-11-22 05:16:32,117 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.46 vs. limit=22.5
2023-11-22 05:16:46,514 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.76 vs. limit=10.0
2023-11-22 05:16:49,296 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.47 vs. limit=22.5
2023-11-22 05:16:50,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1810606.6666666667, ans=0.0
2023-11-22 05:16:55,787 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271600
2023-11-22 05:16:58,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=1810673.3333333333, ans=0.05
2023-11-22 05:17:03,620 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.14 vs. limit=15.0
2023-11-22 05:17:21,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1810740.0, ans=0.125
2023-11-22 05:17:26,019 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7100, loss[loss=0.07967, simple_loss=0.1041, pruned_loss=0.01801, audio_tagging_loss=0.009603, over 15319.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09363, pruned_loss=0.01547, audio_tagging_loss=0.00951, over 3037056.69 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:17:44,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1810873.3333333333, ans=0.2
2023-11-22 05:17:56,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1810940.0, ans=0.125
2023-11-22 05:18:00,632 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271650
2023-11-22 05:18:01,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1810940.0, ans=0.0
2023-11-22 05:18:05,326 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.851e+01 8.205e+01 9.069e+01 1.012e+02 2.750e+02, threshold=1.814e+02, percent-clipped=1.0
2023-11-22 05:18:07,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1811006.6666666667, ans=0.0
2023-11-22 05:18:12,754 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.52 vs. limit=22.5
2023-11-22 05:18:30,062 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7150, loss[loss=0.0662, simple_loss=0.08964, pruned_loss=0.01295, audio_tagging_loss=0.008431, over 15510.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09409, pruned_loss=0.01546, audio_tagging_loss=0.009469, over 3038263.32 frames. ], batch size: 60, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:18:35,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1811140.0, ans=0.125
2023-11-22 05:18:44,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1811206.6666666667, ans=0.125
2023-11-22 05:19:00,786 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.19 vs. limit=15.0
2023-11-22 05:19:05,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271700
2023-11-22 05:19:10,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1811340.0, ans=0.125
2023-11-22 05:19:17,485 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.86 vs. limit=6.0
2023-11-22 05:19:21,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=1811406.6666666667, ans=0.025
2023-11-22 05:19:34,522 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7200, loss[loss=0.06136, simple_loss=0.079, pruned_loss=0.01174, audio_tagging_loss=0.01012, over 16364.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09297, pruned_loss=0.01525, audio_tagging_loss=0.009636, over 3032062.39 frames. ], batch size: 60, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:19:44,435 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.55 vs. limit=6.0
2023-11-22 05:19:45,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1811473.3333333333, ans=0.1
2023-11-22 05:19:48,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1811540.0, ans=0.07
2023-11-22 05:20:01,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1811606.6666666667, ans=0.1
2023-11-22 05:20:09,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271750
2023-11-22 05:20:14,759 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.124e+01 9.002e+01 9.849e+01 1.230e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 05:20:17,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1811673.3333333333, ans=0.1
2023-11-22 05:20:24,934 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.29 vs. limit=5.0
2023-11-22 05:20:29,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.62 vs. limit=15.0
2023-11-22 05:20:34,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1811740.0, ans=0.125
2023-11-22 05:20:40,026 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7250, loss[loss=0.08164, simple_loss=0.121, pruned_loss=0.01461, audio_tagging_loss=0.006529, over 15048.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09291, pruned_loss=0.01524, audio_tagging_loss=0.009684, over 3031726.91 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:20:46,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1811806.6666666667, ans=10.0
2023-11-22 05:20:51,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1811873.3333333333, ans=0.0
2023-11-22 05:21:14,430 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271800
2023-11-22 05:21:14,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=1811940.0, ans=10.0
2023-11-22 05:21:17,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1812006.6666666667, ans=0.1
2023-11-22 05:21:31,297 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.56 vs. limit=15.0
2023-11-22 05:21:44,766 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7300, loss[loss=0.0761, simple_loss=0.09935, pruned_loss=0.01809, audio_tagging_loss=0.008331, over 13824.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09385, pruned_loss=0.01541, audio_tagging_loss=0.009614, over 3038530.44 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:21:44,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1812140.0, ans=0.1
2023-11-22 05:21:58,656 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.11 vs. limit=22.5
2023-11-22 05:21:59,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1812206.6666666667, ans=0.125
2023-11-22 05:22:20,002 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271850
2023-11-22 05:22:25,980 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.604e+01 8.310e+01 9.010e+01 9.721e+01 2.858e+02, threshold=1.802e+02, percent-clipped=1.0
2023-11-22 05:22:37,250 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-22 05:22:38,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1812406.6666666667, ans=0.125
2023-11-22 05:22:43,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1812406.6666666667, ans=0.1
2023-11-22 05:22:49,430 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7350, loss[loss=0.07098, simple_loss=0.0973, pruned_loss=0.01505, audio_tagging_loss=0.007279, over 15615.00 frames. ], tot_loss[loss=0.07164, simple_loss=0.09388, pruned_loss=0.01526, audio_tagging_loss=0.009442, over 3047798.15 frames. ], batch size: 60, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:22:49,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1812473.3333333333, ans=0.125
2023-11-22 05:23:01,450 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.39 vs. limit=15.0
2023-11-22 05:23:25,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271900
2023-11-22 05:23:33,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1812673.3333333333, ans=0.125
2023-11-22 05:23:39,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1812673.3333333333, ans=0.0
2023-11-22 05:23:40,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1812740.0, ans=0.0
2023-11-22 05:23:54,492 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7400, loss[loss=0.09187, simple_loss=0.129, pruned_loss=0.01916, audio_tagging_loss=0.008233, over 13964.00 frames. ], tot_loss[loss=0.07153, simple_loss=0.09369, pruned_loss=0.01534, audio_tagging_loss=0.009341, over 3044980.13 frames. ], batch size: 53, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:23:54,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1812806.6666666667, ans=0.125
2023-11-22 05:24:04,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1812806.6666666667, ans=0.125
2023-11-22 05:24:05,813 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.67 vs. limit=12.0
2023-11-22 05:24:09,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1812873.3333333333, ans=0.0
2023-11-22 05:24:17,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1812873.3333333333, ans=0.0
2023-11-22 05:24:30,096 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 271950
2023-11-22 05:24:36,090 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.784e+01 7.965e+01 8.642e+01 9.211e+01 1.272e+02, threshold=1.728e+02, percent-clipped=0.0
2023-11-22 05:24:43,156 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:24:53,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1813073.3333333333, ans=0.2
2023-11-22 05:25:00,100 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7450, loss[loss=0.08744, simple_loss=0.1215, pruned_loss=0.01872, audio_tagging_loss=0.007952, over 14989.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09339, pruned_loss=0.01538, audio_tagging_loss=0.009379, over 3039792.73 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:25:07,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1813140.0, ans=0.125
2023-11-22 05:25:22,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1813206.6666666667, ans=0.0
2023-11-22 05:25:23,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1813206.6666666667, ans=10.0
2023-11-22 05:25:34,690 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272000
2023-11-22 05:25:36,195 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-272000.pt
2023-11-22 05:25:48,641 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.53 vs. limit=6.0
2023-11-22 05:25:57,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1813406.6666666667, ans=0.125
2023-11-22 05:26:03,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1813406.6666666667, ans=0.125
2023-11-22 05:26:08,235 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7500, loss[loss=0.06401, simple_loss=0.0893, pruned_loss=0.01126, audio_tagging_loss=0.008105, over 15234.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09429, pruned_loss=0.01543, audio_tagging_loss=0.009223, over 3052560.10 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:26:27,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1813540.0, ans=0.0
2023-11-22 05:26:43,635 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272050
2023-11-22 05:26:49,569 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.448e+01 8.968e+01 9.582e+01 2.186e+02, threshold=1.794e+02, percent-clipped=1.0
2023-11-22 05:26:54,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1813673.3333333333, ans=0.09899494936611666
2023-11-22 05:26:55,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1813673.3333333333, ans=0.125
2023-11-22 05:27:03,638 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.01 vs. limit=22.5
2023-11-22 05:27:13,298 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7550, loss[loss=0.07107, simple_loss=0.09175, pruned_loss=0.01561, audio_tagging_loss=0.009579, over 14677.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09519, pruned_loss=0.01569, audio_tagging_loss=0.00914, over 3056962.96 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:27:13,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1813806.6666666667, ans=0.0
2023-11-22 05:27:22,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1813806.6666666667, ans=0.125
2023-11-22 05:27:35,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1813873.3333333333, ans=0.1
2023-11-22 05:27:38,492 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.28 vs. limit=22.5
2023-11-22 05:27:48,138 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272100
2023-11-22 05:28:18,030 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7600, loss[loss=0.06086, simple_loss=0.06751, pruned_loss=0.01802, audio_tagging_loss=0.009094, over 14354.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09404, pruned_loss=0.01546, audio_tagging_loss=0.0093, over 3053698.95 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:28:21,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1814140.0, ans=0.125
2023-11-22 05:28:44,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.91 vs. limit=22.5
2023-11-22 05:28:46,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1814273.3333333333, ans=0.1
2023-11-22 05:28:51,879 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272150
2023-11-22 05:28:57,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1814340.0, ans=0.0
2023-11-22 05:28:59,152 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.478e+01 8.103e+01 8.710e+01 9.390e+01 1.351e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 05:29:20,743 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7650, loss[loss=0.05077, simple_loss=0.06637, pruned_loss=0.00839, audio_tagging_loss=0.009198, over 13518.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09324, pruned_loss=0.01533, audio_tagging_loss=0.009352, over 3043865.23 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:29:37,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1814540.0, ans=0.1
2023-11-22 05:29:49,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1814606.6666666667, ans=0.1
2023-11-22 05:29:56,437 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272200
2023-11-22 05:30:08,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1814673.3333333333, ans=0.5
2023-11-22 05:30:21,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1814740.0, ans=0.125
2023-11-22 05:30:25,870 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7700, loss[loss=0.08763, simple_loss=0.1223, pruned_loss=0.0193, audio_tagging_loss=0.007168, over 14822.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.0937, pruned_loss=0.01524, audio_tagging_loss=0.009281, over 3045349.94 frames. ], batch size: 53, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:30:33,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1814806.6666666667, ans=0.125
2023-11-22 05:30:35,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1814806.6666666667, ans=0.0
2023-11-22 05:31:01,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272250
2023-11-22 05:31:09,430 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 7.911e+01 8.690e+01 9.336e+01 1.339e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 05:31:14,753 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:31:14,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1815006.6666666667, ans=0.125
2023-11-22 05:31:25,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1815073.3333333333, ans=0.05
2023-11-22 05:31:31,253 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7750, loss[loss=0.06281, simple_loss=0.07432, pruned_loss=0.01329, audio_tagging_loss=0.01237, over 13878.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.0943, pruned_loss=0.01539, audio_tagging_loss=0.009404, over 3046155.65 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:31:46,647 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.26 vs. limit=15.0
2023-11-22 05:31:47,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1815206.6666666667, ans=0.125
2023-11-22 05:32:06,518 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272300
2023-11-22 05:32:08,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1815273.3333333333, ans=0.125
2023-11-22 05:32:16,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=1815340.0, ans=0.5
2023-11-22 05:32:36,670 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7800, loss[loss=0.09064, simple_loss=0.1159, pruned_loss=0.02479, audio_tagging_loss=0.007894, over 15253.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09428, pruned_loss=0.01539, audio_tagging_loss=0.009436, over 3045581.07 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:32:45,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.49 vs. limit=15.0
2023-11-22 05:32:50,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1815540.0, ans=0.125
2023-11-22 05:32:57,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1815540.0, ans=0.0
2023-11-22 05:33:02,209 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.84 vs. limit=15.0
2023-11-22 05:33:03,287 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.94 vs. limit=22.5
2023-11-22 05:33:09,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1815606.6666666667, ans=0.0
2023-11-22 05:33:11,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272350
2023-11-22 05:33:17,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1815673.3333333333, ans=0.09899494936611666
2023-11-22 05:33:18,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1815673.3333333333, ans=0.125
2023-11-22 05:33:19,310 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 8.265e+01 8.937e+01 9.454e+01 1.226e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 05:33:25,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1815673.3333333333, ans=0.125
2023-11-22 05:33:41,785 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7850, loss[loss=0.07171, simple_loss=0.09256, pruned_loss=0.01641, audio_tagging_loss=0.009021, over 14908.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09453, pruned_loss=0.01545, audio_tagging_loss=0.009452, over 3041920.51 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:33:47,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1815806.6666666667, ans=0.05
2023-11-22 05:33:58,876 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.50 vs. limit=15.0
2023-11-22 05:33:59,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1815873.3333333333, ans=0.0
2023-11-22 05:34:12,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1815940.0, ans=0.125
2023-11-22 05:34:16,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1815940.0, ans=0.2
2023-11-22 05:34:17,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272400
2023-11-22 05:34:35,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1816073.3333333333, ans=0.0
2023-11-22 05:34:39,399 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.66 vs. limit=15.0
2023-11-22 05:34:47,320 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7900, loss[loss=0.0807, simple_loss=0.1096, pruned_loss=0.01573, audio_tagging_loss=0.01015, over 15484.00 frames. ], tot_loss[loss=0.07325, simple_loss=0.09598, pruned_loss=0.01579, audio_tagging_loss=0.009474, over 3051926.76 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:34:49,144 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.25 vs. limit=15.0
2023-11-22 05:34:59,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1816206.6666666667, ans=0.0
2023-11-22 05:34:59,408 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.25 vs. limit=15.0
2023-11-22 05:35:01,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=1816206.6666666667, ans=15.0
2023-11-22 05:35:03,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1816206.6666666667, ans=0.2
2023-11-22 05:35:22,021 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272450
2023-11-22 05:35:30,031 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.475e+01 8.108e+01 8.831e+01 9.576e+01 1.163e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 05:35:43,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1816406.6666666667, ans=0.0
2023-11-22 05:35:51,682 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 7950, loss[loss=0.08259, simple_loss=0.107, pruned_loss=0.01855, audio_tagging_loss=0.01052, over 14323.00 frames. ], tot_loss[loss=0.07336, simple_loss=0.09579, pruned_loss=0.01585, audio_tagging_loss=0.00961, over 3047103.43 frames. ], batch size: 53, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:36:07,947 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 05:36:27,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272500
2023-11-22 05:36:33,281 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.28 vs. limit=22.5
2023-11-22 05:36:57,557 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8000, loss[loss=0.07622, simple_loss=0.1016, pruned_loss=0.016, audio_tagging_loss=0.009429, over 15910.00 frames. ], tot_loss[loss=0.07338, simple_loss=0.09565, pruned_loss=0.01586, audio_tagging_loss=0.00969, over 3046369.60 frames. ], batch size: 59, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:37:04,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1816806.6666666667, ans=0.2
2023-11-22 05:37:27,721 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.51 vs. limit=15.0
2023-11-22 05:37:32,145 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272550
2023-11-22 05:37:38,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1817006.6666666667, ans=0.2
2023-11-22 05:37:38,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1817006.6666666667, ans=0.0
2023-11-22 05:37:40,496 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.372e+01 8.294e+01 8.813e+01 9.754e+01 1.240e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 05:37:46,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1817006.6666666667, ans=0.0
2023-11-22 05:37:46,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1817006.6666666667, ans=0.125
2023-11-22 05:37:49,185 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.42 vs. limit=15.0
2023-11-22 05:38:02,652 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8050, loss[loss=0.04529, simple_loss=0.05983, pruned_loss=0.007541, audio_tagging_loss=0.007832, over 14641.00 frames. ], tot_loss[loss=0.07259, simple_loss=0.0943, pruned_loss=0.01568, audio_tagging_loss=0.009761, over 3046695.34 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:38:04,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1817140.0, ans=0.0
2023-11-22 05:38:30,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.48 vs. limit=15.0
2023-11-22 05:38:30,426 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.79 vs. limit=12.0
2023-11-22 05:38:37,299 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272600
2023-11-22 05:38:41,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1817340.0, ans=0.125
2023-11-22 05:38:48,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1817340.0, ans=0.125
2023-11-22 05:39:02,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1817406.6666666667, ans=0.0
2023-11-22 05:39:06,916 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8100, loss[loss=0.07515, simple_loss=0.104, pruned_loss=0.01338, audio_tagging_loss=0.009792, over 15450.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09451, pruned_loss=0.01567, audio_tagging_loss=0.009621, over 3046768.55 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:39:20,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1817540.0, ans=0.025
2023-11-22 05:39:22,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1817540.0, ans=0.07
2023-11-22 05:39:25,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1817540.0, ans=0.125
2023-11-22 05:39:41,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272650
2023-11-22 05:39:50,715 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.525e+01 8.265e+01 8.788e+01 9.527e+01 1.186e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 05:39:51,676 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.69 vs. limit=15.0
2023-11-22 05:40:10,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1817806.6666666667, ans=0.0
2023-11-22 05:40:11,843 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8150, loss[loss=0.07932, simple_loss=0.09505, pruned_loss=0.02068, audio_tagging_loss=0.01111, over 13875.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09498, pruned_loss=0.01577, audio_tagging_loss=0.009371, over 3053480.12 frames. ], batch size: 54, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:40:46,706 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272700
2023-11-22 05:40:51,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1818006.6666666667, ans=0.035
2023-11-22 05:40:55,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1818006.6666666667, ans=0.0
2023-11-22 05:41:13,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1818073.3333333333, ans=0.125
2023-11-22 05:41:14,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1818073.3333333333, ans=0.125
2023-11-22 05:41:16,707 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8200, loss[loss=0.09226, simple_loss=0.122, pruned_loss=0.02235, audio_tagging_loss=0.008887, over 16073.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09386, pruned_loss=0.01547, audio_tagging_loss=0.009318, over 3053809.69 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:41:16,780 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 05:41:22,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1818140.0, ans=0.1
2023-11-22 05:41:29,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1818206.6666666667, ans=0.125
2023-11-22 05:41:38,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1818206.6666666667, ans=0.1
2023-11-22 05:41:50,922 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272750
2023-11-22 05:42:00,277 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.465e+01 8.322e+01 8.892e+01 9.603e+01 1.403e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 05:42:10,164 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.74 vs. limit=15.0
2023-11-22 05:42:21,088 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8250, loss[loss=0.06947, simple_loss=0.09439, pruned_loss=0.01436, audio_tagging_loss=0.007911, over 14954.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09417, pruned_loss=0.0156, audio_tagging_loss=0.009198, over 3050834.59 frames. ], batch size: 57, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:42:34,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1818540.0, ans=0.125
2023-11-22 05:42:52,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1818606.6666666667, ans=0.0
2023-11-22 05:42:56,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272800
2023-11-22 05:42:56,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1818606.6666666667, ans=0.0
2023-11-22 05:42:59,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.51 vs. limit=22.5
2023-11-22 05:43:18,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1818740.0, ans=0.0
2023-11-22 05:43:22,077 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:43:25,821 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8300, loss[loss=0.06772, simple_loss=0.09521, pruned_loss=0.01187, audio_tagging_loss=0.008245, over 15091.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09502, pruned_loss=0.01571, audio_tagging_loss=0.009307, over 3055745.19 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:43:41,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1818873.3333333333, ans=0.125
2023-11-22 05:44:01,261 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272850
2023-11-22 05:44:09,829 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.565e+01 8.221e+01 8.844e+01 9.406e+01 1.147e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 05:44:27,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1819073.3333333333, ans=0.125
2023-11-22 05:44:30,684 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.93 vs. limit=15.0
2023-11-22 05:44:31,050 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8350, loss[loss=0.07324, simple_loss=0.08619, pruned_loss=0.01933, audio_tagging_loss=0.01081, over 15335.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09428, pruned_loss=0.01564, audio_tagging_loss=0.009238, over 3048244.45 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 16.0
2023-11-22 05:44:31,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1819140.0, ans=0.125
2023-11-22 05:44:46,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1819206.6666666667, ans=0.1
2023-11-22 05:45:04,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272900
2023-11-22 05:45:11,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1819340.0, ans=0.2
2023-11-22 05:45:26,376 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.46 vs. limit=15.0
2023-11-22 05:45:34,885 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8400, loss[loss=0.1042, simple_loss=0.1321, pruned_loss=0.02733, audio_tagging_loss=0.01078, over 14283.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09512, pruned_loss=0.01577, audio_tagging_loss=0.009243, over 3046612.22 frames. ], batch size: 53, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:46:09,212 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 272950
2023-11-22 05:46:10,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1819606.6666666667, ans=0.125
2023-11-22 05:46:17,486 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.034e+01 8.663e+01 9.140e+01 1.267e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 05:46:19,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1819673.3333333333, ans=0.125
2023-11-22 05:46:26,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.28 vs. limit=15.0
2023-11-22 05:46:28,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1819740.0, ans=0.0
2023-11-22 05:46:37,827 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8450, loss[loss=0.08099, simple_loss=0.1079, pruned_loss=0.0174, audio_tagging_loss=0.009641, over 15827.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09477, pruned_loss=0.01561, audio_tagging_loss=0.009252, over 3046395.03 frames. ], batch size: 56, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:46:38,379 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.56 vs. limit=15.0
2023-11-22 05:46:43,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1819806.6666666667, ans=0.0
2023-11-22 05:46:44,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1819806.6666666667, ans=0.0
2023-11-22 05:46:53,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1819873.3333333333, ans=0.125
2023-11-22 05:47:12,794 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273000
2023-11-22 05:47:14,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1819940.0, ans=0.2
2023-11-22 05:47:18,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1820006.6666666667, ans=0.0
2023-11-22 05:47:20,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1820006.6666666667, ans=0.125
2023-11-22 05:47:24,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1820006.6666666667, ans=0.2
2023-11-22 05:47:41,863 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8500, loss[loss=0.05312, simple_loss=0.06902, pruned_loss=0.01047, audio_tagging_loss=0.008138, over 15367.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09409, pruned_loss=0.01545, audio_tagging_loss=0.009261, over 3050194.61 frames. ], batch size: 58, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:47:59,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1820206.6666666667, ans=0.2
2023-11-22 05:48:04,956 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.24 vs. limit=22.5
2023-11-22 05:48:16,523 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273050
2023-11-22 05:48:24,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1820340.0, ans=0.0
2023-11-22 05:48:25,397 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.253e+01 8.845e+01 9.555e+01 1.261e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 05:48:25,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1820340.0, ans=0.125
2023-11-22 05:48:32,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1820406.6666666667, ans=0.2
2023-11-22 05:48:37,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1820406.6666666667, ans=0.0
2023-11-22 05:48:46,824 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8550, loss[loss=0.06668, simple_loss=0.08719, pruned_loss=0.01397, audio_tagging_loss=0.009111, over 15436.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09317, pruned_loss=0.01526, audio_tagging_loss=0.009318, over 3049740.88 frames. ], batch size: 59, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:48:52,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=1820473.3333333333, ans=15.0
2023-11-22 05:48:53,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1820473.3333333333, ans=0.1
2023-11-22 05:49:12,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1820606.6666666667, ans=0.0
2023-11-22 05:49:18,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.whiten.whitening_limit, batch_count=1820606.6666666667, ans=12.0
2023-11-22 05:49:21,354 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273100
2023-11-22 05:49:39,076 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.77 vs. limit=6.0
2023-11-22 05:49:47,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1820740.0, ans=0.125
2023-11-22 05:49:50,452 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8600, loss[loss=0.09852, simple_loss=0.1338, pruned_loss=0.02254, audio_tagging_loss=0.009099, over 14771.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09421, pruned_loss=0.01535, audio_tagging_loss=0.009343, over 3051970.95 frames. ], batch size: 55, lr: 2.96e-03, grad_scale: 32.0
2023-11-22 05:49:56,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1820806.6666666667, ans=0.0
2023-11-22 05:49:58,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1820806.6666666667, ans=0.1
2023-11-22 05:50:11,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1820873.3333333333, ans=0.07
2023-11-22 05:50:13,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1820873.3333333333, ans=0.0
2023-11-22 05:50:19,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=8.12 vs. limit=12.0
2023-11-22 05:50:25,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273150
2023-11-22 05:50:27,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1820940.0, ans=0.1
2023-11-22 05:50:32,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1821006.6666666667, ans=0.125
2023-11-22 05:50:35,544 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.265e+01 8.649e+01 9.295e+01 1.205e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 05:50:54,630 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8650, loss[loss=0.08676, simple_loss=0.1121, pruned_loss=0.02377, audio_tagging_loss=0.006939, over 15141.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09425, pruned_loss=0.01544, audio_tagging_loss=0.009357, over 3051038.59 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:51:13,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1821206.6666666667, ans=0.125
2023-11-22 05:51:29,184 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273200
2023-11-22 05:51:53,171 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.18 vs. limit=15.0
2023-11-22 05:51:59,124 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8700, loss[loss=0.05838, simple_loss=0.0737, pruned_loss=0.01128, audio_tagging_loss=0.01025, over 15011.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09394, pruned_loss=0.01542, audio_tagging_loss=0.009471, over 3045781.86 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:52:03,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1821473.3333333333, ans=0.1
2023-11-22 05:52:33,723 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273250
2023-11-22 05:52:41,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1821673.3333333333, ans=0.2
2023-11-22 05:52:43,989 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.468e+01 7.986e+01 8.723e+01 9.547e+01 1.201e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 05:53:03,596 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8750, loss[loss=0.0729, simple_loss=0.0862, pruned_loss=0.01785, audio_tagging_loss=0.01194, over 15031.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09403, pruned_loss=0.01543, audio_tagging_loss=0.009533, over 3044050.00 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:53:09,245 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.74 vs. limit=15.0
2023-11-22 05:53:22,414 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.00 vs. limit=15.0
2023-11-22 05:53:26,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1821873.3333333333, ans=0.125
2023-11-22 05:53:33,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1821940.0, ans=0.2
2023-11-22 05:53:38,798 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273300
2023-11-22 05:53:45,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.66 vs. limit=12.0
2023-11-22 05:54:07,872 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8800, loss[loss=0.09637, simple_loss=0.1297, pruned_loss=0.02186, audio_tagging_loss=0.009647, over 15448.00 frames. ], tot_loss[loss=0.07257, simple_loss=0.09488, pruned_loss=0.0155, audio_tagging_loss=0.009632, over 3048816.86 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 05:54:39,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1822273.3333333333, ans=0.2
2023-11-22 05:54:42,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273350
2023-11-22 05:54:52,326 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 8.080e+01 8.934e+01 9.573e+01 1.295e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 05:55:11,548 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8850, loss[loss=0.0695, simple_loss=0.09343, pruned_loss=0.01355, audio_tagging_loss=0.009242, over 15000.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09472, pruned_loss=0.01551, audio_tagging_loss=0.009563, over 3044900.83 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 05:55:23,830 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 05:55:38,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=1822606.6666666667, ans=0.025
2023-11-22 05:55:41,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1822606.6666666667, ans=0.125
2023-11-22 05:55:44,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1822606.6666666667, ans=0.125
2023-11-22 05:55:46,291 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273400
2023-11-22 05:56:16,873 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8900, loss[loss=0.06456, simple_loss=0.07748, pruned_loss=0.01408, audio_tagging_loss=0.01174, over 14015.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.09545, pruned_loss=0.01546, audio_tagging_loss=0.009431, over 3051295.18 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 05:56:22,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.11 vs. limit=15.0
2023-11-22 05:56:25,182 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.42 vs. limit=5.0
2023-11-22 05:56:51,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273450
2023-11-22 05:56:51,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1822940.0, ans=0.2
2023-11-22 05:56:59,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1823006.6666666667, ans=0.0
2023-11-22 05:57:03,205 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.200e+01 8.739e+01 9.306e+01 1.295e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 05:57:07,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1823073.3333333333, ans=0.0
2023-11-22 05:57:13,794 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.12 vs. limit=12.0
2023-11-22 05:57:15,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1823073.3333333333, ans=0.2
2023-11-22 05:57:15,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1823073.3333333333, ans=0.2
2023-11-22 05:57:20,953 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 8950, loss[loss=0.07596, simple_loss=0.1048, pruned_loss=0.0174, audio_tagging_loss=0.006175, over 15487.00 frames. ], tot_loss[loss=0.07317, simple_loss=0.09636, pruned_loss=0.01569, audio_tagging_loss=0.009301, over 3049660.77 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:57:36,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1823206.6666666667, ans=0.1
2023-11-22 05:57:50,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1823273.3333333333, ans=0.125
2023-11-22 05:57:56,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273500
2023-11-22 05:58:04,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1823340.0, ans=0.0
2023-11-22 05:58:14,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1823406.6666666667, ans=0.035
2023-11-22 05:58:14,613 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.05 vs. limit=15.0
2023-11-22 05:58:25,527 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9000, loss[loss=0.07044, simple_loss=0.08879, pruned_loss=0.01657, audio_tagging_loss=0.009473, over 16250.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.0955, pruned_loss=0.01552, audio_tagging_loss=0.009335, over 3055000.27 frames. ], batch size: 63, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 05:58:25,539 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 05:58:59,594 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.2803, 3.1470, 2.8999, 2.7801, 3.3511, 3.4548, 3.1706, 3.6304],
       device='cuda:0')
2023-11-22 05:59:06,010 INFO [train_asr.py:1253] (0/4) Epoch 23, validation: loss=0.06035, simple_loss=0.05169, pruned_loss=0.005137, audio_tagging_loss=0.02937, over 4681554.00 frames. 
2023-11-22 05:59:06,011 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 05:59:07,628 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 05:59:21,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1823540.0, ans=0.0
2023-11-22 05:59:32,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1823606.6666666667, ans=0.125
2023-11-22 05:59:37,461 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.35 vs. limit=15.0
2023-11-22 05:59:40,491 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273550
2023-11-22 05:59:52,546 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.165e+01 8.843e+01 9.497e+01 1.751e+02, threshold=1.769e+02, percent-clipped=1.0
2023-11-22 06:00:02,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1823740.0, ans=0.0
2023-11-22 06:00:10,311 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9050, loss[loss=0.05791, simple_loss=0.07667, pruned_loss=0.01122, audio_tagging_loss=0.008361, over 15793.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.09597, pruned_loss=0.01562, audio_tagging_loss=0.009184, over 3046841.85 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:00:12,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.77 vs. limit=15.0
2023-11-22 06:00:35,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1823940.0, ans=0.07
2023-11-22 06:00:45,207 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273600
2023-11-22 06:00:57,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1824006.6666666667, ans=0.2
2023-11-22 06:01:10,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1824073.3333333333, ans=0.125
2023-11-22 06:01:14,756 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9100, loss[loss=0.08031, simple_loss=0.1118, pruned_loss=0.01644, audio_tagging_loss=0.007972, over 15388.00 frames. ], tot_loss[loss=0.07306, simple_loss=0.0962, pruned_loss=0.01582, audio_tagging_loss=0.009145, over 3041525.69 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:01:24,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1824140.0, ans=0.125
2023-11-22 06:01:47,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1824273.3333333333, ans=0.1
2023-11-22 06:01:49,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273650
2023-11-22 06:01:57,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1824340.0, ans=0.125
2023-11-22 06:01:58,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1824340.0, ans=0.125
2023-11-22 06:02:01,431 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.950e+01 8.073e+01 8.633e+01 9.456e+01 1.130e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-22 06:02:08,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1824406.6666666667, ans=0.1
2023-11-22 06:02:10,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1824406.6666666667, ans=0.125
2023-11-22 06:02:19,618 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9150, loss[loss=0.05917, simple_loss=0.07008, pruned_loss=0.01204, audio_tagging_loss=0.01209, over 15032.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09565, pruned_loss=0.01559, audio_tagging_loss=0.009132, over 3050302.89 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:02:23,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1824473.3333333333, ans=0.2
2023-11-22 06:02:26,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1824473.3333333333, ans=0.5
2023-11-22 06:02:46,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1824606.6666666667, ans=0.1
2023-11-22 06:02:54,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273700
2023-11-22 06:03:08,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1824673.3333333333, ans=0.125
2023-11-22 06:03:14,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1824740.0, ans=0.125
2023-11-22 06:03:25,054 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9200, loss[loss=0.06066, simple_loss=0.07783, pruned_loss=0.01222, audio_tagging_loss=0.00952, over 15800.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09562, pruned_loss=0.01566, audio_tagging_loss=0.009092, over 3050019.00 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:03:40,762 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:03:59,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273750
2023-11-22 06:04:12,813 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 8.119e+01 8.781e+01 9.554e+01 1.352e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 06:04:29,597 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9250, loss[loss=0.07132, simple_loss=0.09295, pruned_loss=0.01698, audio_tagging_loss=0.007862, over 15270.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09514, pruned_loss=0.01559, audio_tagging_loss=0.009111, over 3051770.88 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:04:32,825 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.68 vs. limit=15.0
2023-11-22 06:04:53,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1825206.6666666667, ans=0.0
2023-11-22 06:05:05,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273800
2023-11-22 06:05:35,266 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9300, loss[loss=0.101, simple_loss=0.1303, pruned_loss=0.02864, audio_tagging_loss=0.007248, over 13879.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.0941, pruned_loss=0.01536, audio_tagging_loss=0.009172, over 3047355.16 frames. ], batch size: 52, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:05:39,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1825473.3333333333, ans=0.125
2023-11-22 06:06:09,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1825606.6666666667, ans=0.125
2023-11-22 06:06:11,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273850
2023-11-22 06:06:19,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1825673.3333333333, ans=0.5
2023-11-22 06:06:24,310 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.188e+01 8.737e+01 9.420e+01 1.693e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 06:06:40,824 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9350, loss[loss=0.07568, simple_loss=0.1038, pruned_loss=0.0172, audio_tagging_loss=0.006565, over 15786.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09368, pruned_loss=0.01528, audio_tagging_loss=0.009209, over 3049914.06 frames. ], batch size: 60, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:06:59,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1825873.3333333333, ans=0.125
2023-11-22 06:07:14,927 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:07:15,983 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273900
2023-11-22 06:07:19,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.06 vs. limit=12.0
2023-11-22 06:07:29,268 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.81 vs. limit=15.0
2023-11-22 06:07:39,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1826073.3333333333, ans=0.125
2023-11-22 06:07:45,866 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9400, loss[loss=0.08604, simple_loss=0.1186, pruned_loss=0.01774, audio_tagging_loss=0.008993, over 13750.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09384, pruned_loss=0.01521, audio_tagging_loss=0.009292, over 3058524.70 frames. ], batch size: 53, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:08:20,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 273950
2023-11-22 06:08:34,419 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.764e+01 8.127e+01 8.861e+01 9.548e+01 1.246e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 06:08:40,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1826406.6666666667, ans=0.0
2023-11-22 06:08:45,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1826406.6666666667, ans=0.0
2023-11-22 06:08:48,518 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:08:50,914 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9450, loss[loss=0.07516, simple_loss=0.1008, pruned_loss=0.0139, audio_tagging_loss=0.01086, over 14938.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09528, pruned_loss=0.01557, audio_tagging_loss=0.00925, over 3050866.33 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:09:05,539 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0
2023-11-22 06:09:14,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1826540.0, ans=0.1
2023-11-22 06:09:22,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1826606.6666666667, ans=0.125
2023-11-22 06:09:25,891 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274000
2023-11-22 06:09:33,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1826673.3333333333, ans=0.125
2023-11-22 06:09:34,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1826673.3333333333, ans=0.07
2023-11-22 06:09:55,615 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9500, loss[loss=0.07718, simple_loss=0.1051, pruned_loss=0.01499, audio_tagging_loss=0.009626, over 15238.00 frames. ], tot_loss[loss=0.07355, simple_loss=0.09677, pruned_loss=0.01584, audio_tagging_loss=0.009327, over 3047852.01 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:10:27,976 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.12 vs. limit=10.0
2023-11-22 06:10:31,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274050
2023-11-22 06:10:31,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1826940.0, ans=0.0
2023-11-22 06:10:39,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1827006.6666666667, ans=0.125
2023-11-22 06:10:43,864 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.476e+01 8.082e+01 8.870e+01 9.586e+01 1.219e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 06:11:01,485 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9550, loss[loss=0.05221, simple_loss=0.06238, pruned_loss=0.008323, audio_tagging_loss=0.0127, over 16067.00 frames. ], tot_loss[loss=0.07315, simple_loss=0.0957, pruned_loss=0.01578, audio_tagging_loss=0.009516, over 3052880.05 frames. ], batch size: 61, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:11:07,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1827140.0, ans=0.2
2023-11-22 06:11:17,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=1827206.6666666667, ans=0.95
2023-11-22 06:11:17,248 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:11:23,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1827206.6666666667, ans=0.125
2023-11-22 06:11:29,180 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.80 vs. limit=10.0
2023-11-22 06:11:36,390 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274100
2023-11-22 06:11:51,214 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:12:02,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1827406.6666666667, ans=0.1
2023-11-22 06:12:06,029 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9600, loss[loss=0.07974, simple_loss=0.1017, pruned_loss=0.02035, audio_tagging_loss=0.008561, over 15155.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09559, pruned_loss=0.01557, audio_tagging_loss=0.009535, over 3056067.41 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:12:27,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.82 vs. limit=15.0
2023-11-22 06:12:35,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1827606.6666666667, ans=0.0
2023-11-22 06:12:35,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1827606.6666666667, ans=0.125
2023-11-22 06:12:36,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1827606.6666666667, ans=0.125
2023-11-22 06:12:40,916 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274150
2023-11-22 06:12:46,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.74 vs. limit=15.0
2023-11-22 06:12:53,499 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.292e+01 9.125e+01 9.909e+01 1.275e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-22 06:12:56,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1827740.0, ans=0.0
2023-11-22 06:13:05,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1827740.0, ans=0.125
2023-11-22 06:13:07,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1827740.0, ans=0.1
2023-11-22 06:13:09,797 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9650, loss[loss=0.0701, simple_loss=0.0924, pruned_loss=0.0152, audio_tagging_loss=0.0087, over 16203.00 frames. ], tot_loss[loss=0.07335, simple_loss=0.09626, pruned_loss=0.01578, audio_tagging_loss=0.009434, over 3058992.42 frames. ], batch size: 61, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:13:13,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1827806.6666666667, ans=0.95
2023-11-22 06:13:31,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1827873.3333333333, ans=0.0
2023-11-22 06:13:31,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1827873.3333333333, ans=0.125
2023-11-22 06:13:40,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1827940.0, ans=0.125
2023-11-22 06:13:45,013 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274200
2023-11-22 06:14:14,377 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9700, loss[loss=0.04998, simple_loss=0.05733, pruned_loss=0.009536, audio_tagging_loss=0.01178, over 14275.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09566, pruned_loss=0.01576, audio_tagging_loss=0.009351, over 3049143.45 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:14:14,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1828140.0, ans=0.125
2023-11-22 06:14:38,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1828273.3333333333, ans=0.2
2023-11-22 06:14:49,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274250
2023-11-22 06:14:55,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1828340.0, ans=0.05
2023-11-22 06:15:01,331 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.325e+01 8.777e+01 9.483e+01 1.567e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-22 06:15:13,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1828406.6666666667, ans=0.0
2023-11-22 06:15:19,154 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9750, loss[loss=0.07474, simple_loss=0.1056, pruned_loss=0.01386, audio_tagging_loss=0.008093, over 14369.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09549, pruned_loss=0.01572, audio_tagging_loss=0.009222, over 3039626.99 frames. ], batch size: 54, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:15:23,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1828473.3333333333, ans=0.125
2023-11-22 06:15:27,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1828473.3333333333, ans=0.125
2023-11-22 06:15:33,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1828540.0, ans=0.5
2023-11-22 06:15:39,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1828540.0, ans=0.0
2023-11-22 06:15:53,758 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274300
2023-11-22 06:15:58,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.52 vs. limit=10.0
2023-11-22 06:16:23,203 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9800, loss[loss=0.05587, simple_loss=0.06736, pruned_loss=0.01236, audio_tagging_loss=0.009824, over 13780.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.0948, pruned_loss=0.01554, audio_tagging_loss=0.009221, over 3034945.88 frames. ], batch size: 53, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:16:58,288 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274350
2023-11-22 06:17:10,632 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.412e+01 8.125e+01 8.803e+01 9.670e+01 1.768e+02, threshold=1.761e+02, percent-clipped=1.0
2023-11-22 06:17:12,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1829006.6666666667, ans=0.125
2023-11-22 06:17:19,804 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:17:27,059 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9850, loss[loss=0.0581, simple_loss=0.07728, pruned_loss=0.01046, audio_tagging_loss=0.009004, over 14330.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09344, pruned_loss=0.01517, audio_tagging_loss=0.009154, over 3032989.94 frames. ], batch size: 55, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:17:47,726 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=7.70 vs. limit=12.0
2023-11-22 06:18:01,853 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274400
2023-11-22 06:18:10,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.27 vs. limit=10.0
2023-11-22 06:18:25,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff3.min_abs, batch_count=1829406.6666666667, ans=0.2
2023-11-22 06:18:31,230 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9900, loss[loss=0.05329, simple_loss=0.06356, pruned_loss=0.009513, audio_tagging_loss=0.012, over 16397.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09338, pruned_loss=0.01512, audio_tagging_loss=0.00916, over 3034494.92 frames. ], batch size: 61, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:18:33,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1829473.3333333333, ans=0.125
2023-11-22 06:19:05,861 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274450
2023-11-22 06:19:15,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1829673.3333333333, ans=0.125
2023-11-22 06:19:17,951 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 8.069e+01 8.830e+01 9.400e+01 1.279e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 06:19:35,142 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 9950, loss[loss=0.07898, simple_loss=0.1168, pruned_loss=0.01353, audio_tagging_loss=0.007024, over 15873.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09362, pruned_loss=0.0151, audio_tagging_loss=0.00918, over 3035278.82 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:19:43,659 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.16 vs. limit=15.0
2023-11-22 06:20:09,498 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274500
2023-11-22 06:20:36,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1830073.3333333333, ans=0.1
2023-11-22 06:20:39,507 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10000, loss[loss=0.07319, simple_loss=0.08773, pruned_loss=0.0185, audio_tagging_loss=0.01083, over 14765.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09324, pruned_loss=0.015, audio_tagging_loss=0.009246, over 3042068.25 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:20:39,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1830140.0, ans=0.125
2023-11-22 06:20:39,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1830140.0, ans=0.125
2023-11-22 06:21:14,534 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274550
2023-11-22 06:21:20,706 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.55 vs. limit=22.5
2023-11-22 06:21:28,400 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.149e+01 8.699e+01 9.689e+01 1.229e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 06:21:29,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1830406.6666666667, ans=0.125
2023-11-22 06:21:43,797 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10050, loss[loss=0.06239, simple_loss=0.08703, pruned_loss=0.009297, audio_tagging_loss=0.009573, over 14922.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09359, pruned_loss=0.01498, audio_tagging_loss=0.009172, over 3050413.96 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:21:58,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1830540.0, ans=0.0
2023-11-22 06:22:07,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1830540.0, ans=0.1
2023-11-22 06:22:18,433 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274600
2023-11-22 06:22:31,743 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.24 vs. limit=15.0
2023-11-22 06:22:32,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1830673.3333333333, ans=0.125
2023-11-22 06:22:33,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1830740.0, ans=0.1
2023-11-22 06:22:48,262 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10100, loss[loss=0.06586, simple_loss=0.09673, pruned_loss=0.01062, audio_tagging_loss=0.006877, over 14993.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09403, pruned_loss=0.01523, audio_tagging_loss=0.009204, over 3050443.75 frames. ], batch size: 56, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:23:10,670 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.71 vs. limit=6.0
2023-11-22 06:23:22,230 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274650
2023-11-22 06:23:34,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1831006.6666666667, ans=0.125
2023-11-22 06:23:36,112 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.350e+01 8.114e+01 8.687e+01 9.390e+01 1.135e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-22 06:23:37,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1831006.6666666667, ans=0.125
2023-11-22 06:23:39,263 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:23:44,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1831073.3333333333, ans=0.95
2023-11-22 06:23:49,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1831073.3333333333, ans=0.125
2023-11-22 06:23:51,653 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.54 vs. limit=15.0
2023-11-22 06:23:52,060 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10150, loss[loss=0.05298, simple_loss=0.06377, pruned_loss=0.0112, audio_tagging_loss=0.009893, over 15545.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09428, pruned_loss=0.0153, audio_tagging_loss=0.00932, over 3050598.77 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:23:53,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1831140.0, ans=0.0
2023-11-22 06:23:57,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1831140.0, ans=0.0
2023-11-22 06:24:19,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1831273.3333333333, ans=0.125
2023-11-22 06:24:21,536 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:24:26,635 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274700
2023-11-22 06:24:41,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1831340.0, ans=0.125
2023-11-22 06:24:56,407 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10200, loss[loss=0.0671, simple_loss=0.08292, pruned_loss=0.01083, audio_tagging_loss=0.01481, over 15286.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09417, pruned_loss=0.01528, audio_tagging_loss=0.009324, over 3055803.88 frames. ], batch size: 58, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:25:01,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1831473.3333333333, ans=0.0
2023-11-22 06:25:15,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1831540.0, ans=0.125
2023-11-22 06:25:19,848 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:25:31,356 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274750
2023-11-22 06:25:38,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1831673.3333333333, ans=0.1
2023-11-22 06:25:39,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1831673.3333333333, ans=0.0
2023-11-22 06:25:45,208 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.234e+01 8.918e+01 9.479e+01 1.480e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 06:26:00,418 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10250, loss[loss=0.09737, simple_loss=0.1328, pruned_loss=0.02426, audio_tagging_loss=0.00672, over 16098.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09453, pruned_loss=0.01529, audio_tagging_loss=0.009422, over 3062728.07 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:26:04,702 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.00 vs. limit=12.0
2023-11-22 06:26:16,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1831873.3333333333, ans=0.2
2023-11-22 06:26:35,146 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274800
2023-11-22 06:26:38,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1832006.6666666667, ans=0.1
2023-11-22 06:27:05,170 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10300, loss[loss=0.06896, simple_loss=0.09572, pruned_loss=0.01262, audio_tagging_loss=0.008477, over 15343.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09408, pruned_loss=0.01525, audio_tagging_loss=0.009504, over 3060571.29 frames. ], batch size: 55, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:27:07,047 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.79 vs. limit=15.0
2023-11-22 06:27:10,414 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:27:18,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1832206.6666666667, ans=0.0
2023-11-22 06:27:26,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1832206.6666666667, ans=0.125
2023-11-22 06:27:29,137 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.24 vs. limit=15.0
2023-11-22 06:27:36,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1832273.3333333333, ans=0.125
2023-11-22 06:27:39,353 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274850
2023-11-22 06:27:43,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1832340.0, ans=0.1
2023-11-22 06:27:52,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1832340.0, ans=0.0
2023-11-22 06:27:53,807 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.276e+01 9.014e+01 9.790e+01 1.332e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-22 06:27:55,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1832406.6666666667, ans=0.125
2023-11-22 06:27:57,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1832406.6666666667, ans=0.125
2023-11-22 06:28:03,136 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.21 vs. limit=22.5
2023-11-22 06:28:09,398 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10350, loss[loss=0.07196, simple_loss=0.08652, pruned_loss=0.01573, audio_tagging_loss=0.01297, over 14884.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09418, pruned_loss=0.01528, audio_tagging_loss=0.009594, over 3059529.22 frames. ], batch size: 57, lr: 2.95e-03, grad_scale: 16.0
2023-11-22 06:28:29,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1832540.0, ans=0.025
2023-11-22 06:28:30,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1832540.0, ans=0.125
2023-11-22 06:28:44,115 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274900
2023-11-22 06:28:58,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1832673.3333333333, ans=0.2
2023-11-22 06:29:04,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1832740.0, ans=0.125
2023-11-22 06:29:10,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1832740.0, ans=0.125
2023-11-22 06:29:13,114 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10400, loss[loss=0.09053, simple_loss=0.1174, pruned_loss=0.02473, audio_tagging_loss=0.007105, over 16065.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09464, pruned_loss=0.0154, audio_tagging_loss=0.009732, over 3053427.91 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:29:23,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1832806.6666666667, ans=0.0
2023-11-22 06:29:43,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1832940.0, ans=0.1
2023-11-22 06:29:46,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1832940.0, ans=0.0
2023-11-22 06:29:47,761 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 274950
2023-11-22 06:29:54,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1833006.6666666667, ans=0.0
2023-11-22 06:30:01,656 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.321e+01 8.235e+01 8.763e+01 9.443e+01 1.759e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 06:30:17,055 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10450, loss[loss=0.08672, simple_loss=0.1162, pruned_loss=0.02065, audio_tagging_loss=0.007983, over 15857.00 frames. ], tot_loss[loss=0.07268, simple_loss=0.09531, pruned_loss=0.01544, audio_tagging_loss=0.009587, over 3047186.01 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:30:51,973 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275000
2023-11-22 06:30:55,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1833340.0, ans=0.1
2023-11-22 06:31:03,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1833340.0, ans=0.1
2023-11-22 06:31:11,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1833406.6666666667, ans=0.125
2023-11-22 06:31:13,608 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.77 vs. limit=15.0
2023-11-22 06:31:21,971 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10500, loss[loss=0.06993, simple_loss=0.07888, pruned_loss=0.02007, audio_tagging_loss=0.01042, over 15051.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.0952, pruned_loss=0.01561, audio_tagging_loss=0.009455, over 3043170.28 frames. ], batch size: 59, lr: 2.95e-03, grad_scale: 32.0
2023-11-22 06:31:27,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1833473.3333333333, ans=0.0
2023-11-22 06:31:34,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1833540.0, ans=0.125
2023-11-22 06:31:49,462 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.78 vs. limit=15.0
2023-11-22 06:31:56,531 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275050
2023-11-22 06:32:10,959 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.531e+01 8.246e+01 8.781e+01 9.587e+01 1.309e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 06:32:19,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1833740.0, ans=0.125
2023-11-22 06:32:26,267 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10550, loss[loss=0.07088, simple_loss=0.09174, pruned_loss=0.01459, audio_tagging_loss=0.01042, over 14688.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.095, pruned_loss=0.01551, audio_tagging_loss=0.009315, over 3043734.09 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:32:43,073 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.69 vs. limit=15.0
2023-11-22 06:33:00,198 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.57 vs. limit=15.0
2023-11-22 06:33:00,752 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275100
2023-11-22 06:33:24,214 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.65 vs. limit=12.0
2023-11-22 06:33:29,412 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10600, loss[loss=0.1026, simple_loss=0.1338, pruned_loss=0.02794, audio_tagging_loss=0.007723, over 16037.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09469, pruned_loss=0.01563, audio_tagging_loss=0.009235, over 3044354.17 frames. ], batch size: 59, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:33:29,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1834140.0, ans=0.1
2023-11-22 06:34:05,859 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275150
2023-11-22 06:34:06,439 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.69 vs. limit=6.0
2023-11-22 06:34:18,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1834340.0, ans=0.0
2023-11-22 06:34:19,503 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 8.081e+01 8.805e+01 9.218e+01 1.246e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 06:34:21,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1834406.6666666667, ans=0.125
2023-11-22 06:34:30,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1834406.6666666667, ans=0.125
2023-11-22 06:34:36,099 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10650, loss[loss=0.08241, simple_loss=0.1101, pruned_loss=0.01823, audio_tagging_loss=0.009105, over 15659.00 frames. ], tot_loss[loss=0.07262, simple_loss=0.09535, pruned_loss=0.01581, audio_tagging_loss=0.009138, over 3043299.84 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:34:42,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1834473.3333333333, ans=0.2
2023-11-22 06:35:00,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1834606.6666666667, ans=0.1
2023-11-22 06:35:10,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275200
2023-11-22 06:35:17,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1834673.3333333333, ans=0.125
2023-11-22 06:35:19,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1834673.3333333333, ans=0.125
2023-11-22 06:35:33,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1834740.0, ans=0.125
2023-11-22 06:35:41,432 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10700, loss[loss=0.04607, simple_loss=0.06017, pruned_loss=0.00611, audio_tagging_loss=0.009872, over 14884.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09436, pruned_loss=0.01558, audio_tagging_loss=0.009165, over 3044855.58 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:35:54,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1834873.3333333333, ans=0.1
2023-11-22 06:35:56,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1834873.3333333333, ans=0.125
2023-11-22 06:36:09,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1834940.0, ans=0.1
2023-11-22 06:36:15,700 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275250
2023-11-22 06:36:30,358 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.298e+01 8.136e+01 8.812e+01 9.444e+01 1.567e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 06:36:40,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1835073.3333333333, ans=0.1
2023-11-22 06:36:45,214 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10750, loss[loss=0.07113, simple_loss=0.09627, pruned_loss=0.01399, audio_tagging_loss=0.008997, over 15822.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09463, pruned_loss=0.01555, audio_tagging_loss=0.009155, over 3044432.16 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:36:50,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1835140.0, ans=0.0
2023-11-22 06:36:52,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1835140.0, ans=0.1
2023-11-22 06:36:58,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=1835206.6666666667, ans=10.0
2023-11-22 06:37:01,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1835206.6666666667, ans=0.125
2023-11-22 06:37:15,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1835273.3333333333, ans=0.0
2023-11-22 06:37:21,030 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275300
2023-11-22 06:37:27,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1835340.0, ans=0.125
2023-11-22 06:37:44,545 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.66 vs. limit=22.5
2023-11-22 06:37:49,824 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10800, loss[loss=0.07077, simple_loss=0.09895, pruned_loss=0.01528, audio_tagging_loss=0.006016, over 14893.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09435, pruned_loss=0.01538, audio_tagging_loss=0.009143, over 3042007.02 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:38:12,648 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.76 vs. limit=5.0
2023-11-22 06:38:25,523 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275350
2023-11-22 06:38:33,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1835673.3333333333, ans=0.1
2023-11-22 06:38:41,118 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.004e+01 8.229e+01 8.855e+01 9.329e+01 1.142e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 06:38:44,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1835740.0, ans=0.125
2023-11-22 06:38:55,426 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.25 vs. limit=12.0
2023-11-22 06:38:56,678 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10850, loss[loss=0.07857, simple_loss=0.1004, pruned_loss=0.01633, audio_tagging_loss=0.01205, over 15608.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09412, pruned_loss=0.01542, audio_tagging_loss=0.009269, over 3041830.00 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:39:03,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1835806.6666666667, ans=0.1
2023-11-22 06:39:31,180 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275400
2023-11-22 06:39:31,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1835940.0, ans=0.125
2023-11-22 06:39:36,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1836006.6666666667, ans=0.07
2023-11-22 06:39:51,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1836073.3333333333, ans=0.04949747468305833
2023-11-22 06:39:53,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1836073.3333333333, ans=0.1
2023-11-22 06:39:54,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1836073.3333333333, ans=0.0
2023-11-22 06:39:56,760 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:40:01,516 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10900, loss[loss=0.07016, simple_loss=0.09384, pruned_loss=0.01361, audio_tagging_loss=0.009623, over 15641.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09442, pruned_loss=0.01548, audio_tagging_loss=0.009252, over 3048194.14 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:40:07,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1836140.0, ans=0.2
2023-11-22 06:40:20,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1836206.6666666667, ans=0.0
2023-11-22 06:40:23,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1836206.6666666667, ans=0.1
2023-11-22 06:40:23,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1836206.6666666667, ans=0.125
2023-11-22 06:40:33,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1836273.3333333333, ans=0.0
2023-11-22 06:40:37,341 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275450
2023-11-22 06:40:50,182 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.49 vs. limit=15.0
2023-11-22 06:40:51,650 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.279e+01 8.746e+01 9.747e+01 1.923e+02, threshold=1.749e+02, percent-clipped=1.0
2023-11-22 06:40:54,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1836406.6666666667, ans=0.125
2023-11-22 06:41:05,888 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 10950, loss[loss=0.09374, simple_loss=0.1313, pruned_loss=0.02156, audio_tagging_loss=0.006559, over 15283.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09481, pruned_loss=0.01555, audio_tagging_loss=0.009353, over 3048678.14 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:41:07,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1836473.3333333333, ans=0.015
2023-11-22 06:41:31,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1836606.6666666667, ans=0.2
2023-11-22 06:41:33,511 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.55 vs. limit=15.0
2023-11-22 06:41:38,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1836606.6666666667, ans=0.1
2023-11-22 06:41:40,368 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275500
2023-11-22 06:41:44,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1836673.3333333333, ans=0.0
2023-11-22 06:41:53,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1836673.3333333333, ans=0.0
2023-11-22 06:41:56,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1836740.0, ans=0.0
2023-11-22 06:42:09,611 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11000, loss[loss=0.07584, simple_loss=0.09906, pruned_loss=0.01929, audio_tagging_loss=0.007028, over 15677.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09414, pruned_loss=0.01545, audio_tagging_loss=0.009441, over 3052075.36 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:42:12,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1836806.6666666667, ans=0.2
2023-11-22 06:42:20,687 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:42:25,080 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.05 vs. limit=15.0
2023-11-22 06:42:31,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=1836873.3333333333, ans=0.05
2023-11-22 06:42:33,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1836873.3333333333, ans=0.025
2023-11-22 06:42:43,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1836940.0, ans=0.025
2023-11-22 06:42:44,577 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275550
2023-11-22 06:42:53,077 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.93 vs. limit=15.0
2023-11-22 06:42:53,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1837006.6666666667, ans=0.1
2023-11-22 06:43:00,978 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 8.356e+01 8.870e+01 9.515e+01 1.517e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 06:43:04,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1837073.3333333333, ans=0.1
2023-11-22 06:43:14,638 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11050, loss[loss=0.08995, simple_loss=0.1106, pruned_loss=0.02291, audio_tagging_loss=0.01176, over 14229.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09408, pruned_loss=0.01549, audio_tagging_loss=0.009505, over 3047070.89 frames. ], batch size: 53, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:43:14,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1837140.0, ans=0.125
2023-11-22 06:43:17,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1837140.0, ans=0.0
2023-11-22 06:43:22,842 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.59 vs. limit=15.0
2023-11-22 06:43:24,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1837140.0, ans=0.125
2023-11-22 06:43:47,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1837273.3333333333, ans=0.025
2023-11-22 06:43:48,997 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275600
2023-11-22 06:43:49,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1837273.3333333333, ans=0.125
2023-11-22 06:44:00,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1837340.0, ans=0.0
2023-11-22 06:44:12,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1837406.6666666667, ans=0.5
2023-11-22 06:44:18,976 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11100, loss[loss=0.06763, simple_loss=0.08084, pruned_loss=0.01473, audio_tagging_loss=0.01248, over 14843.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09378, pruned_loss=0.01545, audio_tagging_loss=0.009709, over 3044654.55 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:44:28,849 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0
2023-11-22 06:44:42,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1837540.0, ans=0.0
2023-11-22 06:44:53,794 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275650
2023-11-22 06:44:56,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1837673.3333333333, ans=0.0
2023-11-22 06:45:03,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1837673.3333333333, ans=0.0
2023-11-22 06:45:11,536 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.680e+01 8.196e+01 8.803e+01 9.419e+01 1.192e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 06:45:15,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1837740.0, ans=0.04949747468305833
2023-11-22 06:45:23,233 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11150, loss[loss=0.07674, simple_loss=0.09933, pruned_loss=0.01567, audio_tagging_loss=0.0114, over 15027.00 frames. ], tot_loss[loss=0.07245, simple_loss=0.09424, pruned_loss=0.01558, audio_tagging_loss=0.009748, over 3051619.77 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 8.0
2023-11-22 06:45:45,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.21 vs. limit=15.0
2023-11-22 06:45:58,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275700
2023-11-22 06:46:05,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1838006.6666666667, ans=10.0
2023-11-22 06:46:05,429 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.31 vs. limit=15.0
2023-11-22 06:46:24,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1838073.3333333333, ans=0.125
2023-11-22 06:46:24,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1838073.3333333333, ans=0.125
2023-11-22 06:46:28,377 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11200, loss[loss=0.09789, simple_loss=0.1336, pruned_loss=0.02445, audio_tagging_loss=0.006622, over 15618.00 frames. ], tot_loss[loss=0.07219, simple_loss=0.09382, pruned_loss=0.01544, audio_tagging_loss=0.009841, over 3051586.29 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:47:02,982 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275750
2023-11-22 06:47:13,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1838340.0, ans=0.125
2023-11-22 06:47:21,148 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.197e+01 8.985e+01 9.663e+01 1.096e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 06:47:25,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1838406.6666666667, ans=0.125
2023-11-22 06:47:32,694 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11250, loss[loss=0.09631, simple_loss=0.1253, pruned_loss=0.02548, audio_tagging_loss=0.008162, over 16077.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09413, pruned_loss=0.01559, audio_tagging_loss=0.009778, over 3048307.07 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:47:39,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1838473.3333333333, ans=0.0
2023-11-22 06:47:58,660 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.20 vs. limit=15.0
2023-11-22 06:48:07,778 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275800
2023-11-22 06:48:24,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1838740.0, ans=0.2
2023-11-22 06:48:25,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1838740.0, ans=0.0
2023-11-22 06:48:30,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1838740.0, ans=0.1
2023-11-22 06:48:35,017 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.13 vs. limit=15.0
2023-11-22 06:48:38,274 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11300, loss[loss=0.09589, simple_loss=0.1229, pruned_loss=0.02686, audio_tagging_loss=0.007564, over 15444.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09401, pruned_loss=0.01544, audio_tagging_loss=0.009609, over 3043872.83 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:48:45,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1838806.6666666667, ans=0.0
2023-11-22 06:49:03,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1838940.0, ans=0.1
2023-11-22 06:49:13,697 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275850
2023-11-22 06:49:15,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1838940.0, ans=0.0
2023-11-22 06:49:25,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=1839006.6666666667, ans=22.5
2023-11-22 06:49:31,029 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 7.924e+01 8.561e+01 9.574e+01 1.338e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-22 06:49:42,806 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11350, loss[loss=0.07675, simple_loss=0.09909, pruned_loss=0.01772, audio_tagging_loss=0.009484, over 15702.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.09395, pruned_loss=0.01546, audio_tagging_loss=0.009481, over 3047674.08 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:50:01,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1839206.6666666667, ans=0.1
2023-11-22 06:50:16,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1839273.3333333333, ans=0.125
2023-11-22 06:50:17,595 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275900
2023-11-22 06:50:28,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1839340.0, ans=0.0
2023-11-22 06:50:42,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1839406.6666666667, ans=0.04949747468305833
2023-11-22 06:50:47,677 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11400, loss[loss=0.07209, simple_loss=0.09447, pruned_loss=0.01679, audio_tagging_loss=0.008075, over 15575.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09343, pruned_loss=0.01533, audio_tagging_loss=0.009444, over 3042905.68 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:50:57,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1839473.3333333333, ans=0.0
2023-11-22 06:51:18,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1839606.6666666667, ans=0.2
2023-11-22 06:51:22,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 275950
2023-11-22 06:51:27,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1839673.3333333333, ans=0.125
2023-11-22 06:51:27,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1839673.3333333333, ans=0.0
2023-11-22 06:51:28,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1839673.3333333333, ans=0.0
2023-11-22 06:51:40,309 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.209e+01 8.167e+01 8.873e+01 9.595e+01 1.109e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 06:51:51,930 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11450, loss[loss=0.06271, simple_loss=0.07497, pruned_loss=0.01664, audio_tagging_loss=0.008581, over 15512.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09345, pruned_loss=0.01548, audio_tagging_loss=0.009487, over 3057230.43 frames. ], batch size: 60, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:51:59,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1839806.6666666667, ans=0.125
2023-11-22 06:52:05,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1839873.3333333333, ans=0.1
2023-11-22 06:52:07,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1839873.3333333333, ans=0.0
2023-11-22 06:52:18,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.66 vs. limit=22.5
2023-11-22 06:52:20,692 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.37 vs. limit=10.0
2023-11-22 06:52:27,319 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276000
2023-11-22 06:52:28,772 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-276000.pt
2023-11-22 06:52:59,302 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.10 vs. limit=15.0
2023-11-22 06:52:59,788 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11500, loss[loss=0.06815, simple_loss=0.08799, pruned_loss=0.01619, audio_tagging_loss=0.007958, over 15394.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.0935, pruned_loss=0.01551, audio_tagging_loss=0.009553, over 3054455.44 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:53:01,635 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.02 vs. limit=15.0
2023-11-22 06:53:05,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1840140.0, ans=0.125
2023-11-22 06:53:08,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1840140.0, ans=0.125
2023-11-22 06:53:11,566 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.83 vs. limit=6.0
2023-11-22 06:53:26,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1840273.3333333333, ans=0.1
2023-11-22 06:53:30,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1840273.3333333333, ans=0.1
2023-11-22 06:53:34,651 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276050
2023-11-22 06:53:38,927 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.18 vs. limit=12.0
2023-11-22 06:53:52,664 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.162e+01 8.710e+01 9.515e+01 1.226e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 06:54:03,888 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11550, loss[loss=0.08059, simple_loss=0.1137, pruned_loss=0.01529, audio_tagging_loss=0.008437, over 15424.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09348, pruned_loss=0.01548, audio_tagging_loss=0.009557, over 3051285.41 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:54:28,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1840540.0, ans=0.2
2023-11-22 06:54:28,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1840540.0, ans=0.125
2023-11-22 06:54:39,923 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276100
2023-11-22 06:54:40,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1840606.6666666667, ans=0.1
2023-11-22 06:54:44,939 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 06:54:50,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1840673.3333333333, ans=0.1
2023-11-22 06:54:52,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1840673.3333333333, ans=0.125
2023-11-22 06:54:53,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1840673.3333333333, ans=0.0
2023-11-22 06:55:09,740 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11600, loss[loss=0.07854, simple_loss=0.099, pruned_loss=0.0205, audio_tagging_loss=0.008544, over 15321.00 frames. ], tot_loss[loss=0.07237, simple_loss=0.09431, pruned_loss=0.01566, audio_tagging_loss=0.00955, over 3055319.32 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:55:09,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1840806.6666666667, ans=0.0
2023-11-22 06:55:11,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1840806.6666666667, ans=0.07
2023-11-22 06:55:19,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1840806.6666666667, ans=0.0
2023-11-22 06:55:33,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1840873.3333333333, ans=0.0
2023-11-22 06:55:37,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1840940.0, ans=0.1
2023-11-22 06:55:38,739 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.88 vs. limit=22.5
2023-11-22 06:55:44,371 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276150
2023-11-22 06:55:50,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1841006.6666666667, ans=0.0
2023-11-22 06:55:52,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.33 vs. limit=15.0
2023-11-22 06:56:03,213 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.255e+01 8.370e+01 8.963e+01 9.598e+01 1.244e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-22 06:56:15,236 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11650, loss[loss=0.04967, simple_loss=0.06345, pruned_loss=0.008574, audio_tagging_loss=0.00937, over 15023.00 frames. ], tot_loss[loss=0.07265, simple_loss=0.09476, pruned_loss=0.01581, audio_tagging_loss=0.009463, over 3050269.50 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:56:15,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.59 vs. limit=15.0
2023-11-22 06:56:27,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1841206.6666666667, ans=0.05
2023-11-22 06:56:37,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1841206.6666666667, ans=0.125
2023-11-22 06:56:46,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1841273.3333333333, ans=0.1
2023-11-22 06:56:50,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276200
2023-11-22 06:57:04,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1841340.0, ans=0.125
2023-11-22 06:57:05,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1841340.0, ans=0.07
2023-11-22 06:57:12,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1841406.6666666667, ans=0.0
2023-11-22 06:57:19,044 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11700, loss[loss=0.06823, simple_loss=0.0879, pruned_loss=0.01447, audio_tagging_loss=0.009811, over 14567.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09309, pruned_loss=0.01542, audio_tagging_loss=0.009494, over 3046567.62 frames. ], batch size: 55, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:57:21,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1841473.3333333333, ans=0.0
2023-11-22 06:57:44,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1841606.6666666667, ans=0.0
2023-11-22 06:57:54,711 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276250
2023-11-22 06:58:11,821 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.571e+01 8.445e+01 9.108e+01 1.005e+02 1.343e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-22 06:58:16,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1841740.0, ans=0.0
2023-11-22 06:58:24,079 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11750, loss[loss=0.0846, simple_loss=0.12, pruned_loss=0.01697, audio_tagging_loss=0.007635, over 16437.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09434, pruned_loss=0.0156, audio_tagging_loss=0.009446, over 3050583.89 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 06:58:24,386 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 06:58:54,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1841940.0, ans=0.125
2023-11-22 06:58:58,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276300
2023-11-22 06:59:28,434 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11800, loss[loss=0.07949, simple_loss=0.09659, pruned_loss=0.02052, audio_tagging_loss=0.01068, over 15138.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.0943, pruned_loss=0.0156, audio_tagging_loss=0.009531, over 3049656.38 frames. ], batch size: 57, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 06:59:36,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1842140.0, ans=0.0
2023-11-22 06:59:56,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1842273.3333333333, ans=0.125
2023-11-22 06:59:57,062 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.49 vs. limit=15.0
2023-11-22 07:00:03,988 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276350
2023-11-22 07:00:19,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1842406.6666666667, ans=0.1
2023-11-22 07:00:24,076 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.151e+01 8.673e+01 9.167e+01 1.144e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-22 07:00:31,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1842406.6666666667, ans=0.0
2023-11-22 07:00:34,091 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11850, loss[loss=0.085, simple_loss=0.113, pruned_loss=0.01876, audio_tagging_loss=0.009711, over 14129.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09432, pruned_loss=0.01554, audio_tagging_loss=0.009583, over 3045510.28 frames. ], batch size: 52, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 07:00:36,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1842473.3333333333, ans=0.0
2023-11-22 07:01:08,926 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276400
2023-11-22 07:01:17,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1842673.3333333333, ans=0.125
2023-11-22 07:01:23,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1842673.3333333333, ans=0.2
2023-11-22 07:01:31,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1842740.0, ans=0.0
2023-11-22 07:01:38,741 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11900, loss[loss=0.07552, simple_loss=0.0981, pruned_loss=0.01483, audio_tagging_loss=0.01164, over 15843.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09424, pruned_loss=0.01544, audio_tagging_loss=0.009597, over 3051269.23 frames. ], batch size: 58, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 07:01:38,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1842806.6666666667, ans=0.0
2023-11-22 07:01:42,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1842806.6666666667, ans=0.125
2023-11-22 07:02:13,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.53 vs. limit=10.0
2023-11-22 07:02:14,063 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276450
2023-11-22 07:02:33,090 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.503e+01 7.969e+01 8.417e+01 9.242e+01 1.141e+02, threshold=1.683e+02, percent-clipped=0.0
2023-11-22 07:02:43,987 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 11950, loss[loss=0.08288, simple_loss=0.106, pruned_loss=0.02036, audio_tagging_loss=0.009506, over 15486.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09409, pruned_loss=0.01554, audio_tagging_loss=0.009659, over 3047931.60 frames. ], batch size: 56, lr: 2.94e-03, grad_scale: 16.0
2023-11-22 07:02:46,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1843140.0, ans=0.125
2023-11-22 07:02:52,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1843140.0, ans=0.1
2023-11-22 07:02:56,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1843206.6666666667, ans=0.125
2023-11-22 07:03:02,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1843206.6666666667, ans=0.125
2023-11-22 07:03:17,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1843273.3333333333, ans=0.125
2023-11-22 07:03:17,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.73 vs. limit=15.0
2023-11-22 07:03:18,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276500
2023-11-22 07:03:46,913 INFO [train_asr.py:1221] (0/4) Epoch 23, batch 12000, loss[loss=0.06516, simple_loss=0.08595, pruned_loss=0.01309, audio_tagging_loss=0.009092, over 15777.00 frames. ], tot_loss[loss=0.07267, simple_loss=0.09476, pruned_loss=0.01568, audio_tagging_loss=0.009614, over 3049587.42 frames. ], batch size: 60, lr: 2.94e-03, grad_scale: 32.0
2023-11-22 07:03:46,917 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 07:04:07,300 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.5913, 3.0308, 3.1660, 2.8184], device='cuda:0')
2023-11-22 07:04:27,935 INFO [train_asr.py:1253] (0/4) Epoch 23, validation: loss=0.05966, simple_loss=0.05174, pruned_loss=0.005186, audio_tagging_loss=0.02861, over 4681554.00 frames. 
2023-11-22 07:04:27,936 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 07:04:28,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1843473.3333333333, ans=0.2
2023-11-22 07:04:43,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1843540.0, ans=0.0
2023-11-22 07:04:57,490 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-23.pt
2023-11-22 07:05:34,213 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 0, loss[loss=0.09603, simple_loss=0.1125, pruned_loss=0.02142, audio_tagging_loss=0.01836, over 15312.00 frames. ], tot_loss[loss=0.09603, simple_loss=0.1125, pruned_loss=0.02142, audio_tagging_loss=0.01836, over 15312.00 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:05:34,229 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 07:05:55,839 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.6677, 1.5414, 3.5284, 3.2034, 2.9628, 3.2315, 2.8539, 3.3483],
       device='cuda:0')
2023-11-22 07:05:58,724 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.6067, 3.6334, 3.8823, 3.4121], device='cuda:0')
2023-11-22 07:06:05,937 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.2381, 4.1977, 4.4157, 4.4300], device='cuda:0')
2023-11-22 07:06:09,683 INFO [train_asr.py:1253] (0/4) Epoch 24, validation: loss=0.05907, simple_loss=0.05179, pruned_loss=0.005258, audio_tagging_loss=0.02792, over 4681554.00 frames. 
2023-11-22 07:06:09,684 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 07:06:13,385 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276550
2023-11-22 07:06:13,660 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:06:32,176 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.629e+01 8.350e+01 8.889e+01 9.652e+01 1.254e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 07:06:41,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1843766.6666666667, ans=0.2
2023-11-22 07:06:43,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1843766.6666666667, ans=0.125
2023-11-22 07:07:02,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1843900.0, ans=0.0
2023-11-22 07:07:13,675 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 50, loss[loss=0.07952, simple_loss=0.09075, pruned_loss=0.01653, audio_tagging_loss=0.01762, over 15761.00 frames. ], tot_loss[loss=0.08087, simple_loss=0.09552, pruned_loss=0.01548, audio_tagging_loss=0.01763, over 694031.36 frames. ], batch size: 61, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:07:16,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1843966.6666666667, ans=0.125
2023-11-22 07:07:17,381 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276600
2023-11-22 07:07:19,401 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.86 vs. limit=15.0
2023-11-22 07:08:13,952 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:08:17,323 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 100, loss[loss=0.08253, simple_loss=0.1071, pruned_loss=0.01897, audio_tagging_loss=0.009995, over 16141.00 frames. ], tot_loss[loss=0.08087, simple_loss=0.09621, pruned_loss=0.01589, audio_tagging_loss=0.01688, over 1216298.38 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:08:21,164 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276650
2023-11-22 07:08:21,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1844300.0, ans=0.125
2023-11-22 07:08:41,005 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.235e+01 8.891e+01 9.359e+01 9.974e+01 1.363e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-22 07:08:42,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1844433.3333333333, ans=10.0
2023-11-22 07:08:45,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1844433.3333333333, ans=0.125
2023-11-22 07:09:02,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1844500.0, ans=0.125
2023-11-22 07:09:22,246 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 150, loss[loss=0.08024, simple_loss=0.1002, pruned_loss=0.02094, audio_tagging_loss=0.009205, over 15809.00 frames. ], tot_loss[loss=0.07797, simple_loss=0.09478, pruned_loss=0.0153, audio_tagging_loss=0.01528, over 1628953.89 frames. ], batch size: 61, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:09:26,053 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276700
2023-11-22 07:09:47,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1844766.6666666667, ans=0.125
2023-11-22 07:09:53,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1844766.6666666667, ans=0.0
2023-11-22 07:09:58,782 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.28 vs. limit=15.0
2023-11-22 07:09:59,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1844833.3333333333, ans=0.0
2023-11-22 07:10:02,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1844833.3333333333, ans=0.1
2023-11-22 07:10:04,312 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.12 vs. limit=22.5
2023-11-22 07:10:09,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1844833.3333333333, ans=0.0
2023-11-22 07:10:12,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1844833.3333333333, ans=0.125
2023-11-22 07:10:15,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1844900.0, ans=0.125
2023-11-22 07:10:27,716 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 200, loss[loss=0.08432, simple_loss=0.1099, pruned_loss=0.01996, audio_tagging_loss=0.009419, over 15534.00 frames. ], tot_loss[loss=0.07662, simple_loss=0.09523, pruned_loss=0.01539, audio_tagging_loss=0.01362, over 1946664.08 frames. ], batch size: 61, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:10:31,551 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276750
2023-11-22 07:10:51,664 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.378e+01 9.056e+01 9.976e+01 1.254e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 07:10:52,680 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.95 vs. limit=15.0
2023-11-22 07:10:54,767 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.58 vs. limit=22.5
2023-11-22 07:11:04,146 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.39 vs. limit=15.0
2023-11-22 07:11:10,293 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.03 vs. limit=15.0
2023-11-22 07:11:19,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1845233.3333333333, ans=0.125
2023-11-22 07:11:23,442 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.61 vs. limit=22.5
2023-11-22 07:11:31,389 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 250, loss[loss=0.07671, simple_loss=0.1059, pruned_loss=0.01553, audio_tagging_loss=0.008255, over 14880.00 frames. ], tot_loss[loss=0.0758, simple_loss=0.09578, pruned_loss=0.01563, audio_tagging_loss=0.01228, over 2193852.39 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:11:35,026 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276800
2023-11-22 07:11:41,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1845300.0, ans=0.0
2023-11-22 07:12:27,358 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=12.0
2023-11-22 07:12:35,792 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:12:36,701 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 300, loss[loss=0.07301, simple_loss=0.09688, pruned_loss=0.01734, audio_tagging_loss=0.007227, over 15189.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09499, pruned_loss=0.01549, audio_tagging_loss=0.01142, over 2383219.72 frames. ], batch size: 60, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:12:39,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1845633.3333333333, ans=0.125
2023-11-22 07:12:40,508 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276850
2023-11-22 07:12:40,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1845633.3333333333, ans=0.125
2023-11-22 07:12:43,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1845633.3333333333, ans=0.125
2023-11-22 07:12:47,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1845700.0, ans=0.025
2023-11-22 07:12:51,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1845700.0, ans=0.0
2023-11-22 07:12:58,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1845700.0, ans=0.0
2023-11-22 07:13:00,925 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.553e+01 8.200e+01 8.925e+01 9.618e+01 1.197e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 07:13:06,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1845766.6666666667, ans=0.125
2023-11-22 07:13:08,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1845766.6666666667, ans=0.07
2023-11-22 07:13:09,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1845766.6666666667, ans=0.125
2023-11-22 07:13:39,445 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.59 vs. limit=12.0
2023-11-22 07:13:39,918 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 350, loss[loss=0.07016, simple_loss=0.08958, pruned_loss=0.01385, audio_tagging_loss=0.01151, over 14784.00 frames. ], tot_loss[loss=0.07357, simple_loss=0.09495, pruned_loss=0.01535, audio_tagging_loss=0.01074, over 2533548.24 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:13:42,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1845966.6666666667, ans=0.1
2023-11-22 07:13:44,760 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276900
2023-11-22 07:13:46,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1845966.6666666667, ans=0.125
2023-11-22 07:13:52,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1846033.3333333333, ans=0.125
2023-11-22 07:14:03,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1846033.3333333333, ans=0.125
2023-11-22 07:14:13,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1846100.0, ans=0.0
2023-11-22 07:14:20,759 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:14:27,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1846166.6666666667, ans=0.125
2023-11-22 07:14:28,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1846166.6666666667, ans=0.1
2023-11-22 07:14:31,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1846233.3333333333, ans=0.0
2023-11-22 07:14:37,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1846233.3333333333, ans=0.2
2023-11-22 07:14:40,523 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.95 vs. limit=15.0
2023-11-22 07:14:44,728 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 400, loss[loss=0.06011, simple_loss=0.07692, pruned_loss=0.01385, audio_tagging_loss=0.007801, over 14818.00 frames. ], tot_loss[loss=0.07369, simple_loss=0.09541, pruned_loss=0.01567, audio_tagging_loss=0.01032, over 2642412.83 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:14:48,661 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 276950
2023-11-22 07:15:02,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1846366.6666666667, ans=0.0
2023-11-22 07:15:06,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1846366.6666666667, ans=0.0
2023-11-22 07:15:09,172 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 7.931e+01 8.582e+01 9.377e+01 1.207e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-22 07:15:10,673 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:15:49,043 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 450, loss[loss=0.08407, simple_loss=0.1152, pruned_loss=0.01914, audio_tagging_loss=0.007317, over 15986.00 frames. ], tot_loss[loss=0.0734, simple_loss=0.09524, pruned_loss=0.01575, audio_tagging_loss=0.01003, over 2726766.38 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:15:49,604 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.40 vs. limit=6.0
2023-11-22 07:15:53,336 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277000
2023-11-22 07:15:55,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.66 vs. limit=12.0
2023-11-22 07:16:11,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1846700.0, ans=0.0
2023-11-22 07:16:37,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1846833.3333333333, ans=0.125
2023-11-22 07:16:47,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1846900.0, ans=0.0
2023-11-22 07:16:53,444 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 500, loss[loss=0.095, simple_loss=0.1387, pruned_loss=0.0193, audio_tagging_loss=0.006363, over 14594.00 frames. ], tot_loss[loss=0.07328, simple_loss=0.09538, pruned_loss=0.01575, audio_tagging_loss=0.009832, over 2792886.52 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:16:55,635 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.32 vs. limit=15.0
2023-11-22 07:16:57,927 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277050
2023-11-22 07:17:14,067 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.24 vs. limit=15.0
2023-11-22 07:17:18,054 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.239e+01 8.246e+01 8.888e+01 9.960e+01 1.317e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 07:17:18,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1847100.0, ans=0.125
2023-11-22 07:17:29,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1847100.0, ans=0.0
2023-11-22 07:17:38,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1847166.6666666667, ans=0.2
2023-11-22 07:17:39,803 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.61 vs. limit=22.5
2023-11-22 07:17:44,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1847233.3333333333, ans=0.0
2023-11-22 07:17:58,828 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 550, loss[loss=0.09646, simple_loss=0.1342, pruned_loss=0.02398, audio_tagging_loss=0.005365, over 15509.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09501, pruned_loss=0.01567, audio_tagging_loss=0.009691, over 2850249.95 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:18:00,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1847300.0, ans=0.125
2023-11-22 07:18:02,581 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277100
2023-11-22 07:18:37,055 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.17 vs. limit=15.0
2023-11-22 07:19:02,826 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 600, loss[loss=0.07349, simple_loss=0.08678, pruned_loss=0.01873, audio_tagging_loss=0.01138, over 15348.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09446, pruned_loss=0.01545, audio_tagging_loss=0.009619, over 2890025.27 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:19:07,284 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277150
2023-11-22 07:19:09,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1847633.3333333333, ans=0.1
2023-11-22 07:19:23,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1847700.0, ans=0.125
2023-11-22 07:19:27,314 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.641e+01 8.220e+01 8.869e+01 9.857e+01 1.255e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 07:19:28,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1847766.6666666667, ans=0.0
2023-11-22 07:19:56,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1847900.0, ans=0.0
2023-11-22 07:20:07,410 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 650, loss[loss=0.06295, simple_loss=0.08313, pruned_loss=0.01304, audio_tagging_loss=0.008344, over 14588.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.095, pruned_loss=0.01551, audio_tagging_loss=0.009531, over 2922536.27 frames. ], batch size: 54, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:20:11,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277200
2023-11-22 07:20:27,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1848033.3333333333, ans=0.2
2023-11-22 07:20:36,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1848100.0, ans=0.125
2023-11-22 07:20:58,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1848233.3333333333, ans=0.09899494936611666
2023-11-22 07:21:11,154 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 700, loss[loss=0.07975, simple_loss=0.1095, pruned_loss=0.01733, audio_tagging_loss=0.007672, over 14700.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09487, pruned_loss=0.01538, audio_tagging_loss=0.009527, over 2957008.33 frames. ], batch size: 54, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:21:15,057 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277250
2023-11-22 07:21:19,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.70 vs. limit=15.0
2023-11-22 07:21:22,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1848300.0, ans=0.125
2023-11-22 07:21:37,780 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.366e+01 8.189e+01 8.629e+01 9.381e+01 1.193e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 07:21:42,420 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.13 vs. limit=12.0
2023-11-22 07:21:49,756 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.96 vs. limit=22.5
2023-11-22 07:22:04,769 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.08 vs. limit=10.0
2023-11-22 07:22:16,044 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 750, loss[loss=0.07624, simple_loss=0.09403, pruned_loss=0.01816, audio_tagging_loss=0.01106, over 15029.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.09462, pruned_loss=0.01539, audio_tagging_loss=0.009593, over 2985940.68 frames. ], batch size: 54, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:22:20,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277300
2023-11-22 07:22:31,547 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.10 vs. limit=15.0
2023-11-22 07:22:33,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1848700.0, ans=0.2
2023-11-22 07:22:38,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1848700.0, ans=0.0
2023-11-22 07:22:48,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1848766.6666666667, ans=0.125
2023-11-22 07:22:57,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1848833.3333333333, ans=0.125
2023-11-22 07:23:09,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1848900.0, ans=0.125
2023-11-22 07:23:14,455 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.77 vs. limit=15.0
2023-11-22 07:23:21,089 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 800, loss[loss=0.06784, simple_loss=0.09261, pruned_loss=0.01368, audio_tagging_loss=0.007853, over 15210.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.0951, pruned_loss=0.01545, audio_tagging_loss=0.009596, over 3001688.30 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:23:24,855 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277350
2023-11-22 07:23:27,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1848966.6666666667, ans=0.09899494936611666
2023-11-22 07:23:39,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.76 vs. limit=6.0
2023-11-22 07:23:46,994 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.163e+01 8.225e+01 8.846e+01 9.560e+01 1.170e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 07:23:48,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1849100.0, ans=0.125
2023-11-22 07:23:51,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1849100.0, ans=0.2
2023-11-22 07:23:52,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1849100.0, ans=0.1
2023-11-22 07:24:06,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1849166.6666666667, ans=0.125
2023-11-22 07:24:07,129 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.18 vs. limit=5.0
2023-11-22 07:24:25,005 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 850, loss[loss=0.08398, simple_loss=0.1129, pruned_loss=0.01825, audio_tagging_loss=0.009301, over 15256.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09607, pruned_loss=0.01575, audio_tagging_loss=0.009662, over 3013513.25 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:24:26,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1849300.0, ans=0.0
2023-11-22 07:24:28,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277400
2023-11-22 07:24:34,705 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.72 vs. limit=15.0
2023-11-22 07:24:55,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1849433.3333333333, ans=0.1
2023-11-22 07:25:01,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1849433.3333333333, ans=0.125
2023-11-22 07:25:25,047 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.25 vs. limit=10.0
2023-11-22 07:25:28,307 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.88 vs. limit=22.5
2023-11-22 07:25:28,877 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 900, loss[loss=0.06419, simple_loss=0.08116, pruned_loss=0.01212, audio_tagging_loss=0.01148, over 14197.00 frames. ], tot_loss[loss=0.07273, simple_loss=0.09502, pruned_loss=0.01547, audio_tagging_loss=0.00975, over 3013876.61 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:25:33,300 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277450
2023-11-22 07:25:56,249 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.215e+01 8.871e+01 9.558e+01 1.650e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 07:26:26,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1849900.0, ans=0.0
2023-11-22 07:26:33,279 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 950, loss[loss=0.0847, simple_loss=0.1139, pruned_loss=0.01999, audio_tagging_loss=0.00778, over 15092.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.09617, pruned_loss=0.01575, audio_tagging_loss=0.009631, over 3026552.81 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:26:37,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277500
2023-11-22 07:26:47,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1850033.3333333333, ans=0.125
2023-11-22 07:26:58,906 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.49 vs. limit=15.0
2023-11-22 07:27:00,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.55 vs. limit=10.0
2023-11-22 07:27:09,003 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=15.0
2023-11-22 07:27:36,858 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1000, loss[loss=0.08877, simple_loss=0.1256, pruned_loss=0.01891, audio_tagging_loss=0.007085, over 16185.00 frames. ], tot_loss[loss=0.07316, simple_loss=0.09554, pruned_loss=0.01585, audio_tagging_loss=0.009542, over 3026712.15 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:27:40,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277550
2023-11-22 07:27:54,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=12.45 vs. limit=15.0
2023-11-22 07:27:57,949 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.57 vs. limit=22.5
2023-11-22 07:28:04,361 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.720e+01 8.008e+01 8.714e+01 9.298e+01 1.223e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 07:28:04,475 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:28:24,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1850500.0, ans=0.125
2023-11-22 07:28:24,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1850500.0, ans=0.2
2023-11-22 07:28:27,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1850566.6666666667, ans=0.125
2023-11-22 07:28:40,595 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1050, loss[loss=0.06173, simple_loss=0.08254, pruned_loss=0.01091, audio_tagging_loss=0.009551, over 15449.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09493, pruned_loss=0.01561, audio_tagging_loss=0.009454, over 3019876.45 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:28:40,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1850633.3333333333, ans=0.015
2023-11-22 07:28:41,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1850633.3333333333, ans=0.2
2023-11-22 07:28:44,421 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277600
2023-11-22 07:29:03,087 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0
2023-11-22 07:29:18,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1850766.6666666667, ans=0.2
2023-11-22 07:29:20,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1850833.3333333333, ans=0.125
2023-11-22 07:29:35,724 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=15.0
2023-11-22 07:29:42,952 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.44 vs. limit=22.5
2023-11-22 07:29:46,041 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1100, loss[loss=0.0666, simple_loss=0.09295, pruned_loss=0.01231, audio_tagging_loss=0.00782, over 14548.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09425, pruned_loss=0.01537, audio_tagging_loss=0.009356, over 3021977.40 frames. ], batch size: 54, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:29:49,681 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:29:49,773 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277650
2023-11-22 07:30:11,994 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.373e+01 8.181e+01 8.784e+01 9.508e+01 1.137e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 07:30:18,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1851100.0, ans=0.125
2023-11-22 07:30:50,579 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1150, loss[loss=0.06882, simple_loss=0.0879, pruned_loss=0.01677, audio_tagging_loss=0.008106, over 16776.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09356, pruned_loss=0.01514, audio_tagging_loss=0.009287, over 3025693.59 frames. ], batch size: 63, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:30:54,376 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277700
2023-11-22 07:30:56,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1851300.0, ans=0.125
2023-11-22 07:31:06,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1851366.6666666667, ans=0.2
2023-11-22 07:31:08,583 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.61 vs. limit=15.0
2023-11-22 07:31:12,659 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.34 vs. limit=22.5
2023-11-22 07:31:24,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1851433.3333333333, ans=0.125
2023-11-22 07:31:33,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1851500.0, ans=0.09899494936611666
2023-11-22 07:31:33,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1851500.0, ans=0.125
2023-11-22 07:31:54,997 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1200, loss[loss=0.07379, simple_loss=0.09227, pruned_loss=0.01871, audio_tagging_loss=0.008941, over 15084.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09305, pruned_loss=0.01503, audio_tagging_loss=0.009359, over 3036663.65 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:31:58,765 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277750
2023-11-22 07:32:22,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1851766.6666666667, ans=0.0
2023-11-22 07:32:23,095 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.729e+01 8.243e+01 8.919e+01 9.746e+01 1.203e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 07:32:46,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1851900.0, ans=0.125
2023-11-22 07:32:54,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1851900.0, ans=0.125
2023-11-22 07:33:00,929 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1250, loss[loss=0.07428, simple_loss=0.1008, pruned_loss=0.01366, audio_tagging_loss=0.01023, over 15673.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09316, pruned_loss=0.015, audio_tagging_loss=0.009281, over 3037478.13 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:33:04,924 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277800
2023-11-22 07:33:06,170 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.63 vs. limit=15.0
2023-11-22 07:33:22,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.21 vs. limit=6.0
2023-11-22 07:34:00,408 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=15.0
2023-11-22 07:34:06,763 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1300, loss[loss=0.07221, simple_loss=0.09429, pruned_loss=0.01655, audio_tagging_loss=0.008516, over 14508.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09295, pruned_loss=0.01494, audio_tagging_loss=0.009237, over 3037712.34 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:34:10,730 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277850
2023-11-22 07:34:17,674 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.70 vs. limit=15.0
2023-11-22 07:34:20,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1852366.6666666667, ans=0.1
2023-11-22 07:34:29,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1852366.6666666667, ans=0.1
2023-11-22 07:34:33,412 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.580e+01 8.077e+01 8.549e+01 9.237e+01 1.175e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 07:34:55,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1852500.0, ans=0.1
2023-11-22 07:34:55,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1852500.0, ans=0.125
2023-11-22 07:35:00,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1852566.6666666667, ans=0.2
2023-11-22 07:35:06,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.93 vs. limit=10.0
2023-11-22 07:35:11,088 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1350, loss[loss=0.05872, simple_loss=0.07305, pruned_loss=0.01138, audio_tagging_loss=0.01082, over 16508.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09266, pruned_loss=0.01485, audio_tagging_loss=0.00926, over 3043565.84 frames. ], batch size: 63, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:35:14,728 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277900
2023-11-22 07:35:18,782 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.83 vs. limit=15.0
2023-11-22 07:35:25,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1852700.0, ans=0.2
2023-11-22 07:35:28,857 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.92 vs. limit=15.0
2023-11-22 07:35:35,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1852766.6666666667, ans=0.0
2023-11-22 07:35:48,252 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.44 vs. limit=22.5
2023-11-22 07:35:57,570 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:35:59,426 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.16 vs. limit=15.0
2023-11-22 07:36:16,021 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1400, loss[loss=0.05821, simple_loss=0.07748, pruned_loss=0.0102, audio_tagging_loss=0.009278, over 15077.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09257, pruned_loss=0.01499, audio_tagging_loss=0.009327, over 3044900.81 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:36:19,201 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.24 vs. limit=15.0
2023-11-22 07:36:19,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 277950
2023-11-22 07:36:33,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1853033.3333333333, ans=0.0
2023-11-22 07:36:42,784 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.691e+01 8.253e+01 8.948e+01 9.652e+01 1.188e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 07:36:47,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1853100.0, ans=0.0
2023-11-22 07:37:05,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1853166.6666666667, ans=0.125
2023-11-22 07:37:08,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1853233.3333333333, ans=0.0
2023-11-22 07:37:20,257 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1450, loss[loss=0.05953, simple_loss=0.08429, pruned_loss=0.009787, audio_tagging_loss=0.0076, over 14970.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09259, pruned_loss=0.01498, audio_tagging_loss=0.009384, over 3046405.64 frames. ], batch size: 55, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:37:22,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1853300.0, ans=0.125
2023-11-22 07:37:24,569 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278000
2023-11-22 07:37:44,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1853433.3333333333, ans=0.0
2023-11-22 07:37:49,603 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=15.0
2023-11-22 07:37:52,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1853433.3333333333, ans=0.0
2023-11-22 07:38:24,850 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1500, loss[loss=0.07531, simple_loss=0.09043, pruned_loss=0.0188, audio_tagging_loss=0.01129, over 15088.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09271, pruned_loss=0.01524, audio_tagging_loss=0.009469, over 3042529.57 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:38:28,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278050
2023-11-22 07:38:43,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1853700.0, ans=0.125
2023-11-22 07:38:52,924 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 8.167e+01 8.667e+01 9.503e+01 1.230e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 07:39:00,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1853766.6666666667, ans=0.125
2023-11-22 07:39:05,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1853833.3333333333, ans=0.1
2023-11-22 07:39:06,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1853833.3333333333, ans=0.125
2023-11-22 07:39:13,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1853833.3333333333, ans=0.1
2023-11-22 07:39:29,707 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1550, loss[loss=0.07416, simple_loss=0.09501, pruned_loss=0.01548, audio_tagging_loss=0.01117, over 14711.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09374, pruned_loss=0.01551, audio_tagging_loss=0.009502, over 3041348.71 frames. ], batch size: 56, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:39:34,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278100
2023-11-22 07:39:44,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.47 vs. limit=15.0
2023-11-22 07:39:47,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.70 vs. limit=22.5
2023-11-22 07:40:01,838 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.57 vs. limit=15.0
2023-11-22 07:40:25,429 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.42 vs. limit=22.5
2023-11-22 07:40:34,421 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1600, loss[loss=0.08106, simple_loss=0.1143, pruned_loss=0.01611, audio_tagging_loss=0.007775, over 16241.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09442, pruned_loss=0.01562, audio_tagging_loss=0.00953, over 3041498.34 frames. ], batch size: 58, lr: 2.87e-03, grad_scale: 32.0
2023-11-22 07:40:38,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278150
2023-11-22 07:40:40,655 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.59 vs. limit=15.0
2023-11-22 07:40:48,652 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.50 vs. limit=15.0
2023-11-22 07:41:00,694 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.21 vs. limit=22.5
2023-11-22 07:41:04,080 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.580e+01 8.136e+01 8.791e+01 9.603e+01 1.214e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 07:41:11,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1854433.3333333333, ans=0.125
2023-11-22 07:41:39,080 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1650, loss[loss=0.06739, simple_loss=0.09275, pruned_loss=0.01287, audio_tagging_loss=0.008142, over 15440.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09451, pruned_loss=0.0155, audio_tagging_loss=0.009554, over 3040982.47 frames. ], batch size: 60, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:41:42,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278200
2023-11-22 07:41:47,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1854633.3333333333, ans=0.125
2023-11-22 07:41:54,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1854700.0, ans=0.0
2023-11-22 07:42:10,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.76 vs. limit=22.5
2023-11-22 07:42:15,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1854766.6666666667, ans=0.125
2023-11-22 07:42:29,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1854900.0, ans=0.0
2023-11-22 07:42:34,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1854900.0, ans=10.0
2023-11-22 07:42:43,469 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1700, loss[loss=0.0608, simple_loss=0.07203, pruned_loss=0.01273, audio_tagging_loss=0.01206, over 15098.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09474, pruned_loss=0.01552, audio_tagging_loss=0.009509, over 3041448.28 frames. ], batch size: 57, lr: 2.87e-03, grad_scale: 16.0
2023-11-22 07:42:44,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1854966.6666666667, ans=0.125
2023-11-22 07:42:47,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278250
2023-11-22 07:42:51,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1854966.6666666667, ans=0.0
2023-11-22 07:43:03,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1855033.3333333333, ans=0.1
2023-11-22 07:43:13,672 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 8.170e+01 8.792e+01 9.443e+01 1.155e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 07:43:35,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1855233.3333333333, ans=0.1
2023-11-22 07:43:48,093 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1750, loss[loss=0.05193, simple_loss=0.06434, pruned_loss=0.009526, audio_tagging_loss=0.01023, over 14034.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09423, pruned_loss=0.01543, audio_tagging_loss=0.009496, over 3041808.35 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:43:51,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278300
2023-11-22 07:43:55,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1855300.0, ans=0.5
2023-11-22 07:43:55,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1855300.0, ans=0.1
2023-11-22 07:44:19,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1855433.3333333333, ans=0.125
2023-11-22 07:44:19,628 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.11 vs. limit=15.0
2023-11-22 07:44:40,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1855566.6666666667, ans=0.05
2023-11-22 07:44:50,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1855566.6666666667, ans=0.1
2023-11-22 07:44:52,588 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1800, loss[loss=0.05421, simple_loss=0.07279, pruned_loss=0.008826, audio_tagging_loss=0.008991, over 15009.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09371, pruned_loss=0.0152, audio_tagging_loss=0.009334, over 3044440.56 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:44:54,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1855633.3333333333, ans=0.125
2023-11-22 07:44:56,960 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278350
2023-11-22 07:45:12,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1855700.0, ans=0.125
2023-11-22 07:45:20,601 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.72 vs. limit=22.5
2023-11-22 07:45:22,555 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.603e+01 7.991e+01 8.555e+01 9.443e+01 1.169e+02, threshold=1.711e+02, percent-clipped=0.0
2023-11-22 07:45:23,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=1855766.6666666667, ans=10.0
2023-11-22 07:45:57,029 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1850, loss[loss=0.07661, simple_loss=0.1072, pruned_loss=0.01761, audio_tagging_loss=0.00537, over 15226.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09431, pruned_loss=0.0155, audio_tagging_loss=0.009328, over 3044618.32 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:46:01,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278400
2023-11-22 07:46:06,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1855966.6666666667, ans=0.0
2023-11-22 07:46:16,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1856033.3333333333, ans=0.125
2023-11-22 07:46:18,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1856033.3333333333, ans=0.04949747468305833
2023-11-22 07:46:22,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1856100.0, ans=0.0
2023-11-22 07:46:43,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1856166.6666666667, ans=0.125
2023-11-22 07:46:48,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=1856233.3333333333, ans=10.0
2023-11-22 07:46:59,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1856233.3333333333, ans=0.0
2023-11-22 07:47:02,773 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1900, loss[loss=0.07096, simple_loss=0.09454, pruned_loss=0.017, audio_tagging_loss=0.006686, over 15500.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09394, pruned_loss=0.01542, audio_tagging_loss=0.009235, over 3045919.82 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:47:06,542 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278450
2023-11-22 07:47:32,030 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 7.925e+01 8.678e+01 9.471e+01 1.078e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-22 07:47:32,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1856433.3333333333, ans=0.125
2023-11-22 07:47:47,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1856500.0, ans=0.0
2023-11-22 07:47:47,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1856500.0, ans=0.0
2023-11-22 07:47:53,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1856566.6666666667, ans=0.0
2023-11-22 07:47:57,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1856566.6666666667, ans=0.2
2023-11-22 07:48:00,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1856566.6666666667, ans=0.04949747468305833
2023-11-22 07:48:01,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.03 vs. limit=10.0
2023-11-22 07:48:07,449 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 1950, loss[loss=0.08566, simple_loss=0.1186, pruned_loss=0.01786, audio_tagging_loss=0.00848, over 15381.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.09428, pruned_loss=0.01556, audio_tagging_loss=0.009169, over 3041838.80 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:48:11,224 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278500
2023-11-22 07:48:13,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1856633.3333333333, ans=0.125
2023-11-22 07:48:13,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1856633.3333333333, ans=0.0
2023-11-22 07:48:47,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1856833.3333333333, ans=0.0
2023-11-22 07:48:57,161 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.18 vs. limit=10.0
2023-11-22 07:49:11,999 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2000, loss[loss=0.07588, simple_loss=0.0961, pruned_loss=0.02091, audio_tagging_loss=0.006922, over 14120.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09348, pruned_loss=0.01544, audio_tagging_loss=0.009328, over 3030074.21 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:49:16,453 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278550
2023-11-22 07:49:23,283 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 07:49:33,421 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.87 vs. limit=15.0
2023-11-22 07:49:34,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=15.0
2023-11-22 07:49:41,981 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.370e+01 9.075e+01 9.910e+01 1.919e+02, threshold=1.815e+02, percent-clipped=1.0
2023-11-22 07:49:57,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1857166.6666666667, ans=0.125
2023-11-22 07:50:06,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1857233.3333333333, ans=0.125
2023-11-22 07:50:09,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1857233.3333333333, ans=0.125
2023-11-22 07:50:17,554 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2050, loss[loss=0.04599, simple_loss=0.0586, pruned_loss=0.007368, audio_tagging_loss=0.009319, over 13963.00 frames. ], tot_loss[loss=0.07148, simple_loss=0.09351, pruned_loss=0.01537, audio_tagging_loss=0.009347, over 3027565.11 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:50:21,384 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278600
2023-11-22 07:50:31,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1857366.6666666667, ans=0.125
2023-11-22 07:50:37,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.55 vs. limit=22.5
2023-11-22 07:50:43,947 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.74 vs. limit=15.0
2023-11-22 07:50:44,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1857433.3333333333, ans=0.125
2023-11-22 07:50:46,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=1857433.3333333333, ans=15.0
2023-11-22 07:51:06,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1857500.0, ans=0.0
2023-11-22 07:51:16,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1857566.6666666667, ans=0.125
2023-11-22 07:51:23,222 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2100, loss[loss=0.06154, simple_loss=0.0847, pruned_loss=0.01099, audio_tagging_loss=0.008191, over 15133.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09397, pruned_loss=0.01545, audio_tagging_loss=0.009401, over 3033540.83 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:51:27,006 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278650
2023-11-22 07:51:33,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1857633.3333333333, ans=0.0
2023-11-22 07:51:50,341 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.64 vs. limit=15.0
2023-11-22 07:51:52,690 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.224e+01 8.868e+01 9.706e+01 1.360e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 07:51:57,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1857766.6666666667, ans=0.125
2023-11-22 07:51:58,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1857766.6666666667, ans=0.0
2023-11-22 07:52:23,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1857900.0, ans=0.0
2023-11-22 07:52:26,029 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2150, loss[loss=0.0766, simple_loss=0.102, pruned_loss=0.01727, audio_tagging_loss=0.008316, over 14636.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09337, pruned_loss=0.0153, audio_tagging_loss=0.009347, over 3033637.46 frames. ], batch size: 53, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:52:29,753 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278700
2023-11-22 07:52:48,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.23 vs. limit=6.0
2023-11-22 07:52:49,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1858033.3333333333, ans=0.04949747468305833
2023-11-22 07:52:56,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1858100.0, ans=0.2
2023-11-22 07:53:02,627 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.38 vs. limit=22.5
2023-11-22 07:53:06,689 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:53:26,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1858233.3333333333, ans=0.2
2023-11-22 07:53:31,938 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2200, loss[loss=0.0906, simple_loss=0.1256, pruned_loss=0.02095, audio_tagging_loss=0.006839, over 15670.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09372, pruned_loss=0.01556, audio_tagging_loss=0.009409, over 3039018.87 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:53:35,725 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278750
2023-11-22 07:53:47,831 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.10 vs. limit=15.0
2023-11-22 07:53:48,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1858366.6666666667, ans=0.0
2023-11-22 07:54:00,459 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.169e+01 8.248e+01 8.861e+01 9.671e+01 1.420e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 07:54:11,428 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.30 vs. limit=15.0
2023-11-22 07:54:21,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1858500.0, ans=0.125
2023-11-22 07:54:36,108 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2250, loss[loss=0.05959, simple_loss=0.08224, pruned_loss=0.009982, audio_tagging_loss=0.008487, over 15573.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09499, pruned_loss=0.01575, audio_tagging_loss=0.009341, over 3046319.04 frames. ], batch size: 60, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:54:38,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1858633.3333333333, ans=0.2
2023-11-22 07:54:39,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278800
2023-11-22 07:54:45,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1858633.3333333333, ans=0.2
2023-11-22 07:54:51,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1858700.0, ans=0.125
2023-11-22 07:55:09,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1858766.6666666667, ans=0.0
2023-11-22 07:55:09,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.74 vs. limit=6.0
2023-11-22 07:55:24,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1858833.3333333333, ans=0.125
2023-11-22 07:55:39,806 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2300, loss[loss=0.05839, simple_loss=0.07855, pruned_loss=0.01009, audio_tagging_loss=0.009029, over 14545.00 frames. ], tot_loss[loss=0.07199, simple_loss=0.09412, pruned_loss=0.01552, audio_tagging_loss=0.009408, over 3038404.44 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:55:43,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278850
2023-11-22 07:56:02,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1859033.3333333333, ans=0.125
2023-11-22 07:56:09,722 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.839e+01 8.100e+01 8.737e+01 9.435e+01 1.101e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 07:56:37,710 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 07:56:45,051 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2350, loss[loss=0.08376, simple_loss=0.1119, pruned_loss=0.01995, audio_tagging_loss=0.007878, over 14970.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09411, pruned_loss=0.01542, audio_tagging_loss=0.00951, over 3039748.80 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 07:56:49,586 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278900
2023-11-22 07:56:51,557 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.56 vs. limit=6.0
2023-11-22 07:56:59,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1859366.6666666667, ans=0.125
2023-11-22 07:57:36,176 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.30 vs. limit=6.0
2023-11-22 07:57:47,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1859566.6666666667, ans=0.1
2023-11-22 07:57:49,787 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2400, loss[loss=0.07322, simple_loss=0.1066, pruned_loss=0.01236, audio_tagging_loss=0.007543, over 16209.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09498, pruned_loss=0.01556, audio_tagging_loss=0.00948, over 3048073.03 frames. ], batch size: 59, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:57:53,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 278950
2023-11-22 07:58:19,862 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.551e+01 8.282e+01 8.837e+01 9.594e+01 1.331e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 07:58:29,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.50 vs. limit=15.0
2023-11-22 07:58:33,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1859833.3333333333, ans=0.125
2023-11-22 07:58:53,813 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2450, loss[loss=0.07937, simple_loss=0.103, pruned_loss=0.01844, audio_tagging_loss=0.009443, over 15617.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09494, pruned_loss=0.01557, audio_tagging_loss=0.00966, over 3045333.65 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:58:54,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1859966.6666666667, ans=0.0
2023-11-22 07:58:57,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279000
2023-11-22 07:59:09,146 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.26 vs. limit=15.0
2023-11-22 07:59:16,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=1860033.3333333333, ans=0.02
2023-11-22 07:59:25,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=1860100.0, ans=0.125
2023-11-22 07:59:28,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1860100.0, ans=0.1
2023-11-22 07:59:56,319 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.72 vs. limit=6.0
2023-11-22 07:59:58,149 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2500, loss[loss=0.06257, simple_loss=0.07986, pruned_loss=0.01207, audio_tagging_loss=0.01058, over 14749.00 frames. ], tot_loss[loss=0.0726, simple_loss=0.09457, pruned_loss=0.01562, audio_tagging_loss=0.009693, over 3043456.96 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 07:59:59,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1860300.0, ans=0.125
2023-11-22 08:00:02,488 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279050
2023-11-22 08:00:02,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1860300.0, ans=0.125
2023-11-22 08:00:27,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1860433.3333333333, ans=0.125
2023-11-22 08:00:29,446 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.591e+01 8.264e+01 8.898e+01 9.687e+01 1.183e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-22 08:00:32,613 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.78 vs. limit=22.5
2023-11-22 08:00:51,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1860566.6666666667, ans=0.125
2023-11-22 08:01:03,364 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2550, loss[loss=0.06311, simple_loss=0.07392, pruned_loss=0.01209, audio_tagging_loss=0.01406, over 15377.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09439, pruned_loss=0.01554, audio_tagging_loss=0.009626, over 3032885.97 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:01:03,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1860633.3333333333, ans=0.0
2023-11-22 08:01:07,194 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279100
2023-11-22 08:01:11,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1860633.3333333333, ans=0.0
2023-11-22 08:01:16,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.14 vs. limit=15.0
2023-11-22 08:01:18,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1860700.0, ans=0.125
2023-11-22 08:01:24,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1860700.0, ans=0.1
2023-11-22 08:02:00,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1860900.0, ans=0.1
2023-11-22 08:02:04,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1860900.0, ans=0.125
2023-11-22 08:02:05,720 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.97 vs. limit=10.0
2023-11-22 08:02:06,707 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:02:09,030 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2600, loss[loss=0.06081, simple_loss=0.08356, pruned_loss=0.0125, audio_tagging_loss=0.00653, over 14668.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09309, pruned_loss=0.01522, audio_tagging_loss=0.009562, over 3035802.43 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:02:10,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1860966.6666666667, ans=0.125
2023-11-22 08:02:12,924 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279150
2023-11-22 08:02:25,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1861033.3333333333, ans=0.125
2023-11-22 08:02:32,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1861033.3333333333, ans=0.09899494936611666
2023-11-22 08:02:39,910 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.406e+01 8.247e+01 8.743e+01 9.488e+01 1.623e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 08:02:46,190 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.54 vs. limit=22.5
2023-11-22 08:02:58,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1861166.6666666667, ans=0.125
2023-11-22 08:03:08,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1861233.3333333333, ans=0.1
2023-11-22 08:03:12,774 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2650, loss[loss=0.08269, simple_loss=0.1073, pruned_loss=0.01989, audio_tagging_loss=0.00913, over 15462.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09332, pruned_loss=0.01525, audio_tagging_loss=0.009455, over 3032061.31 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:03:14,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1861300.0, ans=0.125
2023-11-22 08:03:17,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279200
2023-11-22 08:03:19,405 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.03 vs. limit=22.5
2023-11-22 08:03:23,341 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.23 vs. limit=10.0
2023-11-22 08:03:39,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1861433.3333333333, ans=0.125
2023-11-22 08:03:45,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.95 vs. limit=10.0
2023-11-22 08:04:18,504 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2700, loss[loss=0.07401, simple_loss=0.09148, pruned_loss=0.01826, audio_tagging_loss=0.01001, over 14628.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09337, pruned_loss=0.01524, audio_tagging_loss=0.009332, over 3038738.82 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 08:04:18,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1861633.3333333333, ans=0.0
2023-11-22 08:04:21,208 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:04:22,248 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279250
2023-11-22 08:04:34,931 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.71 vs. limit=15.0
2023-11-22 08:04:49,547 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:04:50,484 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.791e+01 7.998e+01 8.544e+01 9.286e+01 1.150e+02, threshold=1.709e+02, percent-clipped=0.0
2023-11-22 08:04:54,205 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.56 vs. limit=10.0
2023-11-22 08:04:55,623 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.38 vs. limit=15.0
2023-11-22 08:05:23,839 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2750, loss[loss=0.07406, simple_loss=0.1022, pruned_loss=0.01367, audio_tagging_loss=0.0093, over 15603.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09257, pruned_loss=0.01508, audio_tagging_loss=0.00941, over 3044588.45 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 16.0
2023-11-22 08:05:27,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279300
2023-11-22 08:05:34,577 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.25 vs. limit=15.0
2023-11-22 08:05:55,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1862100.0, ans=0.125
2023-11-22 08:06:12,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1862166.6666666667, ans=0.125
2023-11-22 08:06:20,802 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:06:21,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1862233.3333333333, ans=0.125
2023-11-22 08:06:28,057 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2800, loss[loss=0.07615, simple_loss=0.1002, pruned_loss=0.01569, audio_tagging_loss=0.01038, over 14528.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09267, pruned_loss=0.01526, audio_tagging_loss=0.00943, over 3042371.44 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:06:32,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279350
2023-11-22 08:06:32,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1862300.0, ans=0.0
2023-11-22 08:06:59,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1862433.3333333333, ans=0.0
2023-11-22 08:07:01,170 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 7.937e+01 8.597e+01 9.316e+01 1.176e+02, threshold=1.719e+02, percent-clipped=0.0
2023-11-22 08:07:08,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1862500.0, ans=0.0
2023-11-22 08:07:16,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1862500.0, ans=0.125
2023-11-22 08:07:33,308 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2850, loss[loss=0.08255, simple_loss=0.114, pruned_loss=0.01809, audio_tagging_loss=0.007465, over 15883.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09258, pruned_loss=0.01511, audio_tagging_loss=0.009339, over 3043761.83 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:07:37,082 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279400
2023-11-22 08:07:37,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.02 vs. limit=22.5
2023-11-22 08:07:49,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1862700.0, ans=0.1
2023-11-22 08:08:01,090 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.96 vs. limit=6.0
2023-11-22 08:08:08,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1862766.6666666667, ans=0.125
2023-11-22 08:08:22,170 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.65 vs. limit=15.0
2023-11-22 08:08:25,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1862900.0, ans=0.125
2023-11-22 08:08:25,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1862900.0, ans=0.0
2023-11-22 08:08:38,453 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2900, loss[loss=0.06339, simple_loss=0.08034, pruned_loss=0.01307, audio_tagging_loss=0.01016, over 15717.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09222, pruned_loss=0.01496, audio_tagging_loss=0.009301, over 3051456.57 frames. ], batch size: 60, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:08:41,918 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:08:42,878 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279450
2023-11-22 08:08:44,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1862966.6666666667, ans=0.2
2023-11-22 08:08:44,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1862966.6666666667, ans=0.0
2023-11-22 08:08:45,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.02 vs. limit=15.0
2023-11-22 08:08:55,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1863033.3333333333, ans=0.0
2023-11-22 08:08:55,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1863033.3333333333, ans=0.04949747468305833
2023-11-22 08:09:02,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1863033.3333333333, ans=0.125
2023-11-22 08:09:09,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1863100.0, ans=0.2
2023-11-22 08:09:11,276 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.692e+01 8.213e+01 8.733e+01 9.343e+01 1.106e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 08:09:13,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.34 vs. limit=10.0
2023-11-22 08:09:43,399 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 2950, loss[loss=0.06807, simple_loss=0.09533, pruned_loss=0.01151, audio_tagging_loss=0.008898, over 15295.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.09305, pruned_loss=0.01506, audio_tagging_loss=0.009252, over 3048893.96 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:09:47,298 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279500
2023-11-22 08:09:48,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1863300.0, ans=0.2
2023-11-22 08:10:11,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1863433.3333333333, ans=0.1
2023-11-22 08:10:14,835 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.77 vs. limit=6.0
2023-11-22 08:10:49,482 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3000, loss[loss=0.09678, simple_loss=0.1281, pruned_loss=0.02359, audio_tagging_loss=0.00917, over 16200.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.09372, pruned_loss=0.0152, audio_tagging_loss=0.009244, over 3051944.04 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:10:49,486 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 08:11:29,451 INFO [train_asr.py:1253] (0/4) Epoch 24, validation: loss=0.0588, simple_loss=0.05168, pruned_loss=0.005124, audio_tagging_loss=0.02784, over 4681554.00 frames. 
2023-11-22 08:11:29,452 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 08:11:33,222 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279550
2023-11-22 08:11:45,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1863700.0, ans=0.125
2023-11-22 08:11:56,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1863766.6666666667, ans=0.125
2023-11-22 08:12:00,633 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-22 08:12:02,512 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.135e+01 8.700e+01 9.423e+01 1.185e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 08:12:32,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1863900.0, ans=0.125
2023-11-22 08:12:32,994 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.95 vs. limit=10.0
2023-11-22 08:12:34,538 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3050, loss[loss=0.06796, simple_loss=0.0878, pruned_loss=0.01535, audio_tagging_loss=0.008707, over 14479.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09428, pruned_loss=0.01527, audio_tagging_loss=0.009276, over 3054072.43 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:12:38,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279600
2023-11-22 08:12:47,392 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:12:55,350 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.08 vs. limit=15.0
2023-11-22 08:12:58,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1864033.3333333333, ans=0.125
2023-11-22 08:12:58,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1864033.3333333333, ans=0.0
2023-11-22 08:13:14,121 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:13:30,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1864233.3333333333, ans=0.2
2023-11-22 08:13:36,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1864233.3333333333, ans=0.125
2023-11-22 08:13:41,241 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3100, loss[loss=0.06178, simple_loss=0.0833, pruned_loss=0.01127, audio_tagging_loss=0.008853, over 15737.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09421, pruned_loss=0.01531, audio_tagging_loss=0.00937, over 3049883.22 frames. ], batch size: 57, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:13:45,143 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279650
2023-11-22 08:13:45,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1864300.0, ans=0.125
2023-11-22 08:14:04,001 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.92 vs. limit=15.0
2023-11-22 08:14:06,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1864433.3333333333, ans=0.2
2023-11-22 08:14:10,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1864433.3333333333, ans=0.125
2023-11-22 08:14:13,541 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.026e+01 8.146e+01 8.664e+01 9.339e+01 1.098e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 08:14:36,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1864566.6666666667, ans=0.0
2023-11-22 08:14:46,724 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3150, loss[loss=0.07741, simple_loss=0.09844, pruned_loss=0.02102, audio_tagging_loss=0.007172, over 14244.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09521, pruned_loss=0.01537, audio_tagging_loss=0.009409, over 3041041.05 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:14:50,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279700
2023-11-22 08:14:50,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1864633.3333333333, ans=0.125
2023-11-22 08:14:55,931 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.32 vs. limit=15.0
2023-11-22 08:14:59,609 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.01 vs. limit=15.0
2023-11-22 08:15:19,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1864766.6666666667, ans=0.0
2023-11-22 08:15:19,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1864766.6666666667, ans=0.125
2023-11-22 08:15:42,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1864900.0, ans=0.125
2023-11-22 08:15:50,940 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3200, loss[loss=0.08494, simple_loss=0.116, pruned_loss=0.02181, audio_tagging_loss=0.005136, over 15142.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09475, pruned_loss=0.01536, audio_tagging_loss=0.009463, over 3044403.24 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:15:52,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1864966.6666666667, ans=0.125
2023-11-22 08:15:55,349 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279750
2023-11-22 08:16:21,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1865100.0, ans=0.125
2023-11-22 08:16:24,180 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.350e+01 8.310e+01 8.993e+01 9.711e+01 1.257e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 08:16:56,988 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3250, loss[loss=0.07979, simple_loss=0.1065, pruned_loss=0.01448, audio_tagging_loss=0.01207, over 14616.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09476, pruned_loss=0.01531, audio_tagging_loss=0.009646, over 3047007.41 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:17:00,778 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279800
2023-11-22 08:17:08,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.79 vs. limit=15.0
2023-11-22 08:17:11,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1865366.6666666667, ans=0.125
2023-11-22 08:17:20,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1865366.6666666667, ans=0.2
2023-11-22 08:17:22,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1865433.3333333333, ans=0.125
2023-11-22 08:17:55,246 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.50 vs. limit=15.0
2023-11-22 08:18:02,039 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3300, loss[loss=0.06375, simple_loss=0.08827, pruned_loss=0.0117, audio_tagging_loss=0.007911, over 15298.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09482, pruned_loss=0.0153, audio_tagging_loss=0.0097, over 3042052.48 frames. ], batch size: 56, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:18:06,405 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279850
2023-11-22 08:18:07,095 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.15 vs. limit=10.0
2023-11-22 08:18:15,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1865700.0, ans=0.0
2023-11-22 08:18:28,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.98 vs. limit=15.0
2023-11-22 08:18:31,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1865766.6666666667, ans=0.0
2023-11-22 08:18:35,001 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.287e+01 8.648e+01 9.394e+01 1.159e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 08:19:04,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1865900.0, ans=0.125
2023-11-22 08:19:06,568 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3350, loss[loss=0.077, simple_loss=0.09796, pruned_loss=0.02037, audio_tagging_loss=0.007645, over 15375.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09392, pruned_loss=0.01529, audio_tagging_loss=0.009647, over 3044596.71 frames. ], batch size: 60, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:19:08,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.31 vs. limit=22.5
2023-11-22 08:19:09,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1865966.6666666667, ans=0.125
2023-11-22 08:19:10,569 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279900
2023-11-22 08:19:26,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1866033.3333333333, ans=0.1
2023-11-22 08:19:39,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1866100.0, ans=0.1
2023-11-22 08:20:02,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1866233.3333333333, ans=0.0
2023-11-22 08:20:10,098 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.27 vs. limit=15.0
2023-11-22 08:20:11,677 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3400, loss[loss=0.1065, simple_loss=0.1465, pruned_loss=0.02882, audio_tagging_loss=0.0044, over 13976.00 frames. ], tot_loss[loss=0.07215, simple_loss=0.09418, pruned_loss=0.01553, audio_tagging_loss=0.009523, over 3041496.06 frames. ], batch size: 52, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:20:15,448 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 279950
2023-11-22 08:20:28,527 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.48 vs. limit=6.0
2023-11-22 08:20:30,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1866366.6666666667, ans=0.2
2023-11-22 08:20:32,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.56 vs. limit=6.0
2023-11-22 08:20:42,979 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.164e+01 8.350e+01 8.855e+01 9.450e+01 1.235e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 08:20:45,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1866433.3333333333, ans=0.0
2023-11-22 08:20:50,987 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.55 vs. limit=15.0
2023-11-22 08:21:15,131 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3450, loss[loss=0.06816, simple_loss=0.08721, pruned_loss=0.01552, audio_tagging_loss=0.009028, over 14328.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09393, pruned_loss=0.01542, audio_tagging_loss=0.009412, over 3037467.56 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:21:16,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1866633.3333333333, ans=0.1
2023-11-22 08:21:19,554 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280000
2023-11-22 08:21:21,088 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-280000.pt
2023-11-22 08:21:53,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1866766.6666666667, ans=0.125
2023-11-22 08:22:23,253 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3500, loss[loss=0.08017, simple_loss=0.1084, pruned_loss=0.01841, audio_tagging_loss=0.007548, over 15965.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09307, pruned_loss=0.0152, audio_tagging_loss=0.009366, over 3039332.13 frames. ], batch size: 58, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:22:24,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.17 vs. limit=22.5
2023-11-22 08:22:26,948 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280050
2023-11-22 08:22:28,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1866966.6666666667, ans=0.125
2023-11-22 08:22:30,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1866966.6666666667, ans=0.0
2023-11-22 08:22:56,274 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.244e+01 8.926e+01 9.794e+01 1.238e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 08:22:56,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1867100.0, ans=0.125
2023-11-22 08:22:57,695 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:23:28,843 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.09 vs. limit=15.0
2023-11-22 08:23:29,251 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3550, loss[loss=0.06797, simple_loss=0.09473, pruned_loss=0.01326, audio_tagging_loss=0.007346, over 14668.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09313, pruned_loss=0.01511, audio_tagging_loss=0.009277, over 3039390.55 frames. ], batch size: 54, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:23:29,935 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.77 vs. limit=15.0
2023-11-22 08:23:31,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1867300.0, ans=0.125
2023-11-22 08:23:33,695 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280100
2023-11-22 08:23:38,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1867300.0, ans=0.125
2023-11-22 08:23:40,484 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.54 vs. limit=22.5
2023-11-22 08:23:44,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1867366.6666666667, ans=0.125
2023-11-22 08:23:46,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1867366.6666666667, ans=0.125
2023-11-22 08:24:21,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff2.min_abs, batch_count=1867566.6666666667, ans=0.1
2023-11-22 08:24:34,299 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3600, loss[loss=0.09202, simple_loss=0.1326, pruned_loss=0.02115, audio_tagging_loss=0.004575, over 14991.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09226, pruned_loss=0.0149, audio_tagging_loss=0.009385, over 3033086.88 frames. ], batch size: 55, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:24:38,052 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280150
2023-11-22 08:24:58,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1867766.6666666667, ans=0.125
2023-11-22 08:25:07,391 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 7.927e+01 8.588e+01 9.278e+01 1.263e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 08:25:39,372 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3650, loss[loss=0.07254, simple_loss=0.09492, pruned_loss=0.01778, audio_tagging_loss=0.007304, over 12895.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09315, pruned_loss=0.01514, audio_tagging_loss=0.009339, over 3034251.69 frames. ], batch size: 51, lr: 2.86e-03, grad_scale: 32.0
2023-11-22 08:25:43,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280200
2023-11-22 08:25:45,323 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.12 vs. limit=15.0
2023-11-22 08:26:00,192 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.28 vs. limit=12.0
2023-11-22 08:26:03,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1868033.3333333333, ans=0.0
2023-11-22 08:26:09,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.51 vs. limit=10.0
2023-11-22 08:26:45,337 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3700, loss[loss=0.05464, simple_loss=0.06794, pruned_loss=0.00831, audio_tagging_loss=0.01235, over 15304.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09339, pruned_loss=0.01517, audio_tagging_loss=0.009263, over 3032444.91 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:26:48,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1868300.0, ans=0.05
2023-11-22 08:26:49,238 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280250
2023-11-22 08:26:50,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1868300.0, ans=0.0
2023-11-22 08:26:50,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1868300.0, ans=10.0
2023-11-22 08:27:18,801 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.613e+01 8.045e+01 8.787e+01 9.464e+01 1.178e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 08:27:20,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1868433.3333333333, ans=0.125
2023-11-22 08:27:31,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1868500.0, ans=0.0
2023-11-22 08:27:36,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1868566.6666666667, ans=0.1
2023-11-22 08:27:46,634 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:27:50,876 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3750, loss[loss=0.07257, simple_loss=0.09918, pruned_loss=0.01613, audio_tagging_loss=0.006858, over 15319.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09504, pruned_loss=0.01555, audio_tagging_loss=0.009107, over 3040093.20 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:27:54,649 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280300
2023-11-22 08:28:26,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1868766.6666666667, ans=0.1
2023-11-22 08:28:29,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1868833.3333333333, ans=0.125
2023-11-22 08:28:35,758 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:28:39,585 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.70 vs. limit=15.0
2023-11-22 08:28:55,616 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3800, loss[loss=0.06462, simple_loss=0.08144, pruned_loss=0.01435, audio_tagging_loss=0.009555, over 14523.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.09543, pruned_loss=0.01563, audio_tagging_loss=0.009235, over 3045275.92 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:28:57,623 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.32 vs. limit=12.0
2023-11-22 08:28:59,325 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280350
2023-11-22 08:29:14,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1869033.3333333333, ans=10.0
2023-11-22 08:29:19,857 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.49 vs. limit=15.0
2023-11-22 08:29:27,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1869100.0, ans=0.1
2023-11-22 08:29:28,706 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.943e+01 8.492e+01 9.128e+01 1.004e+02 1.242e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-22 08:29:30,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1869100.0, ans=0.2
2023-11-22 08:29:59,889 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3850, loss[loss=0.09629, simple_loss=0.1354, pruned_loss=0.02136, audio_tagging_loss=0.007212, over 16092.00 frames. ], tot_loss[loss=0.07301, simple_loss=0.09595, pruned_loss=0.01571, audio_tagging_loss=0.009326, over 3037964.31 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:30:02,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1869300.0, ans=0.125
2023-11-22 08:30:04,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280400
2023-11-22 08:30:20,145 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.59 vs. limit=15.0
2023-11-22 08:30:37,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.41 vs. limit=6.0
2023-11-22 08:30:44,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1869500.0, ans=0.125
2023-11-22 08:30:48,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1869500.0, ans=0.125
2023-11-22 08:31:04,443 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3900, loss[loss=0.07514, simple_loss=0.1076, pruned_loss=0.01127, audio_tagging_loss=0.01005, over 16309.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09554, pruned_loss=0.01549, audio_tagging_loss=0.009436, over 3036855.21 frames. ], batch size: 60, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:31:04,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1869633.3333333333, ans=0.1
2023-11-22 08:31:08,392 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280450
2023-11-22 08:31:23,743 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.77 vs. limit=10.0
2023-11-22 08:31:34,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1869766.6666666667, ans=0.125
2023-11-22 08:31:35,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1869766.6666666667, ans=0.1
2023-11-22 08:31:38,010 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.837e+01 8.174e+01 8.873e+01 9.506e+01 1.121e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 08:31:44,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1869833.3333333333, ans=0.1
2023-11-22 08:31:49,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1869833.3333333333, ans=0.125
2023-11-22 08:32:10,116 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 3950, loss[loss=0.07663, simple_loss=0.09423, pruned_loss=0.01876, audio_tagging_loss=0.01076, over 14654.00 frames. ], tot_loss[loss=0.07249, simple_loss=0.09507, pruned_loss=0.01548, audio_tagging_loss=0.009482, over 3028774.66 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:32:13,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280500
2023-11-22 08:32:27,681 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.48 vs. limit=15.0
2023-11-22 08:32:42,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1870100.0, ans=0.125
2023-11-22 08:32:45,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1870100.0, ans=0.0
2023-11-22 08:32:49,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1870166.6666666667, ans=0.125
2023-11-22 08:32:50,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=1870166.6666666667, ans=0.05
2023-11-22 08:33:01,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1870233.3333333333, ans=0.125
2023-11-22 08:33:02,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1870233.3333333333, ans=0.1
2023-11-22 08:33:06,875 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.34 vs. limit=15.0
2023-11-22 08:33:10,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.85 vs. limit=15.0
2023-11-22 08:33:13,422 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4000, loss[loss=0.06527, simple_loss=0.0866, pruned_loss=0.01272, audio_tagging_loss=0.009249, over 15137.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.09585, pruned_loss=0.01561, audio_tagging_loss=0.009487, over 3037562.33 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:33:17,839 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280550
2023-11-22 08:33:48,396 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.016e+01 8.419e+01 9.002e+01 9.753e+01 1.228e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 08:33:51,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1870500.0, ans=0.2
2023-11-22 08:33:57,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=1870500.0, ans=0.07
2023-11-22 08:34:13,676 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.34 vs. limit=22.5
2023-11-22 08:34:16,254 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.68 vs. limit=12.0
2023-11-22 08:34:18,137 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4050, loss[loss=0.1036, simple_loss=0.134, pruned_loss=0.02839, audio_tagging_loss=0.008243, over 14416.00 frames. ], tot_loss[loss=0.07296, simple_loss=0.09564, pruned_loss=0.01561, audio_tagging_loss=0.009528, over 3033439.18 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:34:21,966 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:34:21,995 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280600
2023-11-22 08:34:26,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1870633.3333333333, ans=0.125
2023-11-22 08:34:41,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.71 vs. limit=15.0
2023-11-22 08:34:45,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1870766.6666666667, ans=0.125
2023-11-22 08:35:12,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1870900.0, ans=0.95
2023-11-22 08:35:22,999 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4100, loss[loss=0.07375, simple_loss=0.1003, pruned_loss=0.01607, audio_tagging_loss=0.007555, over 14585.00 frames. ], tot_loss[loss=0.07349, simple_loss=0.09619, pruned_loss=0.01584, audio_tagging_loss=0.009552, over 3028509.47 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:35:27,394 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280650
2023-11-22 08:35:45,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1871033.3333333333, ans=0.2
2023-11-22 08:35:56,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.98 vs. limit=6.0
2023-11-22 08:35:57,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1871100.0, ans=0.125
2023-11-22 08:35:58,180 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.230e+01 8.854e+01 9.579e+01 2.804e+02, threshold=1.771e+02, percent-clipped=1.0
2023-11-22 08:36:19,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1871233.3333333333, ans=0.5
2023-11-22 08:36:27,908 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4150, loss[loss=0.07352, simple_loss=0.103, pruned_loss=0.01551, audio_tagging_loss=0.006521, over 16400.00 frames. ], tot_loss[loss=0.07274, simple_loss=0.09576, pruned_loss=0.01549, audio_tagging_loss=0.009374, over 3033178.39 frames. ], batch size: 59, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:36:31,189 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.19 vs. limit=15.0
2023-11-22 08:36:31,682 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280700
2023-11-22 08:36:49,675 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.53 vs. limit=6.0
2023-11-22 08:36:51,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1871366.6666666667, ans=0.125
2023-11-22 08:37:04,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1871433.3333333333, ans=0.2
2023-11-22 08:37:11,074 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.37 vs. limit=15.0
2023-11-22 08:37:14,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1871500.0, ans=0.025
2023-11-22 08:37:15,468 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:37:28,493 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.94 vs. limit=12.0
2023-11-22 08:37:32,657 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4200, loss[loss=0.08419, simple_loss=0.1145, pruned_loss=0.01779, audio_tagging_loss=0.009162, over 15249.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.0952, pruned_loss=0.01544, audio_tagging_loss=0.009393, over 3032378.86 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:37:36,383 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280750
2023-11-22 08:37:49,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.28 vs. limit=10.0
2023-11-22 08:37:54,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1871700.0, ans=0.0
2023-11-22 08:37:58,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1871766.6666666667, ans=0.125
2023-11-22 08:38:07,277 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.921e+01 8.240e+01 8.912e+01 9.752e+01 1.165e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 08:38:11,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.24 vs. limit=15.0
2023-11-22 08:38:36,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1871966.6666666667, ans=0.125
2023-11-22 08:38:37,260 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4250, loss[loss=0.05462, simple_loss=0.06983, pruned_loss=0.01023, audio_tagging_loss=0.009474, over 14667.00 frames. ], tot_loss[loss=0.07277, simple_loss=0.09594, pruned_loss=0.01553, audio_tagging_loss=0.009277, over 3038027.41 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:38:41,789 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280800
2023-11-22 08:38:51,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1872033.3333333333, ans=0.0
2023-11-22 08:39:10,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1872100.0, ans=0.125
2023-11-22 08:39:22,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.76 vs. limit=22.5
2023-11-22 08:39:43,321 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4300, loss[loss=0.07982, simple_loss=0.1099, pruned_loss=0.01786, audio_tagging_loss=0.006995, over 15203.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09504, pruned_loss=0.01535, audio_tagging_loss=0.009292, over 3038202.59 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:39:47,930 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280850
2023-11-22 08:40:14,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1872433.3333333333, ans=0.125
2023-11-22 08:40:18,686 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.298e+01 8.496e+01 9.037e+01 9.736e+01 1.212e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-22 08:40:30,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1872500.0, ans=0.0
2023-11-22 08:40:32,302 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.48 vs. limit=15.0
2023-11-22 08:40:34,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1872566.6666666667, ans=0.125
2023-11-22 08:40:47,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.59 vs. limit=15.0
2023-11-22 08:40:49,256 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4350, loss[loss=0.08868, simple_loss=0.1148, pruned_loss=0.02272, audio_tagging_loss=0.008578, over 15705.00 frames. ], tot_loss[loss=0.07269, simple_loss=0.09585, pruned_loss=0.01548, audio_tagging_loss=0.009285, over 3042009.57 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:40:53,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280900
2023-11-22 08:40:59,703 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.99 vs. limit=15.0
2023-11-22 08:41:03,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1872700.0, ans=0.125
2023-11-22 08:41:07,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1872700.0, ans=0.0
2023-11-22 08:41:09,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1872700.0, ans=0.0
2023-11-22 08:41:23,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1872766.6666666667, ans=0.0
2023-11-22 08:41:51,424 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.34 vs. limit=10.0
2023-11-22 08:41:53,080 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4400, loss[loss=0.07715, simple_loss=0.1148, pruned_loss=0.01332, audio_tagging_loss=0.00642, over 15345.00 frames. ], tot_loss[loss=0.07288, simple_loss=0.0961, pruned_loss=0.01561, audio_tagging_loss=0.009217, over 3046373.09 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:41:55,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1872966.6666666667, ans=0.125
2023-11-22 08:41:56,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 280950
2023-11-22 08:42:12,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1873033.3333333333, ans=0.125
2023-11-22 08:42:22,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1873100.0, ans=0.125
2023-11-22 08:42:29,226 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.700e+01 8.342e+01 8.985e+01 9.628e+01 1.162e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 08:42:32,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1873166.6666666667, ans=0.0
2023-11-22 08:42:36,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.36 vs. limit=15.0
2023-11-22 08:42:43,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1873166.6666666667, ans=0.0
2023-11-22 08:42:58,224 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4450, loss[loss=0.05425, simple_loss=0.06475, pruned_loss=0.01299, audio_tagging_loss=0.008886, over 15234.00 frames. ], tot_loss[loss=0.07305, simple_loss=0.09639, pruned_loss=0.01571, audio_tagging_loss=0.00915, over 3047608.17 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:43:01,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281000
2023-11-22 08:43:55,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1873566.6666666667, ans=0.125
2023-11-22 08:43:56,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1873566.6666666667, ans=0.125
2023-11-22 08:44:03,768 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4500, loss[loss=0.07729, simple_loss=0.1022, pruned_loss=0.01701, audio_tagging_loss=0.009162, over 15236.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09576, pruned_loss=0.01544, audio_tagging_loss=0.009132, over 3045053.73 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:44:05,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1873633.3333333333, ans=0.125
2023-11-22 08:44:07,705 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281050
2023-11-22 08:44:08,323 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.84 vs. limit=15.0
2023-11-22 08:44:28,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1873766.6666666667, ans=0.04949747468305833
2023-11-22 08:44:37,536 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.057e+01 8.748e+01 9.801e+01 1.166e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 08:44:44,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.21 vs. limit=15.0
2023-11-22 08:44:45,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1873833.3333333333, ans=0.1
2023-11-22 08:44:45,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1873833.3333333333, ans=0.1
2023-11-22 08:44:47,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1873833.3333333333, ans=0.0
2023-11-22 08:44:51,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1873833.3333333333, ans=0.1
2023-11-22 08:45:01,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1873900.0, ans=10.0
2023-11-22 08:45:07,127 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4550, loss[loss=0.06032, simple_loss=0.07282, pruned_loss=0.01092, audio_tagging_loss=0.013, over 13825.00 frames. ], tot_loss[loss=0.07293, simple_loss=0.09641, pruned_loss=0.01562, audio_tagging_loss=0.009107, over 3053137.17 frames. ], batch size: 52, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:45:10,919 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281100
2023-11-22 08:45:14,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1873966.6666666667, ans=0.2
2023-11-22 08:45:25,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1874033.3333333333, ans=0.0
2023-11-22 08:45:30,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1874033.3333333333, ans=0.125
2023-11-22 08:45:49,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1874166.6666666667, ans=0.125
2023-11-22 08:45:50,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1874166.6666666667, ans=0.1
2023-11-22 08:45:56,324 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 08:46:02,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1874233.3333333333, ans=0.2
2023-11-22 08:46:11,520 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4600, loss[loss=0.07719, simple_loss=0.1019, pruned_loss=0.0173, audio_tagging_loss=0.008935, over 15870.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09461, pruned_loss=0.01518, audio_tagging_loss=0.009326, over 3052397.30 frames. ], batch size: 60, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:46:15,309 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281150
2023-11-22 08:46:23,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1874366.6666666667, ans=0.125
2023-11-22 08:46:43,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1874433.3333333333, ans=0.0
2023-11-22 08:46:46,494 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.769e+01 7.994e+01 8.789e+01 9.560e+01 1.302e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 08:46:50,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1874500.0, ans=0.125
2023-11-22 08:47:16,077 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4650, loss[loss=0.08114, simple_loss=0.1106, pruned_loss=0.01923, audio_tagging_loss=0.006633, over 15758.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09468, pruned_loss=0.01547, audio_tagging_loss=0.009366, over 3056732.62 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:47:16,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1874633.3333333333, ans=0.2
2023-11-22 08:47:20,481 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281200
2023-11-22 08:47:29,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1874700.0, ans=0.125
2023-11-22 08:47:34,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1874700.0, ans=0.0
2023-11-22 08:47:40,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1874700.0, ans=0.0
2023-11-22 08:48:15,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1874900.0, ans=0.1
2023-11-22 08:48:21,837 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4700, loss[loss=0.08006, simple_loss=0.1058, pruned_loss=0.01854, audio_tagging_loss=0.008617, over 15384.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09371, pruned_loss=0.01525, audio_tagging_loss=0.009475, over 3057497.60 frames. ], batch size: 58, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:48:25,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281250
2023-11-22 08:48:25,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1874966.6666666667, ans=0.125
2023-11-22 08:48:33,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1875033.3333333333, ans=0.125
2023-11-22 08:48:46,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1875100.0, ans=0.125
2023-11-22 08:48:57,767 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.881e+01 8.141e+01 8.777e+01 9.599e+01 1.409e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-22 08:49:03,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1875166.6666666667, ans=0.125
2023-11-22 08:49:05,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.22 vs. limit=10.0
2023-11-22 08:49:05,224 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=15.0
2023-11-22 08:49:08,820 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.40 vs. limit=22.5
2023-11-22 08:49:20,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1875233.3333333333, ans=0.0
2023-11-22 08:49:25,817 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4750, loss[loss=0.08149, simple_loss=0.1096, pruned_loss=0.01952, audio_tagging_loss=0.007174, over 15161.00 frames. ], tot_loss[loss=0.07171, simple_loss=0.09387, pruned_loss=0.01529, audio_tagging_loss=0.009485, over 3050875.31 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:49:28,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1875300.0, ans=0.0
2023-11-22 08:49:29,577 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281300
2023-11-22 08:50:14,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1875500.0, ans=0.125
2023-11-22 08:50:24,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1875566.6666666667, ans=0.125
2023-11-22 08:50:26,323 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:50:29,577 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4800, loss[loss=0.06516, simple_loss=0.07452, pruned_loss=0.01173, audio_tagging_loss=0.01617, over 14175.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09327, pruned_loss=0.01525, audio_tagging_loss=0.009563, over 3049100.03 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:50:34,374 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281350
2023-11-22 08:50:34,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1875633.3333333333, ans=0.0
2023-11-22 08:50:39,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1875633.3333333333, ans=0.125
2023-11-22 08:51:02,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1875766.6666666667, ans=0.125
2023-11-22 08:51:07,404 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.460e+01 8.056e+01 8.734e+01 9.359e+01 1.284e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 08:51:07,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1875833.3333333333, ans=0.2
2023-11-22 08:51:16,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1875833.3333333333, ans=0.1
2023-11-22 08:51:34,897 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4850, loss[loss=0.05401, simple_loss=0.07122, pruned_loss=0.00712, audio_tagging_loss=0.01127, over 14725.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09424, pruned_loss=0.01536, audio_tagging_loss=0.009548, over 3046226.57 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:51:38,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281400
2023-11-22 08:51:42,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1875966.6666666667, ans=0.2
2023-11-22 08:51:51,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1876033.3333333333, ans=0.125
2023-11-22 08:51:52,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1876033.3333333333, ans=0.125
2023-11-22 08:51:53,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.99 vs. limit=15.0
2023-11-22 08:51:55,935 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.17 vs. limit=15.0
2023-11-22 08:52:01,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1876100.0, ans=0.125
2023-11-22 08:52:04,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1876100.0, ans=0.1
2023-11-22 08:52:20,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1876166.6666666667, ans=0.1
2023-11-22 08:52:26,005 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.80 vs. limit=22.5
2023-11-22 08:52:34,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1876233.3333333333, ans=0.0
2023-11-22 08:52:39,577 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4900, loss[loss=0.06691, simple_loss=0.08925, pruned_loss=0.01386, audio_tagging_loss=0.008418, over 14754.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09444, pruned_loss=0.01544, audio_tagging_loss=0.009521, over 3038172.83 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:52:42,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1876300.0, ans=0.0
2023-11-22 08:52:43,320 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281450
2023-11-22 08:53:14,388 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.07 vs. limit=6.0
2023-11-22 08:53:16,823 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.163e+01 8.764e+01 9.409e+01 1.191e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 08:53:31,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1876566.6666666667, ans=0.0
2023-11-22 08:53:43,101 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 4950, loss[loss=0.04353, simple_loss=0.04857, pruned_loss=0.008505, audio_tagging_loss=0.01074, over 13688.00 frames. ], tot_loss[loss=0.07178, simple_loss=0.09406, pruned_loss=0.01535, audio_tagging_loss=0.009403, over 3034985.92 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:53:47,681 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281500
2023-11-22 08:54:19,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1876766.6666666667, ans=0.0
2023-11-22 08:54:31,895 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=15.0
2023-11-22 08:54:34,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.66 vs. limit=15.0
2023-11-22 08:54:38,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1876900.0, ans=0.125
2023-11-22 08:54:48,547 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5000, loss[loss=0.05835, simple_loss=0.07123, pruned_loss=0.01144, audio_tagging_loss=0.01129, over 14739.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09491, pruned_loss=0.01558, audio_tagging_loss=0.009278, over 3039604.53 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:54:50,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1876966.6666666667, ans=0.0
2023-11-22 08:54:52,327 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281550
2023-11-22 08:55:04,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1877033.3333333333, ans=0.125
2023-11-22 08:55:23,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1877100.0, ans=0.125
2023-11-22 08:55:25,180 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.261e+01 8.670e+01 9.360e+01 1.108e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-22 08:55:31,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1877166.6666666667, ans=0.1
2023-11-22 08:55:37,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1877166.6666666667, ans=0.0
2023-11-22 08:55:52,740 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5050, loss[loss=0.07685, simple_loss=0.09776, pruned_loss=0.01954, audio_tagging_loss=0.008432, over 15785.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09418, pruned_loss=0.01523, audio_tagging_loss=0.009293, over 3038318.33 frames. ], batch size: 61, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:55:57,048 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281600
2023-11-22 08:56:12,460 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.69 vs. limit=12.0
2023-11-22 08:56:17,777 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:56:29,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1877433.3333333333, ans=0.2
2023-11-22 08:56:29,311 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:56:38,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1877500.0, ans=0.2
2023-11-22 08:56:39,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1877500.0, ans=0.125
2023-11-22 08:56:45,376 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.85 vs. limit=22.5
2023-11-22 08:56:46,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1877566.6666666667, ans=0.125
2023-11-22 08:56:46,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1877566.6666666667, ans=0.125
2023-11-22 08:56:54,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1877566.6666666667, ans=0.2
2023-11-22 08:56:57,108 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5100, loss[loss=0.08501, simple_loss=0.1157, pruned_loss=0.02005, audio_tagging_loss=0.007118, over 15149.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09414, pruned_loss=0.01526, audio_tagging_loss=0.009223, over 3038892.36 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:57:00,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281650
2023-11-22 08:57:12,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1877700.0, ans=0.2
2023-11-22 08:57:17,334 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.25 vs. limit=15.0
2023-11-22 08:57:21,802 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 08:57:31,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1877766.6666666667, ans=0.125
2023-11-22 08:57:34,832 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.920e+01 7.966e+01 8.782e+01 9.481e+01 2.193e+02, threshold=1.756e+02, percent-clipped=1.0
2023-11-22 08:57:35,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1877833.3333333333, ans=0.0
2023-11-22 08:57:44,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1877833.3333333333, ans=0.0
2023-11-22 08:57:46,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1877833.3333333333, ans=0.125
2023-11-22 08:58:01,521 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5150, loss[loss=0.07695, simple_loss=0.1001, pruned_loss=0.01709, audio_tagging_loss=0.009814, over 15510.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09406, pruned_loss=0.01522, audio_tagging_loss=0.009297, over 3048526.23 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 08:58:04,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1877966.6666666667, ans=0.125
2023-11-22 08:58:05,232 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281700
2023-11-22 08:58:05,634 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.84 vs. limit=15.0
2023-11-22 08:58:10,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1877966.6666666667, ans=0.0
2023-11-22 08:58:31,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1878100.0, ans=0.0
2023-11-22 08:58:31,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1878100.0, ans=0.125
2023-11-22 08:58:32,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1878100.0, ans=0.125
2023-11-22 08:58:33,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.44 vs. limit=15.0
2023-11-22 08:58:33,994 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.57 vs. limit=15.0
2023-11-22 08:58:38,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1878166.6666666667, ans=0.1
2023-11-22 08:58:54,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1878233.3333333333, ans=0.1
2023-11-22 08:59:04,822 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5200, loss[loss=0.09452, simple_loss=0.1289, pruned_loss=0.02076, audio_tagging_loss=0.009304, over 15777.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09454, pruned_loss=0.01536, audio_tagging_loss=0.009192, over 3041420.01 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 08:59:08,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281750
2023-11-22 08:59:23,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1878366.6666666667, ans=0.2
2023-11-22 08:59:29,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1878433.3333333333, ans=0.125
2023-11-22 08:59:34,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1878433.3333333333, ans=0.0
2023-11-22 08:59:42,536 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.104e+01 8.582e+01 9.336e+01 1.183e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-22 09:00:00,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1878566.6666666667, ans=0.125
2023-11-22 09:00:09,535 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5250, loss[loss=0.06839, simple_loss=0.09385, pruned_loss=0.01422, audio_tagging_loss=0.007243, over 14567.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09458, pruned_loss=0.01524, audio_tagging_loss=0.009164, over 3045357.81 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 09:00:13,326 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281800
2023-11-22 09:00:45,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1878766.6666666667, ans=0.125
2023-11-22 09:01:08,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.61 vs. limit=15.0
2023-11-22 09:01:10,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1878900.0, ans=0.0
2023-11-22 09:01:14,908 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5300, loss[loss=0.09209, simple_loss=0.129, pruned_loss=0.02191, audio_tagging_loss=0.005699, over 15311.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09548, pruned_loss=0.0154, audio_tagging_loss=0.009117, over 3051406.92 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 09:01:18,605 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281850
2023-11-22 09:01:41,887 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.71 vs. limit=22.5
2023-11-22 09:01:44,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1879100.0, ans=0.2
2023-11-22 09:01:53,092 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.133e+01 8.772e+01 9.528e+01 1.202e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 09:01:53,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1879166.6666666667, ans=0.125
2023-11-22 09:01:57,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1879166.6666666667, ans=0.0
2023-11-22 09:02:08,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1879233.3333333333, ans=0.0
2023-11-22 09:02:13,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1879233.3333333333, ans=0.125
2023-11-22 09:02:19,277 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5350, loss[loss=0.0734, simple_loss=0.08521, pruned_loss=0.01873, audio_tagging_loss=0.01207, over 16173.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.0951, pruned_loss=0.01543, audio_tagging_loss=0.009228, over 3046313.13 frames. ], batch size: 60, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:02:22,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1879300.0, ans=0.07
2023-11-22 09:02:23,119 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281900
2023-11-22 09:02:48,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1879433.3333333333, ans=0.125
2023-11-22 09:03:17,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1879566.6666666667, ans=0.0
2023-11-22 09:03:21,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1879566.6666666667, ans=0.07
2023-11-22 09:03:23,613 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5400, loss[loss=0.07848, simple_loss=0.1064, pruned_loss=0.01453, audio_tagging_loss=0.01075, over 15263.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09505, pruned_loss=0.01532, audio_tagging_loss=0.009336, over 3040356.79 frames. ], batch size: 55, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:03:27,982 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 281950
2023-11-22 09:03:40,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1879700.0, ans=0.0
2023-11-22 09:03:50,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1879766.6666666667, ans=0.125
2023-11-22 09:04:01,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1879833.3333333333, ans=0.025
2023-11-22 09:04:02,269 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.131e+01 8.289e+01 8.769e+01 9.344e+01 1.157e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 09:04:11,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1879833.3333333333, ans=0.2
2023-11-22 09:04:22,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1879900.0, ans=0.125
2023-11-22 09:04:29,474 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5450, loss[loss=0.09221, simple_loss=0.1265, pruned_loss=0.01844, audio_tagging_loss=0.01052, over 15592.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09412, pruned_loss=0.01526, audio_tagging_loss=0.009442, over 3042141.83 frames. ], batch size: 54, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:04:33,210 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282000
2023-11-22 09:04:33,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1879966.6666666667, ans=0.0
2023-11-22 09:04:33,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1879966.6666666667, ans=0.1
2023-11-22 09:04:38,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1879966.6666666667, ans=0.0
2023-11-22 09:04:39,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1879966.6666666667, ans=0.125
2023-11-22 09:04:47,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1880033.3333333333, ans=0.125
2023-11-22 09:04:52,537 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.91 vs. limit=22.5
2023-11-22 09:04:56,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1880100.0, ans=0.025
2023-11-22 09:05:01,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1880100.0, ans=0.125
2023-11-22 09:05:19,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1880166.6666666667, ans=0.0
2023-11-22 09:05:33,766 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5500, loss[loss=0.04225, simple_loss=0.0498, pruned_loss=0.004992, audio_tagging_loss=0.01236, over 14771.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09408, pruned_loss=0.01525, audio_tagging_loss=0.009442, over 3045216.78 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:05:35,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1880300.0, ans=0.04949747468305833
2023-11-22 09:05:37,526 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282050
2023-11-22 09:05:43,132 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=15.0
2023-11-22 09:05:47,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1880366.6666666667, ans=0.125
2023-11-22 09:05:50,935 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.06 vs. limit=15.0
2023-11-22 09:05:52,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.86 vs. limit=15.0
2023-11-22 09:05:55,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1880366.6666666667, ans=0.2
2023-11-22 09:06:03,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=1880433.3333333333, ans=0.95
2023-11-22 09:06:12,697 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 8.257e+01 8.828e+01 9.557e+01 1.200e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 09:06:21,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1880500.0, ans=0.07
2023-11-22 09:06:34,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1880566.6666666667, ans=0.0
2023-11-22 09:06:38,163 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5550, loss[loss=0.07093, simple_loss=0.09863, pruned_loss=0.01412, audio_tagging_loss=0.007493, over 15815.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09389, pruned_loss=0.01507, audio_tagging_loss=0.009531, over 3044515.07 frames. ], batch size: 57, lr: 2.85e-03, grad_scale: 16.0
2023-11-22 09:06:41,946 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282100
2023-11-22 09:06:55,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1880700.0, ans=0.0
2023-11-22 09:07:16,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1880833.3333333333, ans=0.125
2023-11-22 09:07:20,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1880833.3333333333, ans=0.0
2023-11-22 09:07:30,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1880900.0, ans=0.0
2023-11-22 09:07:33,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1880900.0, ans=0.0
2023-11-22 09:07:43,089 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5600, loss[loss=0.07008, simple_loss=0.09931, pruned_loss=0.01269, audio_tagging_loss=0.007735, over 15226.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09506, pruned_loss=0.01516, audio_tagging_loss=0.00955, over 3045822.01 frames. ], batch size: 56, lr: 2.85e-03, grad_scale: 32.0
2023-11-22 09:07:47,501 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282150
2023-11-22 09:08:21,232 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.179e+01 8.745e+01 9.416e+01 1.882e+02, threshold=1.749e+02, percent-clipped=1.0
2023-11-22 09:08:30,614 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 09:08:43,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1881233.3333333333, ans=0.125
2023-11-22 09:08:47,868 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5650, loss[loss=0.06762, simple_loss=0.08751, pruned_loss=0.01087, audio_tagging_loss=0.013, over 16952.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09481, pruned_loss=0.01522, audio_tagging_loss=0.00962, over 3042129.01 frames. ], batch size: 64, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:08:51,680 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282200
2023-11-22 09:09:27,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1881500.0, ans=0.0
2023-11-22 09:09:52,850 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5700, loss[loss=0.07644, simple_loss=0.1004, pruned_loss=0.01835, audio_tagging_loss=0.007877, over 14882.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.0938, pruned_loss=0.01497, audio_tagging_loss=0.009606, over 3046444.49 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:09:56,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282250
2023-11-22 09:09:57,184 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.27 vs. limit=6.0
2023-11-22 09:10:13,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1881700.0, ans=0.1
2023-11-22 09:10:23,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1881766.6666666667, ans=0.05
2023-11-22 09:10:30,654 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.915e+01 8.108e+01 8.715e+01 9.329e+01 1.149e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 09:10:55,638 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5750, loss[loss=0.07584, simple_loss=0.09955, pruned_loss=0.01625, audio_tagging_loss=0.009811, over 14964.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.0942, pruned_loss=0.01512, audio_tagging_loss=0.009506, over 3046838.75 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:10:59,965 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282300
2023-11-22 09:11:00,115 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:11:05,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1881966.6666666667, ans=0.07
2023-11-22 09:11:33,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1882166.6666666667, ans=0.05
2023-11-22 09:11:36,802 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.01 vs. limit=22.5
2023-11-22 09:11:40,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1882166.6666666667, ans=0.025
2023-11-22 09:12:01,186 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5800, loss[loss=0.08645, simple_loss=0.124, pruned_loss=0.01733, audio_tagging_loss=0.007119, over 15810.00 frames. ], tot_loss[loss=0.07194, simple_loss=0.09448, pruned_loss=0.01528, audio_tagging_loss=0.009421, over 3040258.09 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:12:01,942 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.80 vs. limit=22.5
2023-11-22 09:12:04,911 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282350
2023-11-22 09:12:13,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1882366.6666666667, ans=0.125
2023-11-22 09:12:16,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1882366.6666666667, ans=0.125
2023-11-22 09:12:39,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1882500.0, ans=0.0
2023-11-22 09:12:40,751 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.350e+01 8.216e+01 8.857e+01 9.622e+01 1.311e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 09:12:43,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1882500.0, ans=0.0
2023-11-22 09:12:46,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1882500.0, ans=0.2
2023-11-22 09:12:49,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1882500.0, ans=0.125
2023-11-22 09:12:55,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1882566.6666666667, ans=0.1
2023-11-22 09:12:55,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1882566.6666666667, ans=0.125
2023-11-22 09:13:03,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_na.min_abs, batch_count=1882633.3333333333, ans=0.02
2023-11-22 09:13:05,431 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5850, loss[loss=0.05491, simple_loss=0.0683, pruned_loss=0.0101, audio_tagging_loss=0.01066, over 14193.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09455, pruned_loss=0.01546, audio_tagging_loss=0.009301, over 3033578.65 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:13:07,413 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.22 vs. limit=22.5
2023-11-22 09:13:09,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282400
2023-11-22 09:13:15,341 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.23 vs. limit=10.0
2023-11-22 09:13:37,891 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.76 vs. limit=22.5
2023-11-22 09:13:54,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1882833.3333333333, ans=10.0
2023-11-22 09:14:08,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1882900.0, ans=0.125
2023-11-22 09:14:10,105 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5900, loss[loss=0.06662, simple_loss=0.09698, pruned_loss=0.01063, audio_tagging_loss=0.007499, over 14949.00 frames. ], tot_loss[loss=0.07264, simple_loss=0.09562, pruned_loss=0.01558, audio_tagging_loss=0.009252, over 3033402.86 frames. ], batch size: 54, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:14:14,449 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282450
2023-11-22 09:14:26,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1883033.3333333333, ans=0.125
2023-11-22 09:14:32,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1883033.3333333333, ans=0.0
2023-11-22 09:14:36,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1883100.0, ans=0.1
2023-11-22 09:14:39,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1883100.0, ans=0.125
2023-11-22 09:14:50,069 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.669e+01 8.181e+01 9.057e+01 9.624e+01 1.134e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 09:14:50,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=1883166.6666666667, ans=0.2
2023-11-22 09:15:14,472 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 5950, loss[loss=0.05121, simple_loss=0.05791, pruned_loss=0.01035, audio_tagging_loss=0.0119, over 14403.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09535, pruned_loss=0.01544, audio_tagging_loss=0.00929, over 3035290.22 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:15:14,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1883300.0, ans=0.0
2023-11-22 09:15:18,801 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282500
2023-11-22 09:15:31,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1883366.6666666667, ans=0.125
2023-11-22 09:15:34,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1883366.6666666667, ans=0.125
2023-11-22 09:15:38,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1883366.6666666667, ans=0.0
2023-11-22 09:15:54,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1883500.0, ans=0.125
2023-11-22 09:16:00,594 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0
2023-11-22 09:16:19,191 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6000, loss[loss=0.07193, simple_loss=0.09855, pruned_loss=0.01499, audio_tagging_loss=0.007665, over 14709.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09503, pruned_loss=0.01549, audio_tagging_loss=0.009228, over 3038301.28 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:16:19,195 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 09:16:42,932 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.5816, 2.4759, 2.3932, 2.2723, 2.6695, 2.6388, 2.6715, 2.6443],
       device='cuda:0')
2023-11-22 09:16:44,970 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.5910, 2.5554, 3.3206, 2.6720], device='cuda:0')
2023-11-22 09:17:01,480 INFO [train_asr.py:1253] (0/4) Epoch 24, validation: loss=0.05933, simple_loss=0.05174, pruned_loss=0.005222, audio_tagging_loss=0.02824, over 4681554.00 frames. 
2023-11-22 09:17:01,481 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 09:17:04,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1883633.3333333333, ans=0.0
2023-11-22 09:17:05,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282550
2023-11-22 09:17:09,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1883633.3333333333, ans=0.125
2023-11-22 09:17:17,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1883700.0, ans=0.0
2023-11-22 09:17:22,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1883700.0, ans=0.125
2023-11-22 09:17:28,273 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.97 vs. limit=6.0
2023-11-22 09:17:29,530 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.77 vs. limit=10.0
2023-11-22 09:17:32,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1883766.6666666667, ans=0.0
2023-11-22 09:17:38,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1883766.6666666667, ans=0.125
2023-11-22 09:17:42,981 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.160e+01 8.694e+01 9.357e+01 1.133e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 09:17:49,664 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 09:18:01,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1883900.0, ans=0.015
2023-11-22 09:18:02,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1883900.0, ans=0.0
2023-11-22 09:18:06,493 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6050, loss[loss=0.06697, simple_loss=0.08704, pruned_loss=0.01268, audio_tagging_loss=0.01077, over 14383.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.09473, pruned_loss=0.01539, audio_tagging_loss=0.009164, over 3037996.33 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:18:10,404 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282600
2023-11-22 09:18:20,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1884033.3333333333, ans=0.2
2023-11-22 09:18:55,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1884166.6666666667, ans=0.125
2023-11-22 09:18:57,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1884233.3333333333, ans=0.0
2023-11-22 09:19:12,041 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6100, loss[loss=0.04567, simple_loss=0.0583, pruned_loss=0.006856, audio_tagging_loss=0.009665, over 14715.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09499, pruned_loss=0.01542, audio_tagging_loss=0.009164, over 3037732.03 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:19:15,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282650
2023-11-22 09:19:17,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1884300.0, ans=0.1
2023-11-22 09:19:36,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1884366.6666666667, ans=0.125
2023-11-22 09:19:42,429 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.87 vs. limit=10.0
2023-11-22 09:19:43,294 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:19:44,759 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.20 vs. limit=15.0
2023-11-22 09:19:46,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1884433.3333333333, ans=0.125
2023-11-22 09:19:47,458 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.35 vs. limit=15.0
2023-11-22 09:19:50,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1884500.0, ans=0.2
2023-11-22 09:19:53,500 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.538e+01 9.030e+01 1.014e+02 1.305e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-22 09:19:53,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1884500.0, ans=0.125
2023-11-22 09:19:57,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1884500.0, ans=0.0
2023-11-22 09:20:13,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1884566.6666666667, ans=0.0
2023-11-22 09:20:16,950 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6150, loss[loss=0.05536, simple_loss=0.07151, pruned_loss=0.01037, audio_tagging_loss=0.009237, over 15416.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09442, pruned_loss=0.0155, audio_tagging_loss=0.009241, over 3038278.39 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:20:20,846 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282700
2023-11-22 09:21:13,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1884900.0, ans=0.1
2023-11-22 09:21:22,147 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6200, loss[loss=0.08575, simple_loss=0.1106, pruned_loss=0.02029, audio_tagging_loss=0.01015, over 15486.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09343, pruned_loss=0.01522, audio_tagging_loss=0.009244, over 3041619.00 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:21:22,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1884966.6666666667, ans=0.125
2023-11-22 09:21:26,002 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282750
2023-11-22 09:21:33,978 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.59 vs. limit=12.0
2023-11-22 09:21:41,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1885033.3333333333, ans=0.125
2023-11-22 09:21:47,646 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.35 vs. limit=22.5
2023-11-22 09:21:50,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1885100.0, ans=10.0
2023-11-22 09:21:56,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1885100.0, ans=0.035
2023-11-22 09:21:57,792 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.25 vs. limit=15.0
2023-11-22 09:22:03,954 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.132e+01 8.774e+01 9.442e+01 1.403e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-22 09:22:14,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1885233.3333333333, ans=0.125
2023-11-22 09:22:20,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1885233.3333333333, ans=0.125
2023-11-22 09:22:21,914 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.24 vs. limit=22.5
2023-11-22 09:22:26,722 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6250, loss[loss=0.07321, simple_loss=0.09216, pruned_loss=0.01674, audio_tagging_loss=0.01039, over 14543.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09333, pruned_loss=0.01503, audio_tagging_loss=0.009453, over 3042194.61 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:22:31,131 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282800
2023-11-22 09:22:35,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1885300.0, ans=0.2
2023-11-22 09:22:38,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1885366.6666666667, ans=0.2
2023-11-22 09:22:55,480 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.35 vs. limit=22.5
2023-11-22 09:22:58,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1885433.3333333333, ans=0.125
2023-11-22 09:22:58,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1885433.3333333333, ans=0.125
2023-11-22 09:23:01,575 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.36 vs. limit=15.0
2023-11-22 09:23:07,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1885500.0, ans=0.2
2023-11-22 09:23:30,901 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6300, loss[loss=0.07967, simple_loss=0.109, pruned_loss=0.0153, audio_tagging_loss=0.009873, over 15641.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09335, pruned_loss=0.01492, audio_tagging_loss=0.009436, over 3045532.18 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:23:35,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282850
2023-11-22 09:24:10,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1885833.3333333333, ans=0.0
2023-11-22 09:24:11,941 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.966e+01 8.510e+01 9.202e+01 1.035e+02 1.385e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-22 09:24:16,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1885833.3333333333, ans=0.125
2023-11-22 09:24:26,545 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.84 vs. limit=12.0
2023-11-22 09:24:35,091 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6350, loss[loss=0.06386, simple_loss=0.08704, pruned_loss=0.01157, audio_tagging_loss=0.00877, over 14951.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.0943, pruned_loss=0.01498, audio_tagging_loss=0.009442, over 3051426.88 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:24:38,861 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282900
2023-11-22 09:24:50,133 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.91 vs. limit=12.0
2023-11-22 09:25:04,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1886100.0, ans=0.2
2023-11-22 09:25:12,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1886166.6666666667, ans=0.125
2023-11-22 09:25:19,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1886166.6666666667, ans=0.2
2023-11-22 09:25:22,407 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.48 vs. limit=6.0
2023-11-22 09:25:38,863 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6400, loss[loss=0.07353, simple_loss=0.09038, pruned_loss=0.01747, audio_tagging_loss=0.01087, over 16350.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09279, pruned_loss=0.01478, audio_tagging_loss=0.009673, over 3042407.51 frames. ], batch size: 62, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:25:43,227 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 282950
2023-11-22 09:25:53,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1886366.6666666667, ans=0.125
2023-11-22 09:25:54,044 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.67 vs. limit=15.0
2023-11-22 09:26:20,180 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.176e+01 8.732e+01 9.554e+01 1.218e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 09:26:20,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1886500.0, ans=0.1
2023-11-22 09:26:21,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1886500.0, ans=0.025
2023-11-22 09:26:43,535 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6450, loss[loss=0.08935, simple_loss=0.1232, pruned_loss=0.01844, audio_tagging_loss=0.009306, over 16918.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09351, pruned_loss=0.01496, audio_tagging_loss=0.009666, over 3047250.31 frames. ], batch size: 61, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:26:47,346 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283000
2023-11-22 09:26:57,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1886700.0, ans=0.125
2023-11-22 09:27:02,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1886700.0, ans=0.125
2023-11-22 09:27:05,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1886700.0, ans=0.1
2023-11-22 09:27:33,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1886833.3333333333, ans=0.0
2023-11-22 09:27:45,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1886900.0, ans=0.125
2023-11-22 09:27:48,096 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.09 vs. limit=15.0
2023-11-22 09:27:48,691 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6500, loss[loss=0.06862, simple_loss=0.08905, pruned_loss=0.01523, audio_tagging_loss=0.00886, over 13962.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09382, pruned_loss=0.01512, audio_tagging_loss=0.009589, over 3051159.94 frames. ], batch size: 53, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:27:52,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283050
2023-11-22 09:28:12,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1887100.0, ans=0.04949747468305833
2023-11-22 09:28:16,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1887100.0, ans=22.5
2023-11-22 09:28:26,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1887166.6666666667, ans=0.2
2023-11-22 09:28:28,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1887166.6666666667, ans=0.125
2023-11-22 09:28:30,692 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.224e+01 8.620e+01 9.457e+01 1.249e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-22 09:28:31,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1887166.6666666667, ans=0.0
2023-11-22 09:28:34,282 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.29 vs. limit=15.0
2023-11-22 09:28:38,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1887233.3333333333, ans=0.0
2023-11-22 09:28:52,100 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6550, loss[loss=0.06128, simple_loss=0.07403, pruned_loss=0.01456, audio_tagging_loss=0.009707, over 13907.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09422, pruned_loss=0.01511, audio_tagging_loss=0.009497, over 3049643.63 frames. ], batch size: 53, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:28:55,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283100
2023-11-22 09:29:34,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1887500.0, ans=0.0
2023-11-22 09:29:56,339 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6600, loss[loss=0.06223, simple_loss=0.07845, pruned_loss=0.01238, audio_tagging_loss=0.01062, over 14648.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09372, pruned_loss=0.01497, audio_tagging_loss=0.009383, over 3050124.71 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:29:58,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1887633.3333333333, ans=0.125
2023-11-22 09:29:59,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283150
2023-11-22 09:30:36,903 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.97 vs. limit=15.0
2023-11-22 09:30:38,602 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.912e+01 8.247e+01 8.842e+01 9.573e+01 1.466e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 09:30:46,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1887900.0, ans=0.07
2023-11-22 09:31:00,508 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6650, loss[loss=0.06617, simple_loss=0.08587, pruned_loss=0.0129, audio_tagging_loss=0.01034, over 14157.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.09425, pruned_loss=0.01507, audio_tagging_loss=0.009201, over 3054592.60 frames. ], batch size: 54, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:31:03,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1887966.6666666667, ans=0.125
2023-11-22 09:31:04,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283200
2023-11-22 09:31:04,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1887966.6666666667, ans=0.125
2023-11-22 09:31:13,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.68 vs. limit=15.0
2023-11-22 09:31:34,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1888100.0, ans=0.0
2023-11-22 09:32:04,429 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6700, loss[loss=0.06772, simple_loss=0.0915, pruned_loss=0.01265, audio_tagging_loss=0.009321, over 14597.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09421, pruned_loss=0.015, audio_tagging_loss=0.00914, over 3053658.89 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:32:05,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1888300.0, ans=0.125
2023-11-22 09:32:08,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283250
2023-11-22 09:32:15,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1888300.0, ans=0.2
2023-11-22 09:32:31,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1888433.3333333333, ans=0.0
2023-11-22 09:32:47,400 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.099e+01 8.643e+01 9.373e+01 1.139e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 09:32:55,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1888566.6666666667, ans=0.0
2023-11-22 09:32:59,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1888566.6666666667, ans=0.125
2023-11-22 09:33:08,704 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6750, loss[loss=0.06892, simple_loss=0.08589, pruned_loss=0.01516, audio_tagging_loss=0.01081, over 14706.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09377, pruned_loss=0.0149, audio_tagging_loss=0.009241, over 3048475.69 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:33:12,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283300
2023-11-22 09:33:41,013 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.39 vs. limit=15.0
2023-11-22 09:33:43,453 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.20 vs. limit=15.0
2023-11-22 09:33:45,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1888833.3333333333, ans=0.125
2023-11-22 09:34:01,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1888900.0, ans=0.2
2023-11-22 09:34:12,848 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6800, loss[loss=0.05194, simple_loss=0.06951, pruned_loss=0.008828, audio_tagging_loss=0.008361, over 14898.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09463, pruned_loss=0.01518, audio_tagging_loss=0.009255, over 3043532.34 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:34:17,154 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283350
2023-11-22 09:34:20,214 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.60 vs. limit=22.5
2023-11-22 09:34:27,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1889033.3333333333, ans=0.0
2023-11-22 09:34:31,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1889033.3333333333, ans=0.125
2023-11-22 09:34:33,717 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:34:34,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=1889033.3333333333, ans=0.05
2023-11-22 09:34:37,700 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.14 vs. limit=15.0
2023-11-22 09:34:45,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1889100.0, ans=0.125
2023-11-22 09:34:51,805 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.08 vs. limit=15.0
2023-11-22 09:34:54,674 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 7.933e+01 8.700e+01 9.598e+01 1.312e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 09:35:10,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1889233.3333333333, ans=0.125
2023-11-22 09:35:17,332 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6850, loss[loss=0.07865, simple_loss=0.1063, pruned_loss=0.01627, audio_tagging_loss=0.009239, over 15616.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09422, pruned_loss=0.01519, audio_tagging_loss=0.009288, over 3036818.21 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:35:21,135 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283400
2023-11-22 09:35:22,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1889300.0, ans=0.125
2023-11-22 09:35:37,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1889366.6666666667, ans=0.125
2023-11-22 09:35:37,639 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.98 vs. limit=15.0
2023-11-22 09:36:15,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1889566.6666666667, ans=0.2
2023-11-22 09:36:21,961 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6900, loss[loss=0.04327, simple_loss=0.05429, pruned_loss=0.006567, audio_tagging_loss=0.00956, over 14682.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09393, pruned_loss=0.01505, audio_tagging_loss=0.00926, over 3034154.74 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:36:25,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283450
2023-11-22 09:36:30,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1889633.3333333333, ans=0.05
2023-11-22 09:36:34,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1889700.0, ans=0.125
2023-11-22 09:36:43,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1889700.0, ans=0.125
2023-11-22 09:37:01,124 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.31 vs. limit=15.0
2023-11-22 09:37:03,981 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.035e+01 8.885e+01 9.571e+01 1.179e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 09:37:05,766 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.76 vs. limit=15.0
2023-11-22 09:37:12,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1889900.0, ans=0.125
2023-11-22 09:37:13,116 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 09:37:19,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1889900.0, ans=0.1
2023-11-22 09:37:25,874 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 6950, loss[loss=0.06639, simple_loss=0.07907, pruned_loss=0.01601, audio_tagging_loss=0.01085, over 15008.00 frames. ], tot_loss[loss=0.07213, simple_loss=0.09515, pruned_loss=0.01533, audio_tagging_loss=0.009233, over 3035488.81 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:37:30,229 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283500
2023-11-22 09:37:33,273 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.92 vs. limit=15.0
2023-11-22 09:37:51,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1890100.0, ans=0.1
2023-11-22 09:38:04,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1890166.6666666667, ans=0.1
2023-11-22 09:38:06,479 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0
2023-11-22 09:38:16,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1890233.3333333333, ans=0.2
2023-11-22 09:38:23,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1890233.3333333333, ans=0.0
2023-11-22 09:38:30,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1890300.0, ans=0.0
2023-11-22 09:38:30,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=1890300.0, ans=0.2
2023-11-22 09:38:31,041 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7000, loss[loss=0.05877, simple_loss=0.07434, pruned_loss=0.00929, audio_tagging_loss=0.01231, over 16791.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.0944, pruned_loss=0.01512, audio_tagging_loss=0.009276, over 3043469.24 frames. ], batch size: 64, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:38:34,718 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283550
2023-11-22 09:39:11,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1890500.0, ans=0.0
2023-11-22 09:39:13,208 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.396e+01 8.305e+01 8.768e+01 9.472e+01 1.582e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 09:39:35,623 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7050, loss[loss=0.07988, simple_loss=0.1073, pruned_loss=0.01463, audio_tagging_loss=0.0116, over 15495.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09426, pruned_loss=0.01516, audio_tagging_loss=0.009285, over 3043860.99 frames. ], batch size: 57, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:39:39,396 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283600
2023-11-22 09:39:42,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1890633.3333333333, ans=0.05
2023-11-22 09:39:49,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1890700.0, ans=0.125
2023-11-22 09:40:05,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1890766.6666666667, ans=0.1
2023-11-22 09:40:17,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1890833.3333333333, ans=10.0
2023-11-22 09:40:28,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.96 vs. limit=22.5
2023-11-22 09:40:36,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1890900.0, ans=0.2
2023-11-22 09:40:39,975 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7100, loss[loss=0.0863, simple_loss=0.1183, pruned_loss=0.02063, audio_tagging_loss=0.006522, over 14811.00 frames. ], tot_loss[loss=0.07195, simple_loss=0.09483, pruned_loss=0.01524, audio_tagging_loss=0.009298, over 3049503.15 frames. ], batch size: 53, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:40:41,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1890966.6666666667, ans=0.0
2023-11-22 09:40:44,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283650
2023-11-22 09:40:48,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1890966.6666666667, ans=0.0
2023-11-22 09:41:06,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1891100.0, ans=0.125
2023-11-22 09:41:12,091 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.15 vs. limit=15.0
2023-11-22 09:41:23,126 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.301e+01 8.958e+01 9.633e+01 1.183e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 09:41:28,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1891166.6666666667, ans=0.1
2023-11-22 09:41:29,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1891166.6666666667, ans=0.0
2023-11-22 09:41:33,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1891233.3333333333, ans=0.1
2023-11-22 09:41:45,093 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7150, loss[loss=0.06073, simple_loss=0.08169, pruned_loss=0.01206, audio_tagging_loss=0.00782, over 15651.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09457, pruned_loss=0.01517, audio_tagging_loss=0.009344, over 3048351.59 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:41:48,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283700
2023-11-22 09:42:03,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1891366.6666666667, ans=0.0
2023-11-22 09:42:18,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1891433.3333333333, ans=0.0
2023-11-22 09:42:20,092 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.47 vs. limit=15.0
2023-11-22 09:42:42,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1891566.6666666667, ans=0.125
2023-11-22 09:42:49,850 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7200, loss[loss=0.08249, simple_loss=0.113, pruned_loss=0.0191, audio_tagging_loss=0.006872, over 14795.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09542, pruned_loss=0.01533, audio_tagging_loss=0.009282, over 3041260.72 frames. ], batch size: 58, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:42:50,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1891633.3333333333, ans=0.0
2023-11-22 09:42:54,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283750
2023-11-22 09:43:32,277 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.224e+01 8.056e+01 8.758e+01 9.674e+01 1.275e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 09:43:34,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1891833.3333333333, ans=0.0
2023-11-22 09:43:54,013 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7250, loss[loss=0.08011, simple_loss=0.1056, pruned_loss=0.01711, audio_tagging_loss=0.01019, over 16262.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09441, pruned_loss=0.01524, audio_tagging_loss=0.009369, over 3041219.16 frames. ], batch size: 60, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:43:57,770 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283800
2023-11-22 09:44:01,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1891966.6666666667, ans=0.2
2023-11-22 09:44:17,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1892033.3333333333, ans=0.125
2023-11-22 09:44:21,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1892100.0, ans=0.2
2023-11-22 09:44:26,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1892100.0, ans=0.125
2023-11-22 09:44:28,495 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.57 vs. limit=6.0
2023-11-22 09:44:39,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1892166.6666666667, ans=0.0
2023-11-22 09:44:41,365 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.04 vs. limit=15.0
2023-11-22 09:44:44,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1892233.3333333333, ans=0.125
2023-11-22 09:44:59,046 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7300, loss[loss=0.06465, simple_loss=0.09484, pruned_loss=0.01142, audio_tagging_loss=0.005817, over 14318.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09434, pruned_loss=0.01525, audio_tagging_loss=0.00939, over 3040618.66 frames. ], batch size: 54, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:44:59,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1892300.0, ans=0.2
2023-11-22 09:45:02,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283850
2023-11-22 09:45:15,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1892366.6666666667, ans=0.125
2023-11-22 09:45:26,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1892433.3333333333, ans=0.125
2023-11-22 09:45:28,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.46 vs. limit=15.0
2023-11-22 09:45:41,602 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.095e+01 8.850e+01 9.580e+01 1.188e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 09:45:46,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1892500.0, ans=0.125
2023-11-22 09:45:47,216 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.31 vs. limit=10.0
2023-11-22 09:45:55,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1892566.6666666667, ans=0.07
2023-11-22 09:45:59,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1892566.6666666667, ans=0.125
2023-11-22 09:45:59,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1892566.6666666667, ans=0.125
2023-11-22 09:46:01,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1892633.3333333333, ans=0.0
2023-11-22 09:46:02,630 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7350, loss[loss=0.07072, simple_loss=0.09285, pruned_loss=0.01528, audio_tagging_loss=0.009009, over 15120.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09335, pruned_loss=0.01513, audio_tagging_loss=0.009213, over 3043574.95 frames. ], batch size: 56, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:46:06,249 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283900
2023-11-22 09:46:31,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1892766.6666666667, ans=0.0
2023-11-22 09:46:55,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1892900.0, ans=0.125
2023-11-22 09:47:07,026 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7400, loss[loss=0.06089, simple_loss=0.07781, pruned_loss=0.0144, audio_tagging_loss=0.007584, over 14812.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09287, pruned_loss=0.01517, audio_tagging_loss=0.009141, over 3047226.07 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:47:10,322 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.91 vs. limit=15.0
2023-11-22 09:47:10,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 283950
2023-11-22 09:47:19,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1893033.3333333333, ans=0.1
2023-11-22 09:47:19,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1893033.3333333333, ans=0.1
2023-11-22 09:47:29,859 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.94 vs. limit=15.0
2023-11-22 09:47:49,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1893166.6666666667, ans=0.0
2023-11-22 09:47:51,346 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 8.493e+01 8.994e+01 9.861e+01 1.121e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 09:47:55,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1893166.6666666667, ans=0.125
2023-11-22 09:47:56,958 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.45 vs. limit=22.5
2023-11-22 09:48:10,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=1893233.3333333333, ans=10.0
2023-11-22 09:48:12,169 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7450, loss[loss=0.07104, simple_loss=0.1026, pruned_loss=0.01268, audio_tagging_loss=0.007052, over 15900.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09317, pruned_loss=0.01515, audio_tagging_loss=0.009176, over 3048983.51 frames. ], batch size: 59, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:48:15,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284000
2023-11-22 09:48:17,338 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-284000.pt
2023-11-22 09:48:22,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1893300.0, ans=0.0
2023-11-22 09:48:31,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1893366.6666666667, ans=0.125
2023-11-22 09:48:31,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1893366.6666666667, ans=0.125
2023-11-22 09:49:19,188 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7500, loss[loss=0.06699, simple_loss=0.08397, pruned_loss=0.01589, audio_tagging_loss=0.00911, over 16202.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09226, pruned_loss=0.01498, audio_tagging_loss=0.009185, over 3046851.00 frames. ], batch size: 61, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:49:22,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284050
2023-11-22 09:49:28,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1893633.3333333333, ans=0.125
2023-11-22 09:49:47,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1893766.6666666667, ans=0.1
2023-11-22 09:50:02,447 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.876e+01 8.170e+01 8.839e+01 9.515e+01 1.142e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 09:50:09,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1893900.0, ans=0.0
2023-11-22 09:50:14,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1893900.0, ans=0.0
2023-11-22 09:50:14,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1893900.0, ans=0.0
2023-11-22 09:50:20,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1893900.0, ans=0.125
2023-11-22 09:50:23,609 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7550, loss[loss=0.06578, simple_loss=0.08346, pruned_loss=0.01113, audio_tagging_loss=0.01291, over 14688.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09304, pruned_loss=0.01502, audio_tagging_loss=0.009129, over 3043925.24 frames. ], batch size: 55, lr: 2.84e-03, grad_scale: 16.0
2023-11-22 09:50:27,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284100
2023-11-22 09:50:27,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1893966.6666666667, ans=0.2
2023-11-22 09:50:33,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1893966.6666666667, ans=0.125
2023-11-22 09:50:36,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1894033.3333333333, ans=0.09899494936611666
2023-11-22 09:50:52,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1894100.0, ans=0.2
2023-11-22 09:50:53,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1894100.0, ans=0.04949747468305833
2023-11-22 09:50:55,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1894100.0, ans=0.125
2023-11-22 09:51:02,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.14 vs. limit=15.0
2023-11-22 09:51:28,603 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7600, loss[loss=0.0525, simple_loss=0.06953, pruned_loss=0.006951, audio_tagging_loss=0.01078, over 15820.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09221, pruned_loss=0.01494, audio_tagging_loss=0.009239, over 3045228.04 frames. ], batch size: 60, lr: 2.84e-03, grad_scale: 32.0
2023-11-22 09:51:30,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1894300.0, ans=0.2
2023-11-22 09:51:31,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1894300.0, ans=0.1
2023-11-22 09:51:32,351 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284150
2023-11-22 09:51:32,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1894300.0, ans=0.125
2023-11-22 09:51:47,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=1894366.6666666667, ans=0.0
2023-11-22 09:52:07,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1894500.0, ans=0.125
2023-11-22 09:52:11,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=1894500.0, ans=0.5
2023-11-22 09:52:12,159 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.245e+01 8.957e+01 9.580e+01 1.201e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 09:52:24,049 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 09:52:32,258 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7650, loss[loss=0.05332, simple_loss=0.07033, pruned_loss=0.01025, audio_tagging_loss=0.007905, over 14466.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09126, pruned_loss=0.01483, audio_tagging_loss=0.009246, over 3042937.84 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:52:33,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1894633.3333333333, ans=0.125
2023-11-22 09:52:36,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284200
2023-11-22 09:53:15,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1894833.3333333333, ans=0.125
2023-11-22 09:53:23,576 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.32 vs. limit=10.0
2023-11-22 09:53:31,012 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.35 vs. limit=22.5
2023-11-22 09:53:36,916 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7700, loss[loss=0.07114, simple_loss=0.08714, pruned_loss=0.01338, audio_tagging_loss=0.01418, over 15416.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09279, pruned_loss=0.01501, audio_tagging_loss=0.009158, over 3048993.76 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:53:37,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1894966.6666666667, ans=0.04949747468305833
2023-11-22 09:53:41,315 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284250
2023-11-22 09:54:09,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1895100.0, ans=0.0
2023-11-22 09:54:19,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1895166.6666666667, ans=0.125
2023-11-22 09:54:20,909 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 7.998e+01 8.702e+01 9.312e+01 1.429e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 09:54:41,581 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7750, loss[loss=0.08793, simple_loss=0.1158, pruned_loss=0.02233, audio_tagging_loss=0.007688, over 14492.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.0936, pruned_loss=0.01516, audio_tagging_loss=0.009216, over 3047735.87 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:54:43,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=1895300.0, ans=0.05
2023-11-22 09:54:45,910 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284300
2023-11-22 09:55:09,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1895433.3333333333, ans=0.125
2023-11-22 09:55:30,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1895500.0, ans=0.0
2023-11-22 09:55:37,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1895566.6666666667, ans=0.125
2023-11-22 09:55:41,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1895566.6666666667, ans=0.1
2023-11-22 09:55:43,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1895566.6666666667, ans=0.125
2023-11-22 09:55:45,646 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7800, loss[loss=0.0878, simple_loss=0.1254, pruned_loss=0.01631, audio_tagging_loss=0.008804, over 15924.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09397, pruned_loss=0.01505, audio_tagging_loss=0.009241, over 3051874.91 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:55:47,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.41 vs. limit=15.0
2023-11-22 09:55:49,342 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284350
2023-11-22 09:56:01,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1895700.0, ans=0.125
2023-11-22 09:56:23,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1895833.3333333333, ans=0.025
2023-11-22 09:56:29,306 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.687e+01 8.112e+01 8.791e+01 9.537e+01 1.111e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 09:56:39,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1895900.0, ans=0.04949747468305833
2023-11-22 09:56:49,593 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7850, loss[loss=0.07754, simple_loss=0.1104, pruned_loss=0.01219, audio_tagging_loss=0.01014, over 15689.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09436, pruned_loss=0.01507, audio_tagging_loss=0.009235, over 3046456.67 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:56:53,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284400
2023-11-22 09:57:06,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1896033.3333333333, ans=0.125
2023-11-22 09:57:06,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1896033.3333333333, ans=0.125
2023-11-22 09:57:14,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1896033.3333333333, ans=0.125
2023-11-22 09:57:17,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1896100.0, ans=0.0
2023-11-22 09:57:19,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1896100.0, ans=0.1
2023-11-22 09:57:25,488 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.03 vs. limit=15.0
2023-11-22 09:57:45,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1896233.3333333333, ans=0.125
2023-11-22 09:57:47,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1896233.3333333333, ans=0.1
2023-11-22 09:57:50,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1896233.3333333333, ans=0.125
2023-11-22 09:57:52,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1896233.3333333333, ans=0.0
2023-11-22 09:57:55,026 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7900, loss[loss=0.09151, simple_loss=0.1329, pruned_loss=0.01802, audio_tagging_loss=0.007049, over 15440.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09533, pruned_loss=0.01525, audio_tagging_loss=0.009295, over 3052112.35 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 09:57:59,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284450
2023-11-22 09:58:00,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1896300.0, ans=0.125
2023-11-22 09:58:37,538 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.160e+01 8.848e+01 9.602e+01 1.825e+02, threshold=1.770e+02, percent-clipped=1.0
2023-11-22 09:58:44,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1896566.6666666667, ans=0.2
2023-11-22 09:58:56,773 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.63 vs. limit=6.0
2023-11-22 09:58:58,652 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 7950, loss[loss=0.06811, simple_loss=0.08181, pruned_loss=0.01409, audio_tagging_loss=0.01312, over 14232.00 frames. ], tot_loss[loss=0.07247, simple_loss=0.09537, pruned_loss=0.0154, audio_tagging_loss=0.009388, over 3055843.70 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 09:59:01,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1896633.3333333333, ans=0.125
2023-11-22 09:59:02,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284500
2023-11-22 09:59:15,235 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 09:59:35,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1896766.6666666667, ans=0.2
2023-11-22 09:59:35,357 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.90 vs. limit=12.0
2023-11-22 09:59:43,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1896833.3333333333, ans=0.0
2023-11-22 09:59:44,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1896833.3333333333, ans=0.1
2023-11-22 09:59:50,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1896900.0, ans=0.0
2023-11-22 10:00:02,915 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8000, loss[loss=0.05857, simple_loss=0.07004, pruned_loss=0.01303, audio_tagging_loss=0.01052, over 16356.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09478, pruned_loss=0.01531, audio_tagging_loss=0.009504, over 3055503.42 frames. ], batch size: 65, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:00:06,797 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284550
2023-11-22 10:00:19,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1897033.3333333333, ans=0.2
2023-11-22 10:00:27,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1897033.3333333333, ans=0.2
2023-11-22 10:00:29,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1897100.0, ans=0.0
2023-11-22 10:00:31,186 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.55 vs. limit=6.0
2023-11-22 10:00:33,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1897100.0, ans=0.125
2023-11-22 10:00:40,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=1897166.6666666667, ans=0.2
2023-11-22 10:00:44,992 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.98 vs. limit=15.0
2023-11-22 10:00:47,710 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.187e+01 8.649e+01 9.156e+01 1.196e+02, threshold=1.730e+02, percent-clipped=0.0
2023-11-22 10:00:52,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=1897166.6666666667, ans=0.2
2023-11-22 10:01:06,856 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8050, loss[loss=0.06757, simple_loss=0.08797, pruned_loss=0.01346, audio_tagging_loss=0.01012, over 15133.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.0938, pruned_loss=0.01511, audio_tagging_loss=0.009618, over 3052588.61 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:01:11,172 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284600
2023-11-22 10:01:41,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1897433.3333333333, ans=0.2
2023-11-22 10:01:47,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1897500.0, ans=0.0
2023-11-22 10:01:48,152 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0
2023-11-22 10:02:12,579 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8100, loss[loss=0.08227, simple_loss=0.1084, pruned_loss=0.02067, audio_tagging_loss=0.00742, over 14258.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09503, pruned_loss=0.01546, audio_tagging_loss=0.009525, over 3054478.46 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:02:16,408 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284650
2023-11-22 10:02:19,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1897633.3333333333, ans=0.125
2023-11-22 10:02:25,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1897700.0, ans=0.125
2023-11-22 10:02:30,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1897700.0, ans=0.0
2023-11-22 10:02:32,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1897700.0, ans=0.0
2023-11-22 10:02:59,270 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.863e+01 8.417e+01 8.877e+01 9.599e+01 1.117e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 10:03:16,305 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8150, loss[loss=0.06858, simple_loss=0.09014, pruned_loss=0.01269, audio_tagging_loss=0.01082, over 14502.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09495, pruned_loss=0.01544, audio_tagging_loss=0.009312, over 3045572.33 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:03:20,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284700
2023-11-22 10:03:37,204 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:03:49,149 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.55 vs. limit=10.0
2023-11-22 10:04:07,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1898233.3333333333, ans=0.125
2023-11-22 10:04:20,965 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8200, loss[loss=0.07094, simple_loss=0.09631, pruned_loss=0.01413, audio_tagging_loss=0.008658, over 15158.00 frames. ], tot_loss[loss=0.07206, simple_loss=0.09486, pruned_loss=0.01534, audio_tagging_loss=0.00929, over 3043666.15 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:04:21,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1898300.0, ans=0.0
2023-11-22 10:04:22,297 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:04:22,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.29 vs. limit=22.5
2023-11-22 10:04:25,394 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284750
2023-11-22 10:04:32,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1898300.0, ans=0.1
2023-11-22 10:04:32,537 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.36 vs. limit=15.0
2023-11-22 10:04:34,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1898366.6666666667, ans=0.0
2023-11-22 10:04:35,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=1898366.6666666667, ans=15.0
2023-11-22 10:04:35,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1898366.6666666667, ans=0.0
2023-11-22 10:04:46,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1898433.3333333333, ans=0.1
2023-11-22 10:05:06,964 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.783e+01 8.106e+01 8.680e+01 9.269e+01 1.329e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-22 10:05:25,268 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8250, loss[loss=0.04493, simple_loss=0.04554, pruned_loss=0.006564, audio_tagging_loss=0.0156, over 15849.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09464, pruned_loss=0.0153, audio_tagging_loss=0.009298, over 3053564.78 frames. ], batch size: 64, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:05:27,351 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.52 vs. limit=10.0
2023-11-22 10:05:29,061 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284800
2023-11-22 10:05:31,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1898633.3333333333, ans=0.0
2023-11-22 10:05:34,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.45 vs. limit=10.0
2023-11-22 10:05:37,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1898700.0, ans=0.125
2023-11-22 10:05:47,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1898700.0, ans=0.125
2023-11-22 10:05:50,521 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.45 vs. limit=22.5
2023-11-22 10:06:17,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1898900.0, ans=0.0
2023-11-22 10:06:27,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1898900.0, ans=0.125
2023-11-22 10:06:27,841 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.64 vs. limit=22.5
2023-11-22 10:06:29,573 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8300, loss[loss=0.07235, simple_loss=0.09201, pruned_loss=0.01566, audio_tagging_loss=0.01069, over 15298.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09373, pruned_loss=0.01516, audio_tagging_loss=0.009346, over 3054656.17 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:06:33,293 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284850
2023-11-22 10:06:41,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1899033.3333333333, ans=0.1
2023-11-22 10:06:49,135 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.40 vs. limit=6.0
2023-11-22 10:07:04,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1899100.0, ans=0.2
2023-11-22 10:07:11,631 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.18 vs. limit=15.0
2023-11-22 10:07:15,713 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 7.988e+01 8.571e+01 9.748e+01 1.296e+02, threshold=1.714e+02, percent-clipped=0.0
2023-11-22 10:07:33,342 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8350, loss[loss=0.07199, simple_loss=0.09161, pruned_loss=0.01493, audio_tagging_loss=0.01126, over 14603.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09329, pruned_loss=0.01512, audio_tagging_loss=0.009323, over 3056449.58 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:07:37,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284900
2023-11-22 10:07:47,387 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.04 vs. limit=6.0
2023-11-22 10:08:04,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1899433.3333333333, ans=0.1
2023-11-22 10:08:31,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1899566.6666666667, ans=0.0
2023-11-22 10:08:38,238 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8400, loss[loss=0.07089, simple_loss=0.08607, pruned_loss=0.01823, audio_tagging_loss=0.009626, over 16889.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09272, pruned_loss=0.01503, audio_tagging_loss=0.009296, over 3047878.05 frames. ], batch size: 64, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:08:41,915 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 284950
2023-11-22 10:08:42,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1899633.3333333333, ans=0.125
2023-11-22 10:08:57,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1899700.0, ans=0.2
2023-11-22 10:09:02,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1899766.6666666667, ans=0.0
2023-11-22 10:09:24,812 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 8.199e+01 8.877e+01 9.641e+01 1.238e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 10:09:42,484 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8450, loss[loss=0.08729, simple_loss=0.1184, pruned_loss=0.01826, audio_tagging_loss=0.009827, over 14170.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09309, pruned_loss=0.01496, audio_tagging_loss=0.009301, over 3045700.93 frames. ], batch size: 54, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:09:42,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1899966.6666666667, ans=0.0
2023-11-22 10:09:46,294 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285000
2023-11-22 10:09:51,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1899966.6666666667, ans=0.09899494936611666
2023-11-22 10:10:02,653 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.51 vs. limit=6.0
2023-11-22 10:10:10,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=1900100.0, ans=15.0
2023-11-22 10:10:11,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1900100.0, ans=0.2
2023-11-22 10:10:16,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1900100.0, ans=0.0
2023-11-22 10:10:22,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1900166.6666666667, ans=0.2
2023-11-22 10:10:23,291 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:10:24,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1900166.6666666667, ans=0.0
2023-11-22 10:10:27,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1900166.6666666667, ans=0.5
2023-11-22 10:10:47,748 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8500, loss[loss=0.06982, simple_loss=0.09858, pruned_loss=0.0121, audio_tagging_loss=0.008433, over 15749.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09389, pruned_loss=0.01504, audio_tagging_loss=0.009281, over 3042371.23 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:10:51,602 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285050
2023-11-22 10:11:12,840 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.81 vs. limit=15.0
2023-11-22 10:11:33,647 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.25 vs. limit=15.0
2023-11-22 10:11:33,986 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.621e+01 8.277e+01 9.051e+01 9.526e+01 1.493e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 10:11:38,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1900566.6666666667, ans=0.1
2023-11-22 10:11:47,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1900566.6666666667, ans=0.1
2023-11-22 10:11:49,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1900566.6666666667, ans=0.125
2023-11-22 10:11:52,685 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8550, loss[loss=0.07089, simple_loss=0.09454, pruned_loss=0.01427, audio_tagging_loss=0.009356, over 15153.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09424, pruned_loss=0.01511, audio_tagging_loss=0.009399, over 3043794.68 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:11:54,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.20 vs. limit=22.5
2023-11-22 10:11:55,967 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.86 vs. limit=15.0
2023-11-22 10:11:56,414 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285100
2023-11-22 10:11:57,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1900633.3333333333, ans=0.125
2023-11-22 10:12:09,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1900700.0, ans=0.2
2023-11-22 10:12:31,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1900833.3333333333, ans=0.1
2023-11-22 10:12:44,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1900900.0, ans=0.2
2023-11-22 10:12:45,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1900900.0, ans=0.0
2023-11-22 10:12:45,756 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.51 vs. limit=15.0
2023-11-22 10:12:55,969 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8600, loss[loss=0.0694, simple_loss=0.0932, pruned_loss=0.01223, audio_tagging_loss=0.01057, over 15431.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09483, pruned_loss=0.01515, audio_tagging_loss=0.009452, over 3047848.64 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:12:59,770 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285150
2023-11-22 10:12:59,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1900966.6666666667, ans=0.0
2023-11-22 10:13:10,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1901033.3333333333, ans=10.0
2023-11-22 10:13:11,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1901033.3333333333, ans=0.1
2023-11-22 10:13:41,806 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.761e+01 8.269e+01 8.846e+01 9.645e+01 1.157e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 10:13:59,989 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8650, loss[loss=0.08647, simple_loss=0.1063, pruned_loss=0.02257, audio_tagging_loss=0.01073, over 15882.00 frames. ], tot_loss[loss=0.07201, simple_loss=0.09465, pruned_loss=0.01522, audio_tagging_loss=0.009473, over 3051876.42 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:14:03,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285200
2023-11-22 10:14:05,938 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.11 vs. limit=6.0
2023-11-22 10:14:32,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1901433.3333333333, ans=0.125
2023-11-22 10:14:57,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1901566.6666666667, ans=0.0
2023-11-22 10:15:05,243 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8700, loss[loss=0.05446, simple_loss=0.06778, pruned_loss=0.01093, audio_tagging_loss=0.009643, over 14771.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09456, pruned_loss=0.01522, audio_tagging_loss=0.009624, over 3043966.73 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:15:09,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285250
2023-11-22 10:15:21,535 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.84 vs. limit=12.0
2023-11-22 10:15:46,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1901833.3333333333, ans=0.0
2023-11-22 10:15:51,790 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.447e+01 8.934e+01 9.629e+01 1.298e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 10:15:56,251 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.33 vs. limit=15.0
2023-11-22 10:15:58,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1901900.0, ans=0.1
2023-11-22 10:16:09,484 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8750, loss[loss=0.0873, simple_loss=0.1154, pruned_loss=0.02111, audio_tagging_loss=0.008496, over 14713.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09431, pruned_loss=0.01515, audio_tagging_loss=0.009626, over 3048038.18 frames. ], batch size: 53, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:16:13,251 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285300
2023-11-22 10:16:15,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1901966.6666666667, ans=0.1
2023-11-22 10:16:19,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1901966.6666666667, ans=0.07
2023-11-22 10:16:30,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1902033.3333333333, ans=0.2
2023-11-22 10:17:03,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1902233.3333333333, ans=0.125
2023-11-22 10:17:13,271 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8800, loss[loss=0.09678, simple_loss=0.1303, pruned_loss=0.02426, audio_tagging_loss=0.007374, over 14439.00 frames. ], tot_loss[loss=0.0724, simple_loss=0.09498, pruned_loss=0.01527, audio_tagging_loss=0.009645, over 3054934.52 frames. ], batch size: 52, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:17:16,977 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285350
2023-11-22 10:17:21,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1902300.0, ans=0.0
2023-11-22 10:17:40,377 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.38 vs. limit=6.0
2023-11-22 10:17:45,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1902433.3333333333, ans=0.0
2023-11-22 10:17:51,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1902500.0, ans=0.125
2023-11-22 10:17:52,221 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.61 vs. limit=15.0
2023-11-22 10:17:59,349 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.772e+01 8.363e+01 9.001e+01 9.837e+01 1.230e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 10:18:07,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1902566.6666666667, ans=0.125
2023-11-22 10:18:08,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1902566.6666666667, ans=0.2
2023-11-22 10:18:17,790 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8850, loss[loss=0.09025, simple_loss=0.1198, pruned_loss=0.02102, audio_tagging_loss=0.009329, over 15216.00 frames. ], tot_loss[loss=0.07278, simple_loss=0.09552, pruned_loss=0.01536, audio_tagging_loss=0.009659, over 3052487.38 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:18:21,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285400
2023-11-22 10:18:23,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1902633.3333333333, ans=0.0
2023-11-22 10:18:32,207 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:18:32,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=1902700.0, ans=10.0
2023-11-22 10:18:33,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1902700.0, ans=0.125
2023-11-22 10:18:42,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1902766.6666666667, ans=0.125
2023-11-22 10:18:45,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1902766.6666666667, ans=0.1
2023-11-22 10:18:48,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1902766.6666666667, ans=0.0
2023-11-22 10:18:51,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1902766.6666666667, ans=0.1
2023-11-22 10:19:02,125 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.97 vs. limit=10.0
2023-11-22 10:19:06,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1902833.3333333333, ans=0.125
2023-11-22 10:19:15,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1902900.0, ans=0.125
2023-11-22 10:19:19,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1902900.0, ans=0.125
2023-11-22 10:19:22,931 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8900, loss[loss=0.07219, simple_loss=0.08885, pruned_loss=0.01602, audio_tagging_loss=0.01174, over 15292.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09498, pruned_loss=0.01524, audio_tagging_loss=0.009573, over 3052449.87 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:19:26,664 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285450
2023-11-22 10:19:31,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1902966.6666666667, ans=0.125
2023-11-22 10:19:44,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1903033.3333333333, ans=0.125
2023-11-22 10:19:51,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1903100.0, ans=0.125
2023-11-22 10:19:55,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1903100.0, ans=0.0
2023-11-22 10:19:56,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1903100.0, ans=0.125
2023-11-22 10:20:06,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1903166.6666666667, ans=0.2
2023-11-22 10:20:10,563 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.554e+01 8.009e+01 8.748e+01 9.224e+01 1.095e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 10:20:26,876 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 8950, loss[loss=0.07296, simple_loss=0.09918, pruned_loss=0.01381, audio_tagging_loss=0.009563, over 15909.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.0953, pruned_loss=0.01523, audio_tagging_loss=0.009393, over 3048554.93 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:20:28,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1903300.0, ans=0.125
2023-11-22 10:20:30,549 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285500
2023-11-22 10:20:36,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1903300.0, ans=0.125
2023-11-22 10:20:51,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1903366.6666666667, ans=0.0
2023-11-22 10:20:53,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1903433.3333333333, ans=0.125
2023-11-22 10:21:02,388 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.44 vs. limit=6.0
2023-11-22 10:21:05,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1903500.0, ans=0.125
2023-11-22 10:21:30,151 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9000, loss[loss=0.08533, simple_loss=0.1142, pruned_loss=0.0204, audio_tagging_loss=0.00781, over 14950.00 frames. ], tot_loss[loss=0.07258, simple_loss=0.0959, pruned_loss=0.01536, audio_tagging_loss=0.009269, over 3051615.66 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:21:30,155 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 10:21:56,464 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.7989, 1.4036, 3.6226, 3.0445, 2.9112, 3.1579, 3.2221, 3.2435],
       device='cuda:0')
2023-11-22 10:22:07,362 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([4.4690, 3.7633, 4.2813, 3.3316], device='cuda:0')
2023-11-22 10:22:12,129 INFO [train_asr.py:1253] (0/4) Epoch 24, validation: loss=0.06037, simple_loss=0.05165, pruned_loss=0.00517, audio_tagging_loss=0.02938, over 4681554.00 frames. 
2023-11-22 10:22:12,130 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 10:22:15,906 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285550
2023-11-22 10:22:21,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1903633.3333333333, ans=0.0
2023-11-22 10:22:28,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1903700.0, ans=0.09899494936611666
2023-11-22 10:22:50,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=1903833.3333333333, ans=0.0
2023-11-22 10:22:59,088 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.344e+01 8.279e+01 8.807e+01 9.755e+01 1.198e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 10:23:08,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1903900.0, ans=0.1
2023-11-22 10:23:15,774 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9050, loss[loss=0.06571, simple_loss=0.08405, pruned_loss=0.01543, audio_tagging_loss=0.008258, over 16864.00 frames. ], tot_loss[loss=0.0722, simple_loss=0.09526, pruned_loss=0.01528, audio_tagging_loss=0.009284, over 3050630.65 frames. ], batch size: 64, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:23:19,507 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285600
2023-11-22 10:23:29,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1904033.3333333333, ans=0.125
2023-11-22 10:23:35,097 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.57 vs. limit=12.0
2023-11-22 10:23:43,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1904100.0, ans=0.2
2023-11-22 10:23:53,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1904166.6666666667, ans=0.125
2023-11-22 10:24:03,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1904166.6666666667, ans=0.2
2023-11-22 10:24:04,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1904166.6666666667, ans=0.0
2023-11-22 10:24:17,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1904233.3333333333, ans=0.2
2023-11-22 10:24:18,134 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.56 vs. limit=10.0
2023-11-22 10:24:20,437 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9100, loss[loss=0.06376, simple_loss=0.08679, pruned_loss=0.01174, audio_tagging_loss=0.008625, over 14806.00 frames. ], tot_loss[loss=0.07239, simple_loss=0.09561, pruned_loss=0.01536, audio_tagging_loss=0.009221, over 3050963.96 frames. ], batch size: 55, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:24:24,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285650
2023-11-22 10:24:31,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1904300.0, ans=0.125
2023-11-22 10:25:06,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1904500.0, ans=0.1
2023-11-22 10:25:07,533 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.735e+01 8.029e+01 8.849e+01 9.580e+01 1.187e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 10:25:09,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1904500.0, ans=0.125
2023-11-22 10:25:24,648 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9150, loss[loss=0.06975, simple_loss=0.1015, pruned_loss=0.01202, audio_tagging_loss=0.00699, over 16409.00 frames. ], tot_loss[loss=0.07261, simple_loss=0.096, pruned_loss=0.01541, audio_tagging_loss=0.009204, over 3049172.68 frames. ], batch size: 60, lr: 2.83e-03, grad_scale: 16.0
2023-11-22 10:25:28,473 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285700
2023-11-22 10:25:28,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1904633.3333333333, ans=0.1
2023-11-22 10:25:58,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1904766.6666666667, ans=0.1
2023-11-22 10:26:04,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1904833.3333333333, ans=0.0
2023-11-22 10:26:06,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1904833.3333333333, ans=0.125
2023-11-22 10:26:28,559 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9200, loss[loss=0.08077, simple_loss=0.1124, pruned_loss=0.01688, audio_tagging_loss=0.007694, over 15144.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09477, pruned_loss=0.01519, audio_tagging_loss=0.009189, over 3050315.23 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:26:32,206 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285750
2023-11-22 10:26:42,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1905033.3333333333, ans=0.125
2023-11-22 10:26:43,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1905033.3333333333, ans=0.125
2023-11-22 10:26:47,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1905033.3333333333, ans=0.125
2023-11-22 10:27:15,867 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.807e+01 8.192e+01 8.757e+01 9.519e+01 1.252e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 10:27:19,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1905233.3333333333, ans=0.2
2023-11-22 10:27:29,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1905233.3333333333, ans=0.0
2023-11-22 10:27:32,497 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9250, loss[loss=0.05763, simple_loss=0.07798, pruned_loss=0.01025, audio_tagging_loss=0.008384, over 14770.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09455, pruned_loss=0.01523, audio_tagging_loss=0.009222, over 3059445.58 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:27:36,333 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285800
2023-11-22 10:27:39,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1905300.0, ans=0.1
2023-11-22 10:27:47,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1905366.6666666667, ans=0.0
2023-11-22 10:27:51,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1905366.6666666667, ans=0.0
2023-11-22 10:28:16,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=1905500.0, ans=0.5
2023-11-22 10:28:16,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1905500.0, ans=0.125
2023-11-22 10:28:37,339 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9300, loss[loss=0.06558, simple_loss=0.08513, pruned_loss=0.01289, audio_tagging_loss=0.01013, over 16020.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09383, pruned_loss=0.01517, audio_tagging_loss=0.009297, over 3060542.74 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:28:41,205 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285850
2023-11-22 10:28:45,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1905633.3333333333, ans=0.1
2023-11-22 10:28:45,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1905633.3333333333, ans=0.125
2023-11-22 10:29:04,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1905766.6666666667, ans=0.125
2023-11-22 10:29:05,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1905766.6666666667, ans=0.0
2023-11-22 10:29:12,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1905766.6666666667, ans=0.125
2023-11-22 10:29:13,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1905766.6666666667, ans=0.2
2023-11-22 10:29:25,415 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.603e+01 8.386e+01 8.892e+01 9.597e+01 1.135e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 10:29:35,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=1905900.0, ans=0.0
2023-11-22 10:29:41,939 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9350, loss[loss=0.05186, simple_loss=0.06503, pruned_loss=0.007083, audio_tagging_loss=0.01226, over 15383.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09463, pruned_loss=0.01517, audio_tagging_loss=0.009264, over 3060563.34 frames. ], batch size: 60, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:29:42,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1905966.6666666667, ans=0.95
2023-11-22 10:29:42,607 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.13 vs. limit=15.0
2023-11-22 10:29:43,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1905966.6666666667, ans=0.0
2023-11-22 10:29:45,658 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285900
2023-11-22 10:29:55,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=1906033.3333333333, ans=0.05
2023-11-22 10:30:06,470 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.19 vs. limit=15.0
2023-11-22 10:30:22,399 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.00 vs. limit=10.0
2023-11-22 10:30:31,244 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.30 vs. limit=6.0
2023-11-22 10:30:36,353 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.29 vs. limit=6.0
2023-11-22 10:30:45,405 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9400, loss[loss=0.07813, simple_loss=0.1027, pruned_loss=0.01836, audio_tagging_loss=0.008424, over 15694.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09402, pruned_loss=0.01513, audio_tagging_loss=0.009435, over 3056048.33 frames. ], batch size: 58, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:30:49,782 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 285950
2023-11-22 10:31:01,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1906366.6666666667, ans=0.0
2023-11-22 10:31:32,535 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.313e+01 8.349e+01 8.920e+01 9.542e+01 1.196e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 10:31:44,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1906566.6666666667, ans=0.125
2023-11-22 10:31:48,478 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:31:49,664 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9450, loss[loss=0.04764, simple_loss=0.05561, pruned_loss=0.01021, audio_tagging_loss=0.009623, over 14811.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09363, pruned_loss=0.01519, audio_tagging_loss=0.009456, over 3049179.24 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:31:54,005 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286000
2023-11-22 10:31:59,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1906633.3333333333, ans=0.2
2023-11-22 10:32:04,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1906700.0, ans=0.125
2023-11-22 10:32:05,898 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.09 vs. limit=15.0
2023-11-22 10:32:07,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1906700.0, ans=0.125
2023-11-22 10:32:41,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1906900.0, ans=0.0
2023-11-22 10:32:49,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=1906900.0, ans=0.025
2023-11-22 10:32:54,486 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9500, loss[loss=0.0661, simple_loss=0.09503, pruned_loss=0.008856, audio_tagging_loss=0.009723, over 15132.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09423, pruned_loss=0.01533, audio_tagging_loss=0.009491, over 3052992.55 frames. ], batch size: 57, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:32:58,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286050
2023-11-22 10:33:00,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1906966.6666666667, ans=0.2
2023-11-22 10:33:00,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1906966.6666666667, ans=0.0
2023-11-22 10:33:26,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1907100.0, ans=0.125
2023-11-22 10:33:29,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1907100.0, ans=0.025
2023-11-22 10:33:42,359 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.223e+01 8.226e+01 8.759e+01 9.603e+01 1.524e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 10:33:45,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1907233.3333333333, ans=0.0
2023-11-22 10:33:49,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1907233.3333333333, ans=0.1
2023-11-22 10:33:55,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1907233.3333333333, ans=0.1
2023-11-22 10:33:57,109 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.24 vs. limit=15.0
2023-11-22 10:33:58,997 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9550, loss[loss=0.07228, simple_loss=0.09066, pruned_loss=0.01871, audio_tagging_loss=0.008247, over 15309.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09462, pruned_loss=0.0154, audio_tagging_loss=0.009505, over 3047935.41 frames. ], batch size: 59, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:34:02,713 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286100
2023-11-22 10:34:18,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1907366.6666666667, ans=0.0
2023-11-22 10:34:20,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1907366.6666666667, ans=0.125
2023-11-22 10:34:29,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1907433.3333333333, ans=0.125
2023-11-22 10:34:43,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=1907500.0, ans=0.95
2023-11-22 10:35:04,223 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9600, loss[loss=0.06711, simple_loss=0.09132, pruned_loss=0.01268, audio_tagging_loss=0.00877, over 15226.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09477, pruned_loss=0.01548, audio_tagging_loss=0.009491, over 3048147.18 frames. ], batch size: 56, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:35:07,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286150
2023-11-22 10:35:35,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1907766.6666666667, ans=0.0
2023-11-22 10:35:51,442 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.074e+01 8.745e+01 9.476e+01 1.191e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 10:35:55,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1907900.0, ans=0.125
2023-11-22 10:35:59,296 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.527e-03
2023-11-22 10:36:07,695 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9650, loss[loss=0.0599, simple_loss=0.08329, pruned_loss=0.01021, audio_tagging_loss=0.008044, over 14453.00 frames. ], tot_loss[loss=0.07239, simple_loss=0.09512, pruned_loss=0.01534, audio_tagging_loss=0.009484, over 3043793.98 frames. ], batch size: 54, lr: 2.83e-03, grad_scale: 32.0
2023-11-22 10:36:09,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1907966.6666666667, ans=0.1
2023-11-22 10:36:12,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286200
2023-11-22 10:36:14,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1907966.6666666667, ans=0.1
2023-11-22 10:36:23,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1908033.3333333333, ans=0.125
2023-11-22 10:36:37,375 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.18 vs. limit=15.0
2023-11-22 10:36:45,217 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.72 vs. limit=15.0
2023-11-22 10:37:07,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1908233.3333333333, ans=0.2
2023-11-22 10:37:12,299 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9700, loss[loss=0.06695, simple_loss=0.08267, pruned_loss=0.01458, audio_tagging_loss=0.01103, over 15231.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09395, pruned_loss=0.01524, audio_tagging_loss=0.009404, over 3042541.79 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:37:16,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286250
2023-11-22 10:37:25,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=1908366.6666666667, ans=15.0
2023-11-22 10:37:59,373 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.165e+01 7.968e+01 8.645e+01 9.511e+01 1.252e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 10:38:01,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1908500.0, ans=0.125
2023-11-22 10:38:15,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1908633.3333333333, ans=0.125
2023-11-22 10:38:16,173 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9750, loss[loss=0.05408, simple_loss=0.0754, pruned_loss=0.008316, audio_tagging_loss=0.00806, over 15528.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09403, pruned_loss=0.01514, audio_tagging_loss=0.009282, over 3041877.27 frames. ], batch size: 61, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:38:17,046 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.19 vs. limit=15.0
2023-11-22 10:38:20,580 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286300
2023-11-22 10:38:26,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1908633.3333333333, ans=0.2
2023-11-22 10:38:33,367 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.14 vs. limit=22.5
2023-11-22 10:38:41,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1908766.6666666667, ans=0.0
2023-11-22 10:38:53,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1908833.3333333333, ans=0.2
2023-11-22 10:39:15,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1908900.0, ans=0.125
2023-11-22 10:39:20,641 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9800, loss[loss=0.0557, simple_loss=0.07174, pruned_loss=0.009743, audio_tagging_loss=0.01009, over 16967.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09434, pruned_loss=0.01515, audio_tagging_loss=0.00926, over 3043647.15 frames. ], batch size: 65, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:39:20,889 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:39:23,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1908966.6666666667, ans=0.0
2023-11-22 10:39:24,327 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286350
2023-11-22 10:39:38,616 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.75 vs. limit=15.0
2023-11-22 10:39:43,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.37 vs. limit=10.0
2023-11-22 10:39:51,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1909100.0, ans=0.2
2023-11-22 10:40:08,396 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.345e+01 8.979e+01 9.748e+01 1.178e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-22 10:40:18,760 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:40:25,478 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9850, loss[loss=0.06441, simple_loss=0.09435, pruned_loss=0.008757, audio_tagging_loss=0.008476, over 16150.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09521, pruned_loss=0.01528, audio_tagging_loss=0.009199, over 3045186.36 frames. ], batch size: 60, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:40:28,575 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.38 vs. limit=15.0
2023-11-22 10:40:29,235 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286400
2023-11-22 10:41:01,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1909433.3333333333, ans=0.2
2023-11-22 10:41:09,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1909500.0, ans=0.125
2023-11-22 10:41:16,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1909566.6666666667, ans=0.0
2023-11-22 10:41:18,190 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.88 vs. limit=15.0
2023-11-22 10:41:21,787 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.25 vs. limit=15.0
2023-11-22 10:41:28,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.03 vs. limit=10.0
2023-11-22 10:41:30,057 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9900, loss[loss=0.0829, simple_loss=0.115, pruned_loss=0.01612, audio_tagging_loss=0.009305, over 15989.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09556, pruned_loss=0.01524, audio_tagging_loss=0.009064, over 3043011.04 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:41:30,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1909633.3333333333, ans=0.125
2023-11-22 10:41:31,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1909633.3333333333, ans=0.05
2023-11-22 10:41:34,482 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286450
2023-11-22 10:41:38,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1909633.3333333333, ans=0.025
2023-11-22 10:41:40,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1909633.3333333333, ans=0.125
2023-11-22 10:42:08,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1909833.3333333333, ans=0.125
2023-11-22 10:42:15,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1909833.3333333333, ans=0.0
2023-11-22 10:42:19,149 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.286e+01 8.212e+01 8.837e+01 9.450e+01 1.215e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 10:42:28,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1909900.0, ans=0.125
2023-11-22 10:42:34,527 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 9950, loss[loss=0.05525, simple_loss=0.06594, pruned_loss=0.009631, audio_tagging_loss=0.01265, over 15015.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09453, pruned_loss=0.01514, audio_tagging_loss=0.009199, over 3046574.81 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:42:34,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1909966.6666666667, ans=0.125
2023-11-22 10:42:38,297 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286500
2023-11-22 10:42:57,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1910033.3333333333, ans=0.125
2023-11-22 10:43:15,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1910166.6666666667, ans=0.0
2023-11-22 10:43:39,015 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10000, loss[loss=0.0558, simple_loss=0.07397, pruned_loss=0.009309, audio_tagging_loss=0.009509, over 15454.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09512, pruned_loss=0.0152, audio_tagging_loss=0.009141, over 3047932.55 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:43:42,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286550
2023-11-22 10:43:52,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1910366.6666666667, ans=0.2
2023-11-22 10:43:55,378 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.89 vs. limit=15.0
2023-11-22 10:43:56,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1910366.6666666667, ans=0.125
2023-11-22 10:44:20,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1910500.0, ans=0.0
2023-11-22 10:44:25,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=1910500.0, ans=0.025
2023-11-22 10:44:27,761 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.515e+01 8.076e+01 8.734e+01 9.444e+01 1.191e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 10:44:31,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1910566.6666666667, ans=0.125
2023-11-22 10:44:34,700 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.87 vs. limit=15.0
2023-11-22 10:44:37,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1910566.6666666667, ans=0.0
2023-11-22 10:44:43,790 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10050, loss[loss=0.09516, simple_loss=0.1306, pruned_loss=0.01973, audio_tagging_loss=0.01012, over 15277.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09569, pruned_loss=0.0154, audio_tagging_loss=0.009112, over 3057392.06 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:44:47,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286600
2023-11-22 10:44:53,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1910633.3333333333, ans=0.5
2023-11-22 10:44:53,900 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.87 vs. limit=22.5
2023-11-22 10:45:18,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1910766.6666666667, ans=0.125
2023-11-22 10:45:42,359 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:45:48,209 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10100, loss[loss=0.09703, simple_loss=0.1265, pruned_loss=0.02501, audio_tagging_loss=0.008763, over 14787.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.0957, pruned_loss=0.01556, audio_tagging_loss=0.009142, over 3050582.59 frames. ], batch size: 54, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:45:52,061 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286650
2023-11-22 10:46:27,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1911166.6666666667, ans=0.95
2023-11-22 10:46:38,287 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.221e+01 8.968e+01 9.727e+01 1.162e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 10:46:40,864 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:46:47,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1911233.3333333333, ans=0.125
2023-11-22 10:46:52,453 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10150, loss[loss=0.06466, simple_loss=0.08587, pruned_loss=0.01366, audio_tagging_loss=0.008064, over 15978.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09498, pruned_loss=0.01546, audio_tagging_loss=0.009222, over 3048220.83 frames. ], batch size: 60, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:46:52,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1911300.0, ans=0.125
2023-11-22 10:46:56,344 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286700
2023-11-22 10:46:56,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1911300.0, ans=0.0
2023-11-22 10:47:23,819 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:47:56,802 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10200, loss[loss=0.0798, simple_loss=0.1089, pruned_loss=0.01765, audio_tagging_loss=0.00768, over 15802.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09435, pruned_loss=0.0153, audio_tagging_loss=0.009332, over 3046964.53 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:47:58,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1911633.3333333333, ans=0.0
2023-11-22 10:47:59,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1911633.3333333333, ans=0.09899494936611666
2023-11-22 10:47:59,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1911633.3333333333, ans=0.0
2023-11-22 10:48:00,545 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286750
2023-11-22 10:48:01,905 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:48:20,863 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 10:48:24,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1911766.6666666667, ans=0.1
2023-11-22 10:48:28,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1911766.6666666667, ans=0.0
2023-11-22 10:48:36,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1911833.3333333333, ans=0.0
2023-11-22 10:48:46,801 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.248e+01 8.751e+01 9.441e+01 1.217e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 10:48:52,627 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.36 vs. limit=15.0
2023-11-22 10:48:53,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1911900.0, ans=0.1
2023-11-22 10:49:01,128 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10250, loss[loss=0.07514, simple_loss=0.1064, pruned_loss=0.01253, audio_tagging_loss=0.009424, over 15708.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09454, pruned_loss=0.01529, audio_tagging_loss=0.009332, over 3053963.63 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:49:04,947 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286800
2023-11-22 10:49:31,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1912100.0, ans=0.125
2023-11-22 10:49:34,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1912100.0, ans=0.0
2023-11-22 10:49:46,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1912166.6666666667, ans=0.2
2023-11-22 10:49:58,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1912233.3333333333, ans=0.125
2023-11-22 10:50:05,638 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10300, loss[loss=0.07622, simple_loss=0.09819, pruned_loss=0.01777, audio_tagging_loss=0.009351, over 14216.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09365, pruned_loss=0.01512, audio_tagging_loss=0.009402, over 3051437.00 frames. ], batch size: 54, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:50:09,992 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286850
2023-11-22 10:50:10,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1912300.0, ans=0.0
2023-11-22 10:50:22,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1912366.6666666667, ans=0.0
2023-11-22 10:50:47,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1912500.0, ans=0.125
2023-11-22 10:50:53,115 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.99 vs. limit=10.0
2023-11-22 10:50:56,529 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 8.119e+01 8.625e+01 9.295e+01 1.228e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 10:51:04,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=1912566.6666666667, ans=0.05
2023-11-22 10:51:09,906 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10350, loss[loss=0.08223, simple_loss=0.1216, pruned_loss=0.01577, audio_tagging_loss=0.005661, over 15711.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09479, pruned_loss=0.01524, audio_tagging_loss=0.009384, over 3056068.22 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 10:51:14,293 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286900
2023-11-22 10:51:15,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1912633.3333333333, ans=0.04949747468305833
2023-11-22 10:51:15,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.54 vs. limit=22.5
2023-11-22 10:51:29,911 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=12.0
2023-11-22 10:51:34,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1912700.0, ans=0.125
2023-11-22 10:51:49,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1912833.3333333333, ans=0.05
2023-11-22 10:52:15,710 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10400, loss[loss=0.04651, simple_loss=0.0576, pruned_loss=0.007393, audio_tagging_loss=0.01031, over 14531.00 frames. ], tot_loss[loss=0.07198, simple_loss=0.09459, pruned_loss=0.01522, audio_tagging_loss=0.009469, over 3053101.29 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:52:19,493 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 286950
2023-11-22 10:52:29,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1913033.3333333333, ans=0.125
2023-11-22 10:52:51,349 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.12 vs. limit=6.0
2023-11-22 10:53:04,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1913166.6666666667, ans=0.125
2023-11-22 10:53:05,472 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.858e+01 8.131e+01 8.865e+01 9.715e+01 1.299e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 10:53:07,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1913233.3333333333, ans=0.0
2023-11-22 10:53:16,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1913233.3333333333, ans=0.125
2023-11-22 10:53:18,889 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10450, loss[loss=0.07112, simple_loss=0.0924, pruned_loss=0.0151, audio_tagging_loss=0.009816, over 16104.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09417, pruned_loss=0.01511, audio_tagging_loss=0.009472, over 3057880.52 frames. ], batch size: 63, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:53:20,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1913300.0, ans=0.2
2023-11-22 10:53:23,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287000
2023-11-22 10:53:40,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1913366.6666666667, ans=0.2
2023-11-22 10:53:57,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1913500.0, ans=0.125
2023-11-22 10:54:09,305 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0
2023-11-22 10:54:11,011 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0
2023-11-22 10:54:22,310 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10500, loss[loss=0.06388, simple_loss=0.08251, pruned_loss=0.01107, audio_tagging_loss=0.01155, over 15411.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09443, pruned_loss=0.01544, audio_tagging_loss=0.009346, over 3052786.58 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:54:26,102 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287050
2023-11-22 10:54:28,629 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.33 vs. limit=15.0
2023-11-22 10:54:57,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1913766.6666666667, ans=0.05
2023-11-22 10:55:01,489 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.14 vs. limit=10.0
2023-11-22 10:55:13,002 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.563e+01 8.123e+01 8.688e+01 9.437e+01 1.274e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 10:55:23,946 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:55:28,271 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10550, loss[loss=0.07097, simple_loss=0.09539, pruned_loss=0.01584, audio_tagging_loss=0.007428, over 14799.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09481, pruned_loss=0.01553, audio_tagging_loss=0.009277, over 3056026.89 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:55:32,760 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287100
2023-11-22 10:55:32,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1913966.6666666667, ans=0.2
2023-11-22 10:55:53,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1914100.0, ans=0.125
2023-11-22 10:55:56,848 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:55:56,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1914100.0, ans=10.0
2023-11-22 10:56:08,940 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-22 10:56:21,732 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.42 vs. limit=15.0
2023-11-22 10:56:33,346 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10600, loss[loss=0.07578, simple_loss=0.1002, pruned_loss=0.0187, audio_tagging_loss=0.006997, over 15365.00 frames. ], tot_loss[loss=0.07163, simple_loss=0.09414, pruned_loss=0.01535, audio_tagging_loss=0.009211, over 3054413.30 frames. ], batch size: 60, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:56:34,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1914300.0, ans=0.04949747468305833
2023-11-22 10:56:37,225 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287150
2023-11-22 10:56:42,651 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:57:23,289 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.829e+01 8.242e+01 9.024e+01 9.598e+01 1.340e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-22 10:57:26,826 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.10 vs. limit=15.0
2023-11-22 10:57:37,625 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10650, loss[loss=0.06349, simple_loss=0.08364, pruned_loss=0.01142, audio_tagging_loss=0.01025, over 15331.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09365, pruned_loss=0.01517, audio_tagging_loss=0.009206, over 3058864.07 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:57:41,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287200
2023-11-22 10:57:43,911 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.34 vs. limit=22.5
2023-11-22 10:57:46,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1914633.3333333333, ans=0.0
2023-11-22 10:57:52,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1914700.0, ans=0.0
2023-11-22 10:57:52,679 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:57:53,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1914700.0, ans=0.125
2023-11-22 10:57:58,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1914700.0, ans=0.125
2023-11-22 10:58:03,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1914766.6666666667, ans=0.0
2023-11-22 10:58:11,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1914766.6666666667, ans=0.0
2023-11-22 10:58:19,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1914833.3333333333, ans=0.07
2023-11-22 10:58:22,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1914833.3333333333, ans=0.0
2023-11-22 10:58:42,746 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10700, loss[loss=0.08724, simple_loss=0.1147, pruned_loss=0.02086, audio_tagging_loss=0.009031, over 15345.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09228, pruned_loss=0.01496, audio_tagging_loss=0.009254, over 3053806.09 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:58:47,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287250
2023-11-22 10:58:49,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1914966.6666666667, ans=0.2
2023-11-22 10:59:02,740 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:59:03,014 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.57 vs. limit=15.0
2023-11-22 10:59:09,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1915100.0, ans=0.0
2023-11-22 10:59:33,548 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 8.175e+01 8.915e+01 9.385e+01 1.445e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-22 10:59:33,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1915233.3333333333, ans=0.0
2023-11-22 10:59:43,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1915233.3333333333, ans=0.125
2023-11-22 10:59:46,618 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 10:59:47,533 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10750, loss[loss=0.1033, simple_loss=0.1343, pruned_loss=0.02866, audio_tagging_loss=0.007509, over 14719.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09334, pruned_loss=0.01514, audio_tagging_loss=0.009185, over 3051904.95 frames. ], batch size: 53, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 10:59:51,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287300
2023-11-22 10:59:58,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1915300.0, ans=0.125
2023-11-22 11:00:06,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1915366.6666666667, ans=0.125
2023-11-22 11:00:16,014 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:00:26,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1915500.0, ans=0.1
2023-11-22 11:00:29,204 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.76 vs. limit=15.0
2023-11-22 11:00:34,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1915500.0, ans=0.025
2023-11-22 11:00:38,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1915566.6666666667, ans=0.1
2023-11-22 11:00:39,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1915566.6666666667, ans=0.125
2023-11-22 11:00:51,827 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10800, loss[loss=0.083, simple_loss=0.1132, pruned_loss=0.01723, audio_tagging_loss=0.009185, over 15424.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09347, pruned_loss=0.01519, audio_tagging_loss=0.009236, over 3053982.85 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:00:55,591 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287350
2023-11-22 11:01:00,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1915633.3333333333, ans=0.125
2023-11-22 11:01:36,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1915833.3333333333, ans=0.1
2023-11-22 11:01:42,406 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.103e+01 8.562e+01 9.437e+01 1.159e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-22 11:01:47,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1915900.0, ans=0.125
2023-11-22 11:01:54,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1915900.0, ans=0.125
2023-11-22 11:01:56,457 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10850, loss[loss=0.06444, simple_loss=0.088, pruned_loss=0.009734, audio_tagging_loss=0.0107, over 14600.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09371, pruned_loss=0.01504, audio_tagging_loss=0.009286, over 3049824.84 frames. ], batch size: 53, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:02:00,909 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287400
2023-11-22 11:02:02,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.73 vs. limit=15.0
2023-11-22 11:02:12,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1916033.3333333333, ans=0.125
2023-11-22 11:02:46,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1916166.6666666667, ans=0.2
2023-11-22 11:02:57,558 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:03:01,198 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10900, loss[loss=0.09308, simple_loss=0.1234, pruned_loss=0.0219, audio_tagging_loss=0.009495, over 16077.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09401, pruned_loss=0.01489, audio_tagging_loss=0.009234, over 3054160.94 frames. ], batch size: 59, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:03:04,967 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287450
2023-11-22 11:03:05,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=1916300.0, ans=0.125
2023-11-22 11:03:08,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1916300.0, ans=0.125
2023-11-22 11:03:22,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1916366.6666666667, ans=0.125
2023-11-22 11:03:27,940 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.04 vs. limit=22.5
2023-11-22 11:03:52,815 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.229e+01 8.892e+01 9.728e+01 1.443e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 11:04:04,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1916566.6666666667, ans=0.125
2023-11-22 11:04:06,222 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 10950, loss[loss=0.08643, simple_loss=0.1148, pruned_loss=0.01957, audio_tagging_loss=0.009468, over 14898.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09421, pruned_loss=0.01511, audio_tagging_loss=0.009243, over 3049209.68 frames. ], batch size: 54, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:04:10,100 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287500
2023-11-22 11:04:52,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=1916833.3333333333, ans=0.125
2023-11-22 11:04:52,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=1916833.3333333333, ans=0.2
2023-11-22 11:05:10,034 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11000, loss[loss=0.07053, simple_loss=0.08631, pruned_loss=0.01401, audio_tagging_loss=0.01336, over 15463.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09408, pruned_loss=0.01494, audio_tagging_loss=0.009369, over 3054127.60 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:05:13,902 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287550
2023-11-22 11:05:21,789 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:05:26,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1917033.3333333333, ans=0.125
2023-11-22 11:05:26,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1917033.3333333333, ans=0.1
2023-11-22 11:05:30,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1917033.3333333333, ans=0.0
2023-11-22 11:05:50,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1917166.6666666667, ans=0.125
2023-11-22 11:06:01,247 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.398e+01 8.039e+01 8.665e+01 9.378e+01 1.568e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 11:06:14,094 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11050, loss[loss=0.06886, simple_loss=0.091, pruned_loss=0.01241, audio_tagging_loss=0.01096, over 15620.00 frames. ], tot_loss[loss=0.07238, simple_loss=0.09539, pruned_loss=0.0154, audio_tagging_loss=0.009284, over 3059981.04 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:06:14,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1917300.0, ans=0.04949747468305833
2023-11-22 11:06:17,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287600
2023-11-22 11:06:18,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1917300.0, ans=0.04949747468305833
2023-11-22 11:06:19,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1917300.0, ans=0.125
2023-11-22 11:06:23,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1917300.0, ans=0.0
2023-11-22 11:06:30,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1917366.6666666667, ans=0.0
2023-11-22 11:06:36,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1917366.6666666667, ans=0.0
2023-11-22 11:06:50,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=21.44 vs. limit=22.5
2023-11-22 11:06:57,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1917500.0, ans=0.1
2023-11-22 11:07:04,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1917566.6666666667, ans=0.0
2023-11-22 11:07:12,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1917566.6666666667, ans=0.125
2023-11-22 11:07:14,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1917566.6666666667, ans=0.125
2023-11-22 11:07:18,053 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11100, loss[loss=0.07525, simple_loss=0.09848, pruned_loss=0.01401, audio_tagging_loss=0.01201, over 16753.00 frames. ], tot_loss[loss=0.07246, simple_loss=0.09537, pruned_loss=0.01536, audio_tagging_loss=0.009413, over 3060156.56 frames. ], batch size: 62, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:07:20,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1917633.3333333333, ans=0.125
2023-11-22 11:07:22,397 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287650
2023-11-22 11:07:25,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1917633.3333333333, ans=0.125
2023-11-22 11:07:26,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1917633.3333333333, ans=0.0
2023-11-22 11:08:09,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.56 vs. limit=22.5
2023-11-22 11:08:09,640 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.844e+01 8.222e+01 9.043e+01 9.663e+01 1.186e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-22 11:08:17,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1917900.0, ans=0.125
2023-11-22 11:08:20,412 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.87 vs. limit=15.0
2023-11-22 11:08:22,596 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11150, loss[loss=0.07224, simple_loss=0.08898, pruned_loss=0.01716, audio_tagging_loss=0.0106, over 15674.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09424, pruned_loss=0.01515, audio_tagging_loss=0.009588, over 3056132.49 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:08:26,281 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287700
2023-11-22 11:08:44,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1918033.3333333333, ans=0.125
2023-11-22 11:09:01,767 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-22 11:09:05,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.16 vs. limit=10.0
2023-11-22 11:09:23,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1918233.3333333333, ans=0.0
2023-11-22 11:09:26,707 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11200, loss[loss=0.07508, simple_loss=0.1016, pruned_loss=0.01401, audio_tagging_loss=0.01024, over 15488.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.095, pruned_loss=0.01534, audio_tagging_loss=0.009575, over 3056417.23 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:09:30,506 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287750
2023-11-22 11:09:32,404 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.25 vs. limit=6.0
2023-11-22 11:09:40,755 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.708e-03
2023-11-22 11:09:56,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1918433.3333333333, ans=0.125
2023-11-22 11:10:04,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1918500.0, ans=0.0
2023-11-22 11:10:14,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1918500.0, ans=0.1
2023-11-22 11:10:15,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1918500.0, ans=0.2
2023-11-22 11:10:17,763 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.119e+01 8.627e+01 9.513e+01 1.631e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 11:10:30,721 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11250, loss[loss=0.06074, simple_loss=0.0868, pruned_loss=0.01032, audio_tagging_loss=0.007022, over 15650.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09507, pruned_loss=0.01529, audio_tagging_loss=0.009502, over 3057122.08 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:10:34,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287800
2023-11-22 11:10:57,586 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.33 vs. limit=22.5
2023-11-22 11:11:02,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1918766.6666666667, ans=0.0
2023-11-22 11:11:29,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1918900.0, ans=0.1
2023-11-22 11:11:35,618 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11300, loss[loss=0.08256, simple_loss=0.1118, pruned_loss=0.01814, audio_tagging_loss=0.0085, over 15043.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09515, pruned_loss=0.01521, audio_tagging_loss=0.009385, over 3053274.99 frames. ], batch size: 55, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:11:39,980 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287850
2023-11-22 11:11:55,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1919033.3333333333, ans=0.0
2023-11-22 11:12:23,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1919166.6666666667, ans=0.125
2023-11-22 11:12:28,510 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.163e+01 8.307e+01 9.060e+01 9.651e+01 1.397e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-22 11:12:38,425 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.12 vs. limit=15.0
2023-11-22 11:12:40,205 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11350, loss[loss=0.07216, simple_loss=0.101, pruned_loss=0.01313, audio_tagging_loss=0.008525, over 15724.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09542, pruned_loss=0.01544, audio_tagging_loss=0.00928, over 3052518.09 frames. ], batch size: 57, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:12:44,147 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287900
2023-11-22 11:12:50,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1919300.0, ans=0.0
2023-11-22 11:12:54,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1919366.6666666667, ans=0.125
2023-11-22 11:13:13,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1919433.3333333333, ans=0.125
2023-11-22 11:13:21,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=1919500.0, ans=0.05
2023-11-22 11:13:23,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1919500.0, ans=0.0
2023-11-22 11:13:26,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1919500.0, ans=0.1
2023-11-22 11:13:26,841 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:13:30,891 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.30 vs. limit=22.5
2023-11-22 11:13:31,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1919566.6666666667, ans=0.1
2023-11-22 11:13:36,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=1919566.6666666667, ans=0.02
2023-11-22 11:13:43,686 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11400, loss[loss=0.06679, simple_loss=0.08229, pruned_loss=0.01619, audio_tagging_loss=0.009454, over 15458.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09522, pruned_loss=0.01547, audio_tagging_loss=0.009204, over 3049441.78 frames. ], batch size: 58, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:13:47,967 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 287950
2023-11-22 11:13:56,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1919700.0, ans=0.2
2023-11-22 11:14:06,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1919700.0, ans=0.0
2023-11-22 11:14:15,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1919766.6666666667, ans=0.0
2023-11-22 11:14:20,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1919766.6666666667, ans=0.125
2023-11-22 11:14:28,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1919833.3333333333, ans=0.125
2023-11-22 11:14:30,209 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.89 vs. limit=15.0
2023-11-22 11:14:32,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1919833.3333333333, ans=0.0
2023-11-22 11:14:33,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.50 vs. limit=12.0
2023-11-22 11:14:36,226 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.311e+01 8.086e+01 8.647e+01 9.538e+01 1.200e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 11:14:40,385 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.98 vs. limit=15.0
2023-11-22 11:14:44,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1919900.0, ans=0.035
2023-11-22 11:14:47,085 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11450, loss[loss=0.05739, simple_loss=0.07295, pruned_loss=0.01118, audio_tagging_loss=0.009743, over 15632.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09443, pruned_loss=0.01525, audio_tagging_loss=0.009198, over 3047623.82 frames. ], batch size: 61, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:14:50,785 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288000
2023-11-22 11:14:52,260 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-288000.pt
2023-11-22 11:15:00,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1919966.6666666667, ans=0.0
2023-11-22 11:15:20,136 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.78 vs. limit=22.5
2023-11-22 11:15:55,753 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11500, loss[loss=0.06222, simple_loss=0.08366, pruned_loss=0.01272, audio_tagging_loss=0.00766, over 15120.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09401, pruned_loss=0.01526, audio_tagging_loss=0.009194, over 3046066.14 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:15:56,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1920300.0, ans=0.2
2023-11-22 11:15:59,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288050
2023-11-22 11:16:04,763 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.22 vs. limit=15.0
2023-11-22 11:16:17,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1920366.6666666667, ans=0.09899494936611666
2023-11-22 11:16:20,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1920433.3333333333, ans=0.125
2023-11-22 11:16:22,235 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:16:47,945 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.254e+01 8.955e+01 9.456e+01 2.559e+02, threshold=1.791e+02, percent-clipped=1.0
2023-11-22 11:16:49,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1920566.6666666667, ans=0.125
2023-11-22 11:16:58,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1920633.3333333333, ans=0.0
2023-11-22 11:16:59,002 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11550, loss[loss=0.0856, simple_loss=0.1232, pruned_loss=0.01721, audio_tagging_loss=0.006808, over 15547.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09365, pruned_loss=0.01515, audio_tagging_loss=0.009276, over 3046861.96 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:17:02,688 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288100
2023-11-22 11:17:12,240 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.44 vs. limit=6.0
2023-11-22 11:17:28,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1920766.6666666667, ans=0.2
2023-11-22 11:17:34,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1920766.6666666667, ans=0.125
2023-11-22 11:17:34,325 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:17:36,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1920833.3333333333, ans=0.1
2023-11-22 11:17:37,690 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:17:38,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1920833.3333333333, ans=0.125
2023-11-22 11:17:48,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1920900.0, ans=0.1
2023-11-22 11:18:02,505 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11600, loss[loss=0.07133, simple_loss=0.09176, pruned_loss=0.01203, audio_tagging_loss=0.01342, over 15511.00 frames. ], tot_loss[loss=0.07135, simple_loss=0.09383, pruned_loss=0.0152, audio_tagging_loss=0.009234, over 3049808.11 frames. ], batch size: 63, lr: 2.82e-03, grad_scale: 32.0
2023-11-22 11:18:06,166 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288150
2023-11-22 11:18:16,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1921033.3333333333, ans=0.125
2023-11-22 11:18:37,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1921100.0, ans=0.0
2023-11-22 11:18:43,180 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.70 vs. limit=22.5
2023-11-22 11:18:49,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1921166.6666666667, ans=0.125
2023-11-22 11:18:56,245 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.284e+01 8.414e+01 8.937e+01 9.626e+01 1.842e+02, threshold=1.787e+02, percent-clipped=1.0
2023-11-22 11:19:07,325 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11650, loss[loss=0.09146, simple_loss=0.111, pruned_loss=0.02647, audio_tagging_loss=0.009467, over 14327.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09372, pruned_loss=0.01515, audio_tagging_loss=0.009273, over 3046704.57 frames. ], batch size: 56, lr: 2.82e-03, grad_scale: 16.0
2023-11-22 11:19:07,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1921300.0, ans=0.125
2023-11-22 11:19:11,761 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288200
2023-11-22 11:19:19,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1921366.6666666667, ans=0.125
2023-11-22 11:19:35,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=1921433.3333333333, ans=0.125
2023-11-22 11:19:49,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.53 vs. limit=10.0
2023-11-22 11:19:53,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1921500.0, ans=0.125
2023-11-22 11:20:05,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1921566.6666666667, ans=0.2
2023-11-22 11:20:11,341 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11700, loss[loss=0.05335, simple_loss=0.06477, pruned_loss=0.01083, audio_tagging_loss=0.01014, over 14172.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.0945, pruned_loss=0.01541, audio_tagging_loss=0.009212, over 3047229.78 frames. ], batch size: 55, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:20:14,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1921633.3333333333, ans=0.1
2023-11-22 11:20:15,013 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288250
2023-11-22 11:20:33,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1921700.0, ans=0.2
2023-11-22 11:20:45,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1921766.6666666667, ans=0.1
2023-11-22 11:20:55,753 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.75 vs. limit=15.0
2023-11-22 11:21:00,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1921833.3333333333, ans=0.125
2023-11-22 11:21:04,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1921900.0, ans=0.1
2023-11-22 11:21:05,065 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.380e+01 8.961e+01 9.495e+01 1.206e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 11:21:07,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1921900.0, ans=0.2
2023-11-22 11:21:13,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1921900.0, ans=0.125
2023-11-22 11:21:14,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1921966.6666666667, ans=0.125
2023-11-22 11:21:15,681 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11750, loss[loss=0.08196, simple_loss=0.1081, pruned_loss=0.01777, audio_tagging_loss=0.01013, over 14637.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09486, pruned_loss=0.01542, audio_tagging_loss=0.009184, over 3047717.16 frames. ], batch size: 54, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:21:19,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288300
2023-11-22 11:21:24,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1921966.6666666667, ans=0.125
2023-11-22 11:21:30,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1922033.3333333333, ans=0.125
2023-11-22 11:21:45,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1922100.0, ans=0.125
2023-11-22 11:21:51,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1922100.0, ans=0.0
2023-11-22 11:21:56,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1922166.6666666667, ans=0.2
2023-11-22 11:22:00,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1922166.6666666667, ans=0.0
2023-11-22 11:22:02,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1922166.6666666667, ans=0.125
2023-11-22 11:22:04,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1922166.6666666667, ans=0.0
2023-11-22 11:22:12,538 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:22:20,069 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11800, loss[loss=0.06137, simple_loss=0.07792, pruned_loss=0.01363, audio_tagging_loss=0.008779, over 14812.00 frames. ], tot_loss[loss=0.07211, simple_loss=0.0949, pruned_loss=0.01545, audio_tagging_loss=0.009217, over 3043723.13 frames. ], batch size: 57, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:22:24,531 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288350
2023-11-22 11:22:24,945 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.10 vs. limit=22.5
2023-11-22 11:22:28,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1922300.0, ans=0.125
2023-11-22 11:22:51,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1922433.3333333333, ans=0.125
2023-11-22 11:22:52,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1922433.3333333333, ans=0.125
2023-11-22 11:23:00,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1922500.0, ans=0.125
2023-11-22 11:23:00,424 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.54 vs. limit=22.5
2023-11-22 11:23:02,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1922500.0, ans=0.035
2023-11-22 11:23:09,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1922500.0, ans=0.2
2023-11-22 11:23:13,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1922566.6666666667, ans=0.125
2023-11-22 11:23:14,676 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.163e+01 8.779e+01 9.476e+01 1.167e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 11:23:16,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1922566.6666666667, ans=0.125
2023-11-22 11:23:20,114 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.91 vs. limit=15.0
2023-11-22 11:23:25,275 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11850, loss[loss=0.07346, simple_loss=0.08704, pruned_loss=0.01885, audio_tagging_loss=0.01109, over 16861.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.09467, pruned_loss=0.0153, audio_tagging_loss=0.0093, over 3048718.71 frames. ], batch size: 65, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:23:29,145 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288400
2023-11-22 11:23:33,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1922633.3333333333, ans=0.125
2023-11-22 11:24:15,826 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:24:18,803 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.41 vs. limit=15.0
2023-11-22 11:24:24,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1922900.0, ans=0.1
2023-11-22 11:24:29,695 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11900, loss[loss=0.06297, simple_loss=0.07556, pruned_loss=0.01477, audio_tagging_loss=0.01042, over 15011.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09433, pruned_loss=0.01516, audio_tagging_loss=0.00946, over 3050754.25 frames. ], batch size: 59, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:24:33,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288450
2023-11-22 11:24:49,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.58 vs. limit=15.0
2023-11-22 11:25:23,681 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.983e+01 8.115e+01 8.834e+01 9.649e+01 1.264e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 11:25:34,098 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 11950, loss[loss=0.06371, simple_loss=0.08054, pruned_loss=0.01417, audio_tagging_loss=0.00926, over 15975.00 frames. ], tot_loss[loss=0.07187, simple_loss=0.09434, pruned_loss=0.01522, audio_tagging_loss=0.009473, over 3050689.76 frames. ], batch size: 58, lr: 2.81e-03, grad_scale: 16.0
2023-11-22 11:25:37,895 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288500
2023-11-22 11:25:43,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1923300.0, ans=0.125
2023-11-22 11:25:46,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1923366.6666666667, ans=0.0
2023-11-22 11:25:46,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1923366.6666666667, ans=0.04949747468305833
2023-11-22 11:25:49,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=1923366.6666666667, ans=0.125
2023-11-22 11:26:04,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.22 vs. limit=22.5
2023-11-22 11:26:18,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1923500.0, ans=0.125
2023-11-22 11:26:21,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1923500.0, ans=0.0
2023-11-22 11:26:23,138 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:26:27,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1923566.6666666667, ans=0.1
2023-11-22 11:26:35,557 INFO [train_asr.py:1221] (0/4) Epoch 24, batch 12000, loss[loss=0.0532, simple_loss=0.06511, pruned_loss=0.01023, audio_tagging_loss=0.01042, over 13842.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.09371, pruned_loss=0.01516, audio_tagging_loss=0.009564, over 3051632.87 frames. ], batch size: 52, lr: 2.81e-03, grad_scale: 32.0
2023-11-22 11:26:35,561 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 11:27:13,105 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.1741, 2.4008, 4.9996, 2.9052], device='cuda:0')
2023-11-22 11:27:13,350 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3335, 5.0164, 4.6838, 5.1414], device='cuda:0')
2023-11-22 11:27:17,217 INFO [train_asr.py:1253] (0/4) Epoch 24, validation: loss=0.05896, simple_loss=0.05166, pruned_loss=0.00516, audio_tagging_loss=0.02797, over 4681554.00 frames. 
2023-11-22 11:27:17,218 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 11:27:20,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.25 vs. limit=8.0
2023-11-22 11:27:20,722 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288550
2023-11-22 11:27:24,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1923633.3333333333, ans=0.0
2023-11-22 11:27:46,698 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-24.pt
2023-11-22 11:28:19,671 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 0, loss[loss=0.07795, simple_loss=0.0887, pruned_loss=0.01223, audio_tagging_loss=0.02137, over 15139.00 frames. ], tot_loss[loss=0.07795, simple_loss=0.0887, pruned_loss=0.01223, audio_tagging_loss=0.02137, over 15139.00 frames. ], batch size: 57, lr: 2.76e-03, grad_scale: 32.0
2023-11-22 11:28:19,674 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 11:28:55,812 INFO [train_asr.py:1253] (0/4) Epoch 25, validation: loss=0.05903, simple_loss=0.05164, pruned_loss=0.005146, audio_tagging_loss=0.02807, over 4681554.00 frames. 
2023-11-22 11:28:55,813 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 11:29:18,964 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.611e+01 9.501e+01 1.042e+02 1.380e+02, threshold=1.900e+02, percent-clipped=0.0
2023-11-22 11:29:19,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1923860.0, ans=0.09899494936611666
2023-11-22 11:29:24,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1923926.6666666667, ans=0.125
2023-11-22 11:29:29,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1923926.6666666667, ans=0.0
2023-11-22 11:29:34,043 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288600
2023-11-22 11:29:51,544 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0
2023-11-22 11:29:53,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1924060.0, ans=0.1
2023-11-22 11:29:58,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1924060.0, ans=0.125
2023-11-22 11:30:00,647 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 50, loss[loss=0.07212, simple_loss=0.07526, pruned_loss=0.01649, audio_tagging_loss=0.01799, over 16466.00 frames. ], tot_loss[loss=0.07945, simple_loss=0.09269, pruned_loss=0.01483, audio_tagging_loss=0.01828, over 684300.55 frames. ], batch size: 62, lr: 2.76e-03, grad_scale: 32.0
2023-11-22 11:30:03,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1924126.6666666667, ans=0.0
2023-11-22 11:30:38,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288650
2023-11-22 11:30:53,496 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.39 vs. limit=15.0
2023-11-22 11:31:04,565 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 100, loss[loss=0.0692, simple_loss=0.08003, pruned_loss=0.01344, audio_tagging_loss=0.01574, over 15377.00 frames. ], tot_loss[loss=0.07894, simple_loss=0.0924, pruned_loss=0.01507, audio_tagging_loss=0.01767, over 1205483.17 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:31:23,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=1924526.6666666667, ans=0.0
2023-11-22 11:31:28,378 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.104e+01 8.752e+01 9.299e+01 1.007e+02 1.330e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-22 11:31:42,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288700
2023-11-22 11:32:10,041 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 150, loss[loss=0.06411, simple_loss=0.08259, pruned_loss=0.01214, audio_tagging_loss=0.01068, over 15232.00 frames. ], tot_loss[loss=0.07636, simple_loss=0.09122, pruned_loss=0.01492, audio_tagging_loss=0.01583, over 1611091.97 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:32:24,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1924860.0, ans=0.125
2023-11-22 11:32:32,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=1924860.0, ans=0.2
2023-11-22 11:32:47,847 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288750
2023-11-22 11:33:14,448 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 200, loss[loss=0.06103, simple_loss=0.07612, pruned_loss=0.01333, audio_tagging_loss=0.009645, over 14440.00 frames. ], tot_loss[loss=0.07434, simple_loss=0.09102, pruned_loss=0.01474, audio_tagging_loss=0.01409, over 1925221.35 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:33:23,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1925126.6666666667, ans=0.125
2023-11-22 11:33:32,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1925193.3333333333, ans=0.0
2023-11-22 11:33:36,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1925193.3333333333, ans=0.2
2023-11-22 11:33:37,542 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.935e+01 8.185e+01 8.728e+01 9.456e+01 1.257e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 11:33:39,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1925260.0, ans=0.2
2023-11-22 11:33:40,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1925260.0, ans=0.125
2023-11-22 11:33:42,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1925260.0, ans=0.0
2023-11-22 11:33:51,618 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288800
2023-11-22 11:34:18,346 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 250, loss[loss=0.06438, simple_loss=0.07868, pruned_loss=0.01511, audio_tagging_loss=0.009927, over 14792.00 frames. ], tot_loss[loss=0.07333, simple_loss=0.09176, pruned_loss=0.01475, audio_tagging_loss=0.01271, over 2169961.09 frames. ], batch size: 55, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:34:18,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1925460.0, ans=0.125
2023-11-22 11:34:22,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1925460.0, ans=0.125
2023-11-22 11:34:29,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1925460.0, ans=0.0
2023-11-22 11:34:51,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1925593.3333333333, ans=0.07
2023-11-22 11:34:53,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1925593.3333333333, ans=0.125
2023-11-22 11:34:53,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1925593.3333333333, ans=0.1
2023-11-22 11:34:55,543 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288850
2023-11-22 11:35:22,329 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.85 vs. limit=15.0
2023-11-22 11:35:22,890 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 300, loss[loss=0.047, simple_loss=0.05662, pruned_loss=0.009083, audio_tagging_loss=0.009612, over 13953.00 frames. ], tot_loss[loss=0.07288, simple_loss=0.09253, pruned_loss=0.01483, audio_tagging_loss=0.01178, over 2370598.35 frames. ], batch size: 54, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:35:45,375 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.100e+01 8.207e+01 8.755e+01 9.551e+01 1.256e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 11:35:49,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1925926.6666666667, ans=0.0
2023-11-22 11:35:58,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1925926.6666666667, ans=0.0
2023-11-22 11:35:59,624 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288900
2023-11-22 11:36:13,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1926060.0, ans=0.125
2023-11-22 11:36:27,378 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 350, loss[loss=0.04532, simple_loss=0.05712, pruned_loss=0.008301, audio_tagging_loss=0.008458, over 14751.00 frames. ], tot_loss[loss=0.07295, simple_loss=0.09373, pruned_loss=0.01504, audio_tagging_loss=0.01103, over 2517893.17 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:36:44,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=1926193.3333333333, ans=0.0
2023-11-22 11:36:57,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1926260.0, ans=0.1
2023-11-22 11:36:57,810 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.92 vs. limit=15.0
2023-11-22 11:37:05,093 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 288950
2023-11-22 11:37:32,219 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 400, loss[loss=0.06268, simple_loss=0.07998, pruned_loss=0.01135, audio_tagging_loss=0.01134, over 15400.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.093, pruned_loss=0.01489, audio_tagging_loss=0.01063, over 2630779.40 frames. ], batch size: 60, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:37:46,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1926526.6666666667, ans=10.0
2023-11-22 11:37:47,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1926526.6666666667, ans=0.125
2023-11-22 11:37:51,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1926526.6666666667, ans=0.125
2023-11-22 11:37:55,732 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.456e+01 8.000e+01 8.654e+01 9.629e+01 1.287e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-22 11:37:57,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1926593.3333333333, ans=0.1
2023-11-22 11:38:01,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1926593.3333333333, ans=0.125
2023-11-22 11:38:02,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1926593.3333333333, ans=0.125
2023-11-22 11:38:03,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1926593.3333333333, ans=0.125
2023-11-22 11:38:09,861 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289000
2023-11-22 11:38:19,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1926660.0, ans=0.0
2023-11-22 11:38:22,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1926660.0, ans=0.125
2023-11-22 11:38:27,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.19 vs. limit=8.0
2023-11-22 11:38:28,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1926726.6666666667, ans=0.125
2023-11-22 11:38:37,048 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 450, loss[loss=0.08802, simple_loss=0.1176, pruned_loss=0.02306, audio_tagging_loss=0.006152, over 16475.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09205, pruned_loss=0.01478, audio_tagging_loss=0.01035, over 2728947.54 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:39:07,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1926926.6666666667, ans=0.1
2023-11-22 11:39:14,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289050
2023-11-22 11:39:18,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1926993.3333333333, ans=0.0
2023-11-22 11:39:23,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1926993.3333333333, ans=0.125
2023-11-22 11:39:23,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1926993.3333333333, ans=0.125
2023-11-22 11:39:24,892 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.33 vs. limit=22.5
2023-11-22 11:39:42,474 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 500, loss[loss=0.07418, simple_loss=0.1015, pruned_loss=0.01405, audio_tagging_loss=0.009404, over 14774.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09238, pruned_loss=0.01482, audio_tagging_loss=0.01009, over 2791266.64 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:39:46,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1927126.6666666667, ans=0.0
2023-11-22 11:39:54,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1927193.3333333333, ans=0.125
2023-11-22 11:40:05,717 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.152e+01 8.293e+01 8.940e+01 9.677e+01 1.305e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 11:40:19,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289100
2023-11-22 11:40:42,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1927393.3333333333, ans=0.0
2023-11-22 11:40:47,560 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 550, loss[loss=0.08263, simple_loss=0.09818, pruned_loss=0.02359, audio_tagging_loss=0.009949, over 14771.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.0928, pruned_loss=0.01489, audio_tagging_loss=0.00992, over 2842085.42 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:41:05,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1927526.6666666667, ans=0.125
2023-11-22 11:41:21,234 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:41:23,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1927593.3333333333, ans=0.09899494936611666
2023-11-22 11:41:24,747 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289150
2023-11-22 11:41:39,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1927726.6666666667, ans=0.035
2023-11-22 11:41:49,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1927726.6666666667, ans=0.0
2023-11-22 11:41:51,756 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 600, loss[loss=0.05863, simple_loss=0.06555, pruned_loss=0.0159, audio_tagging_loss=0.009954, over 13518.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09179, pruned_loss=0.01464, audio_tagging_loss=0.00983, over 2882541.74 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:41:58,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1927793.3333333333, ans=0.0
2023-11-22 11:42:02,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1927793.3333333333, ans=0.125
2023-11-22 11:42:16,622 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.617e+01 7.918e+01 8.734e+01 9.306e+01 1.583e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 11:42:29,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289200
2023-11-22 11:42:57,742 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 650, loss[loss=0.05191, simple_loss=0.06942, pruned_loss=0.008937, audio_tagging_loss=0.00826, over 14856.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09284, pruned_loss=0.01477, audio_tagging_loss=0.009732, over 2918425.92 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:43:00,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1928126.6666666667, ans=0.0
2023-11-22 11:43:07,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1928126.6666666667, ans=0.125
2023-11-22 11:43:09,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1928193.3333333333, ans=0.125
2023-11-22 11:43:28,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1928260.0, ans=0.0
2023-11-22 11:43:34,934 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289250
2023-11-22 11:44:01,468 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 700, loss[loss=0.07694, simple_loss=0.09919, pruned_loss=0.01842, audio_tagging_loss=0.008925, over 15145.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.0922, pruned_loss=0.01459, audio_tagging_loss=0.009624, over 2942199.24 frames. ], batch size: 55, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:44:01,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1928460.0, ans=0.2
2023-11-22 11:44:26,164 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.565e+01 8.213e+01 8.759e+01 9.279e+01 1.127e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 11:44:38,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1928593.3333333333, ans=0.1
2023-11-22 11:44:38,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1928593.3333333333, ans=0.0
2023-11-22 11:44:39,785 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289300
2023-11-22 11:44:46,305 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.08 vs. limit=15.0
2023-11-22 11:44:57,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1928726.6666666667, ans=0.1
2023-11-22 11:45:04,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1928793.3333333333, ans=0.0
2023-11-22 11:45:05,589 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 750, loss[loss=0.08592, simple_loss=0.1256, pruned_loss=0.01629, audio_tagging_loss=0.00682, over 15044.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09333, pruned_loss=0.01487, audio_tagging_loss=0.009603, over 2970090.71 frames. ], batch size: 54, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:45:18,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1928860.0, ans=0.125
2023-11-22 11:45:43,744 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289350
2023-11-22 11:46:10,235 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 800, loss[loss=0.05769, simple_loss=0.07882, pruned_loss=0.00931, audio_tagging_loss=0.008972, over 14534.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09434, pruned_loss=0.01527, audio_tagging_loss=0.009649, over 2984776.88 frames. ], batch size: 54, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:46:17,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1929126.6666666667, ans=0.2
2023-11-22 11:46:18,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=1929126.6666666667, ans=0.2
2023-11-22 11:46:32,262 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:46:36,821 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.118e+01 8.155e+01 8.807e+01 9.499e+01 1.198e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 11:46:39,620 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.765e-03
2023-11-22 11:46:42,266 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.13 vs. limit=15.0
2023-11-22 11:46:48,661 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289400
2023-11-22 11:46:49,145 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.55 vs. limit=15.0
2023-11-22 11:47:07,604 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.63 vs. limit=22.5
2023-11-22 11:47:10,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1929393.3333333333, ans=0.1
2023-11-22 11:47:16,264 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 850, loss[loss=0.07523, simple_loss=0.1014, pruned_loss=0.01678, audio_tagging_loss=0.007737, over 14769.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.09382, pruned_loss=0.01514, audio_tagging_loss=0.009737, over 2994976.74 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:47:29,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=1929526.6666666667, ans=0.125
2023-11-22 11:47:30,164 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.18 vs. limit=15.0
2023-11-22 11:47:41,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1929593.3333333333, ans=0.125
2023-11-22 11:47:54,061 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289450
2023-11-22 11:47:57,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1929660.0, ans=0.2
2023-11-22 11:48:21,278 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 900, loss[loss=0.06208, simple_loss=0.07847, pruned_loss=0.01127, audio_tagging_loss=0.01158, over 15790.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09371, pruned_loss=0.01521, audio_tagging_loss=0.00979, over 3001872.89 frames. ], batch size: 62, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:48:34,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1929860.0, ans=0.2
2023-11-22 11:48:46,853 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.150e+01 8.654e+01 9.309e+01 1.377e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-22 11:48:47,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1929926.6666666667, ans=0.125
2023-11-22 11:48:55,848 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.42 vs. limit=15.0
2023-11-22 11:48:58,883 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289500
2023-11-22 11:49:03,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1929993.3333333333, ans=0.0
2023-11-22 11:49:23,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1930060.0, ans=0.1
2023-11-22 11:49:26,023 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 950, loss[loss=0.05589, simple_loss=0.07037, pruned_loss=0.01231, audio_tagging_loss=0.008395, over 15697.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.0948, pruned_loss=0.01543, audio_tagging_loss=0.009656, over 3014735.69 frames. ], batch size: 60, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:49:26,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_na.min_abs, batch_count=1930126.6666666667, ans=0.02
2023-11-22 11:49:36,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1930126.6666666667, ans=0.125
2023-11-22 11:49:39,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1930193.3333333333, ans=0.1
2023-11-22 11:49:47,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1930193.3333333333, ans=0.125
2023-11-22 11:50:04,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289550
2023-11-22 11:50:08,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1930326.6666666667, ans=0.0
2023-11-22 11:50:10,034 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.84 vs. limit=22.5
2023-11-22 11:50:11,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.69 vs. limit=15.0
2023-11-22 11:50:19,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1930393.3333333333, ans=0.1
2023-11-22 11:50:22,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1930393.3333333333, ans=0.125
2023-11-22 11:50:23,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1930393.3333333333, ans=0.1
2023-11-22 11:50:23,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.21 vs. limit=6.0
2023-11-22 11:50:30,820 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1000, loss[loss=0.08167, simple_loss=0.1048, pruned_loss=0.01933, audio_tagging_loss=0.009931, over 16157.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09437, pruned_loss=0.0154, audio_tagging_loss=0.009495, over 3013341.27 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:50:35,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1930460.0, ans=0.1
2023-11-22 11:50:39,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=1930460.0, ans=0.125
2023-11-22 11:50:47,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1930526.6666666667, ans=0.0
2023-11-22 11:50:48,510 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 11:50:57,565 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.182e+01 8.192e+01 8.953e+01 9.947e+01 1.294e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 11:50:58,854 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:51:08,608 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289600
2023-11-22 11:51:36,495 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1050, loss[loss=0.0744, simple_loss=0.1012, pruned_loss=0.01692, audio_tagging_loss=0.006889, over 15559.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09374, pruned_loss=0.01518, audio_tagging_loss=0.009347, over 3017082.83 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:51:37,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.57 vs. limit=12.0
2023-11-22 11:51:43,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1930793.3333333333, ans=0.0
2023-11-22 11:52:07,479 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.62 vs. limit=15.0
2023-11-22 11:52:13,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289650
2023-11-22 11:52:20,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1930993.3333333333, ans=0.2
2023-11-22 11:52:40,634 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1100, loss[loss=0.05641, simple_loss=0.07215, pruned_loss=0.01224, audio_tagging_loss=0.008098, over 16279.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09357, pruned_loss=0.01516, audio_tagging_loss=0.009328, over 3026280.65 frames. ], batch size: 63, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:52:41,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1931126.6666666667, ans=0.125
2023-11-22 11:52:44,296 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:53:06,996 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.014e+01 8.517e+01 9.464e+01 1.257e+02, threshold=1.703e+02, percent-clipped=0.0
2023-11-22 11:53:18,185 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289700
2023-11-22 11:53:45,309 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1150, loss[loss=0.04714, simple_loss=0.06636, pruned_loss=0.004921, audio_tagging_loss=0.009042, over 15878.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09392, pruned_loss=0.0152, audio_tagging_loss=0.009263, over 3032499.48 frames. ], batch size: 61, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:53:52,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1931460.0, ans=0.0
2023-11-22 11:54:19,755 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.49 vs. limit=15.0
2023-11-22 11:54:21,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=1931593.3333333333, ans=0.2
2023-11-22 11:54:22,865 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289750
2023-11-22 11:54:24,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1931660.0, ans=0.125
2023-11-22 11:54:37,905 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.54 vs. limit=15.0
2023-11-22 11:54:38,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1931726.6666666667, ans=0.1
2023-11-22 11:54:41,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=1931726.6666666667, ans=0.0
2023-11-22 11:54:42,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1931726.6666666667, ans=0.1
2023-11-22 11:54:50,871 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1200, loss[loss=0.07978, simple_loss=0.1005, pruned_loss=0.01947, audio_tagging_loss=0.01003, over 16358.00 frames. ], tot_loss[loss=0.07199, simple_loss=0.09491, pruned_loss=0.01537, audio_tagging_loss=0.009163, over 3037428.42 frames. ], batch size: 63, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:55:04,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff3.min_abs, batch_count=1931860.0, ans=0.2
2023-11-22 11:55:12,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1931860.0, ans=0.125
2023-11-22 11:55:16,763 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.526e+01 8.025e+01 8.580e+01 9.215e+01 1.708e+02, threshold=1.716e+02, percent-clipped=1.0
2023-11-22 11:55:16,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1931926.6666666667, ans=0.125
2023-11-22 11:55:24,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1931926.6666666667, ans=0.125
2023-11-22 11:55:27,483 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.55 vs. limit=15.0
2023-11-22 11:55:28,155 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289800
2023-11-22 11:55:31,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1931993.3333333333, ans=0.2
2023-11-22 11:55:41,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1931993.3333333333, ans=0.0
2023-11-22 11:55:42,875 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.08 vs. limit=15.0
2023-11-22 11:55:49,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1932060.0, ans=0.125
2023-11-22 11:55:55,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1932126.6666666667, ans=0.1
2023-11-22 11:55:56,227 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1250, loss[loss=0.06459, simple_loss=0.08076, pruned_loss=0.01409, audio_tagging_loss=0.01012, over 14409.00 frames. ], tot_loss[loss=0.0717, simple_loss=0.09464, pruned_loss=0.01521, audio_tagging_loss=0.00917, over 3031730.39 frames. ], batch size: 54, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:56:18,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1932193.3333333333, ans=0.125
2023-11-22 11:56:33,925 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289850
2023-11-22 11:56:42,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1932326.6666666667, ans=0.2
2023-11-22 11:56:44,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1932326.6666666667, ans=0.1
2023-11-22 11:56:45,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1932326.6666666667, ans=0.1
2023-11-22 11:57:00,761 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1300, loss[loss=0.07004, simple_loss=0.09554, pruned_loss=0.01515, audio_tagging_loss=0.007115, over 15192.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09404, pruned_loss=0.01506, audio_tagging_loss=0.009148, over 3025237.03 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:57:01,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1932460.0, ans=0.2
2023-11-22 11:57:26,645 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.940e+01 8.056e+01 8.885e+01 9.522e+01 1.296e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 11:57:38,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289900
2023-11-22 11:57:38,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1932660.0, ans=0.125
2023-11-22 11:58:05,439 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1350, loss[loss=0.06965, simple_loss=0.09037, pruned_loss=0.01385, audio_tagging_loss=0.01062, over 16463.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09328, pruned_loss=0.01497, audio_tagging_loss=0.009274, over 3032026.27 frames. ], batch size: 63, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 11:58:22,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1932860.0, ans=0.125
2023-11-22 11:58:22,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1932860.0, ans=0.125
2023-11-22 11:58:38,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1932926.6666666667, ans=0.125
2023-11-22 11:58:42,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 289950
2023-11-22 11:58:46,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.whiten.whitening_limit, batch_count=1932993.3333333333, ans=12.0
2023-11-22 11:58:52,201 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 11:59:01,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1933060.0, ans=0.125
2023-11-22 11:59:05,737 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.71 vs. limit=15.0
2023-11-22 11:59:09,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1933126.6666666667, ans=0.0
2023-11-22 11:59:10,470 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1400, loss[loss=0.06142, simple_loss=0.08424, pruned_loss=0.00838, audio_tagging_loss=0.01092, over 14769.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09347, pruned_loss=0.01493, audio_tagging_loss=0.009205, over 3037729.48 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 11:59:12,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1933126.6666666667, ans=0.0
2023-11-22 11:59:33,791 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.00 vs. limit=15.0
2023-11-22 11:59:36,915 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.595e+01 8.260e+01 8.672e+01 9.990e+01 1.427e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-22 11:59:38,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1933260.0, ans=0.125
2023-11-22 11:59:47,681 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290000
2023-11-22 12:00:15,350 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1450, loss[loss=0.07572, simple_loss=0.09917, pruned_loss=0.01516, audio_tagging_loss=0.01098, over 16335.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09382, pruned_loss=0.01489, audio_tagging_loss=0.009275, over 3038307.85 frames. ], batch size: 62, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:00:23,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1933460.0, ans=0.0
2023-11-22 12:00:36,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.21 vs. limit=22.5
2023-11-22 12:00:53,455 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290050
2023-11-22 12:01:13,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1933726.6666666667, ans=0.0
2023-11-22 12:01:20,159 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1500, loss[loss=0.06989, simple_loss=0.09336, pruned_loss=0.01406, audio_tagging_loss=0.009154, over 15813.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09393, pruned_loss=0.01485, audio_tagging_loss=0.009338, over 3035495.06 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:01:42,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1933860.0, ans=0.1
2023-11-22 12:01:46,718 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.126e+01 8.732e+01 9.513e+01 1.269e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 12:01:48,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1933926.6666666667, ans=0.125
2023-11-22 12:01:54,410 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.72 vs. limit=15.0
2023-11-22 12:01:57,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290100
2023-11-22 12:02:24,732 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1550, loss[loss=0.06813, simple_loss=0.08613, pruned_loss=0.01462, audio_tagging_loss=0.01045, over 15527.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.0936, pruned_loss=0.01488, audio_tagging_loss=0.009432, over 3030751.98 frames. ], batch size: 59, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:02:26,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1934126.6666666667, ans=0.125
2023-11-22 12:02:26,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1934126.6666666667, ans=0.125
2023-11-22 12:02:28,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1934126.6666666667, ans=0.04949747468305833
2023-11-22 12:02:33,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1934126.6666666667, ans=0.2
2023-11-22 12:02:41,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1934193.3333333333, ans=0.0
2023-11-22 12:03:02,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290150
2023-11-22 12:03:09,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1934326.6666666667, ans=0.1
2023-11-22 12:03:09,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1934326.6666666667, ans=0.2
2023-11-22 12:03:20,961 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.17 vs. limit=15.0
2023-11-22 12:03:25,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1934393.3333333333, ans=0.125
2023-11-22 12:03:25,905 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.32 vs. limit=15.0
2023-11-22 12:03:30,918 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1600, loss[loss=0.05201, simple_loss=0.06387, pruned_loss=0.01022, audio_tagging_loss=0.009851, over 15790.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09371, pruned_loss=0.01496, audio_tagging_loss=0.009508, over 3034582.53 frames. ], batch size: 62, lr: 2.75e-03, grad_scale: 32.0
2023-11-22 12:03:58,926 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.903e+01 8.200e+01 8.897e+01 9.742e+01 1.180e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 12:04:08,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290200
2023-11-22 12:04:15,002 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.55 vs. limit=6.0
2023-11-22 12:04:26,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1934726.6666666667, ans=0.0
2023-11-22 12:04:35,568 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1650, loss[loss=0.06358, simple_loss=0.08099, pruned_loss=0.01096, audio_tagging_loss=0.01213, over 14173.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09451, pruned_loss=0.01514, audio_tagging_loss=0.009455, over 3034016.00 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:04:43,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1934793.3333333333, ans=0.125
2023-11-22 12:05:04,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1934926.6666666667, ans=0.125
2023-11-22 12:05:07,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=1934926.6666666667, ans=0.125
2023-11-22 12:05:13,976 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290250
2023-11-22 12:05:15,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1934993.3333333333, ans=0.125
2023-11-22 12:05:24,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1934993.3333333333, ans=10.0
2023-11-22 12:05:28,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=1935060.0, ans=0.125
2023-11-22 12:05:29,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1935060.0, ans=0.0
2023-11-22 12:05:40,399 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1700, loss[loss=0.09064, simple_loss=0.123, pruned_loss=0.02048, audio_tagging_loss=0.008677, over 13615.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09534, pruned_loss=0.01528, audio_tagging_loss=0.009458, over 3041771.86 frames. ], batch size: 53, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:05:45,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1935126.6666666667, ans=0.0
2023-11-22 12:05:47,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=1935126.6666666667, ans=0.0
2023-11-22 12:05:50,090 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.35 vs. limit=6.0
2023-11-22 12:06:08,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1935260.0, ans=0.1
2023-11-22 12:06:10,640 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.333e+01 9.072e+01 9.807e+01 1.367e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-22 12:06:18,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290300
2023-11-22 12:06:18,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1935326.6666666667, ans=0.125
2023-11-22 12:06:30,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1935326.6666666667, ans=0.125
2023-11-22 12:06:43,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1935393.3333333333, ans=0.125
2023-11-22 12:06:45,799 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1750, loss[loss=0.06195, simple_loss=0.07834, pruned_loss=0.01325, audio_tagging_loss=0.009538, over 15516.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09523, pruned_loss=0.01524, audio_tagging_loss=0.009458, over 3040869.94 frames. ], batch size: 61, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:06:47,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1935460.0, ans=0.125
2023-11-22 12:06:58,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1935526.6666666667, ans=0.125
2023-11-22 12:07:22,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1935593.3333333333, ans=0.07
2023-11-22 12:07:23,636 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290350
2023-11-22 12:07:41,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1935726.6666666667, ans=0.1
2023-11-22 12:07:50,640 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1800, loss[loss=0.06969, simple_loss=0.09205, pruned_loss=0.01415, audio_tagging_loss=0.009519, over 14510.00 frames. ], tot_loss[loss=0.07197, simple_loss=0.09477, pruned_loss=0.0152, audio_tagging_loss=0.00939, over 3037222.78 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:08:01,874 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.01 vs. limit=15.0
2023-11-22 12:08:19,857 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.690e+01 8.156e+01 8.799e+01 9.735e+01 1.149e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 12:08:28,532 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290400
2023-11-22 12:08:55,126 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1850, loss[loss=0.06719, simple_loss=0.07982, pruned_loss=0.01368, audio_tagging_loss=0.0136, over 15967.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09417, pruned_loss=0.01521, audio_tagging_loss=0.009368, over 3034311.97 frames. ], batch size: 61, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:08:56,915 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.05 vs. limit=10.0
2023-11-22 12:09:02,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1936126.6666666667, ans=0.125
2023-11-22 12:09:09,952 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.90 vs. limit=6.0
2023-11-22 12:09:13,057 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.88 vs. limit=12.0
2023-11-22 12:09:21,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1936260.0, ans=0.125
2023-11-22 12:09:33,441 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290450
2023-11-22 12:09:41,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=1936326.6666666667, ans=0.2
2023-11-22 12:09:59,889 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1900, loss[loss=0.06249, simple_loss=0.08498, pruned_loss=0.009062, audio_tagging_loss=0.01094, over 14911.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09398, pruned_loss=0.01513, audio_tagging_loss=0.009335, over 3043476.34 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:10:14,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1936526.6666666667, ans=0.0
2023-11-22 12:10:16,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1936526.6666666667, ans=0.1
2023-11-22 12:10:19,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.00 vs. limit=22.5
2023-11-22 12:10:20,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1936526.6666666667, ans=0.125
2023-11-22 12:10:29,145 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.157e+01 8.748e+01 9.500e+01 1.354e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 12:10:37,253 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290500
2023-11-22 12:11:04,302 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 1950, loss[loss=0.07826, simple_loss=0.0987, pruned_loss=0.02031, audio_tagging_loss=0.008601, over 14388.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09395, pruned_loss=0.01507, audio_tagging_loss=0.009319, over 3045325.18 frames. ], batch size: 52, lr: 2.75e-03, grad_scale: 8.0
2023-11-22 12:11:16,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1936860.0, ans=0.0
2023-11-22 12:11:29,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1936926.6666666667, ans=0.125
2023-11-22 12:11:41,389 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290550
2023-11-22 12:12:00,452 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.74 vs. limit=15.0
2023-11-22 12:12:02,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=1937060.0, ans=0.5
2023-11-22 12:12:08,550 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2000, loss[loss=0.08452, simple_loss=0.1148, pruned_loss=0.01667, audio_tagging_loss=0.01043, over 15699.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09344, pruned_loss=0.01502, audio_tagging_loss=0.009325, over 3041382.36 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:12:19,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1937126.6666666667, ans=0.125
2023-11-22 12:12:35,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1937260.0, ans=0.1
2023-11-22 12:12:38,528 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.037e+01 8.579e+01 9.384e+01 1.253e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-22 12:12:40,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1937260.0, ans=0.125
2023-11-22 12:12:45,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290600
2023-11-22 12:12:50,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1937326.6666666667, ans=0.125
2023-11-22 12:13:13,308 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2050, loss[loss=0.06994, simple_loss=0.09451, pruned_loss=0.0154, audio_tagging_loss=0.007278, over 15156.00 frames. ], tot_loss[loss=0.07183, simple_loss=0.09445, pruned_loss=0.01536, audio_tagging_loss=0.00925, over 3038336.99 frames. ], batch size: 56, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:13:17,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1937460.0, ans=0.1
2023-11-22 12:13:21,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1937460.0, ans=0.2
2023-11-22 12:13:47,090 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:13:50,728 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290650
2023-11-22 12:14:04,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1937726.6666666667, ans=0.0
2023-11-22 12:14:18,216 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2100, loss[loss=0.08764, simple_loss=0.119, pruned_loss=0.01847, audio_tagging_loss=0.009644, over 15087.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09468, pruned_loss=0.01527, audio_tagging_loss=0.009249, over 3038906.13 frames. ], batch size: 57, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:14:32,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1937860.0, ans=0.1
2023-11-22 12:14:38,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1937860.0, ans=0.0
2023-11-22 12:14:47,391 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.119e+01 8.164e+01 8.808e+01 9.351e+01 1.143e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 12:14:50,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1937926.6666666667, ans=0.1
2023-11-22 12:14:50,622 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.50 vs. limit=15.0
2023-11-22 12:14:54,708 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290700
2023-11-22 12:15:11,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=1938060.0, ans=0.1
2023-11-22 12:15:13,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1938060.0, ans=0.2
2023-11-22 12:15:20,709 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.12 vs. limit=15.0
2023-11-22 12:15:22,359 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2150, loss[loss=0.09873, simple_loss=0.1366, pruned_loss=0.02505, audio_tagging_loss=0.005385, over 15861.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09497, pruned_loss=0.01527, audio_tagging_loss=0.009208, over 3050550.59 frames. ], batch size: 58, lr: 2.75e-03, grad_scale: 16.0
2023-11-22 12:15:32,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1938126.6666666667, ans=0.07
2023-11-22 12:15:35,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1938193.3333333333, ans=10.0
2023-11-22 12:15:54,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1938260.0, ans=0.125
2023-11-22 12:15:59,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290750
2023-11-22 12:16:01,980 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:16:03,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1938326.6666666667, ans=0.0
2023-11-22 12:16:05,882 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-22 12:16:14,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1938393.3333333333, ans=0.0
2023-11-22 12:16:26,557 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2200, loss[loss=0.07871, simple_loss=0.108, pruned_loss=0.0162, audio_tagging_loss=0.008521, over 15978.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09404, pruned_loss=0.01516, audio_tagging_loss=0.009207, over 3044445.93 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:16:32,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1938460.0, ans=0.1
2023-11-22 12:16:41,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=15.16 vs. limit=15.0
2023-11-22 12:16:56,426 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.156e+01 8.385e+01 8.783e+01 9.663e+01 1.140e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 12:16:57,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1938593.3333333333, ans=0.125
2023-11-22 12:16:59,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1938593.3333333333, ans=0.125
2023-11-22 12:17:03,928 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290800
2023-11-22 12:17:30,946 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2250, loss[loss=0.05841, simple_loss=0.07841, pruned_loss=0.01219, audio_tagging_loss=0.007021, over 17015.00 frames. ], tot_loss[loss=0.07231, simple_loss=0.09515, pruned_loss=0.0155, audio_tagging_loss=0.009238, over 3049053.85 frames. ], batch size: 63, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:17:41,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1938793.3333333333, ans=0.125
2023-11-22 12:17:50,959 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.73 vs. limit=12.0
2023-11-22 12:18:08,474 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290850
2023-11-22 12:18:08,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1938993.3333333333, ans=0.125
2023-11-22 12:18:29,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1939060.0, ans=0.0
2023-11-22 12:18:30,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1939060.0, ans=0.2
2023-11-22 12:18:35,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1939126.6666666667, ans=0.0
2023-11-22 12:18:36,069 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2300, loss[loss=0.06027, simple_loss=0.07706, pruned_loss=0.01125, audio_tagging_loss=0.0105, over 15411.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09527, pruned_loss=0.0154, audio_tagging_loss=0.009298, over 3043813.44 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:18:53,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1939193.3333333333, ans=0.0
2023-11-22 12:18:55,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=1939193.3333333333, ans=0.0
2023-11-22 12:18:57,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1939193.3333333333, ans=0.1
2023-11-22 12:19:06,742 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.734e+01 8.285e+01 8.843e+01 9.474e+01 1.631e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 12:19:12,983 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290900
2023-11-22 12:19:23,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1939326.6666666667, ans=0.1
2023-11-22 12:19:31,978 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:19:39,821 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2350, loss[loss=0.05891, simple_loss=0.08172, pruned_loss=0.009069, audio_tagging_loss=0.008983, over 15564.00 frames. ], tot_loss[loss=0.07218, simple_loss=0.09505, pruned_loss=0.01532, audio_tagging_loss=0.009331, over 3043680.30 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:20:14,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1939593.3333333333, ans=0.125
2023-11-22 12:20:17,925 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 290950
2023-11-22 12:20:39,962 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0
2023-11-22 12:20:44,660 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2400, loss[loss=0.08327, simple_loss=0.1186, pruned_loss=0.01499, audio_tagging_loss=0.008966, over 14853.00 frames. ], tot_loss[loss=0.0725, simple_loss=0.09532, pruned_loss=0.01544, audio_tagging_loss=0.009405, over 3045859.31 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:21:09,852 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.87 vs. limit=22.5
2023-11-22 12:21:15,531 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.176e+01 8.394e+01 8.930e+01 9.736e+01 4.109e+02, threshold=1.786e+02, percent-clipped=1.0
2023-11-22 12:21:18,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1939926.6666666667, ans=0.125
2023-11-22 12:21:21,755 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291000
2023-11-22 12:21:41,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.72 vs. limit=15.0
2023-11-22 12:21:50,309 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2450, loss[loss=0.0696, simple_loss=0.09439, pruned_loss=0.01284, audio_tagging_loss=0.009561, over 15348.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09516, pruned_loss=0.01532, audio_tagging_loss=0.009453, over 3045508.87 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:22:00,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1940126.6666666667, ans=0.125
2023-11-22 12:22:14,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1940193.3333333333, ans=0.125
2023-11-22 12:22:18,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=1940260.0, ans=10.0
2023-11-22 12:22:23,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1940260.0, ans=0.0
2023-11-22 12:22:28,653 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291050
2023-11-22 12:22:49,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1940393.3333333333, ans=0.0
2023-11-22 12:22:56,586 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2500, loss[loss=0.07074, simple_loss=0.09662, pruned_loss=0.01569, audio_tagging_loss=0.006738, over 15698.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09499, pruned_loss=0.01528, audio_tagging_loss=0.009444, over 3055235.72 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:23:16,812 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.40 vs. limit=12.0
2023-11-22 12:23:17,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1940526.6666666667, ans=0.0
2023-11-22 12:23:26,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1940593.3333333333, ans=0.125
2023-11-22 12:23:26,948 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.685e+01 8.038e+01 8.739e+01 9.351e+01 1.232e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 12:23:33,948 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291100
2023-11-22 12:23:35,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1940660.0, ans=0.125
2023-11-22 12:23:39,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1940660.0, ans=0.0
2023-11-22 12:24:01,983 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2550, loss[loss=0.07753, simple_loss=0.1018, pruned_loss=0.01949, audio_tagging_loss=0.007131, over 15241.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09497, pruned_loss=0.01535, audio_tagging_loss=0.009342, over 3054459.31 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:24:10,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1940793.3333333333, ans=0.125
2023-11-22 12:24:40,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291150
2023-11-22 12:24:52,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1940993.3333333333, ans=0.0
2023-11-22 12:25:06,909 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2600, loss[loss=0.05903, simple_loss=0.07642, pruned_loss=0.01282, audio_tagging_loss=0.008006, over 15890.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09495, pruned_loss=0.01531, audio_tagging_loss=0.00914, over 3052550.65 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:25:13,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1941126.6666666667, ans=0.125
2023-11-22 12:25:25,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1941193.3333333333, ans=10.0
2023-11-22 12:25:33,586 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.39 vs. limit=15.0
2023-11-22 12:25:37,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1941260.0, ans=0.125
2023-11-22 12:25:39,209 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.159e+01 9.048e+01 9.727e+01 1.323e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 12:25:46,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291200
2023-11-22 12:25:48,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1941326.6666666667, ans=0.0
2023-11-22 12:26:11,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1941393.3333333333, ans=0.125
2023-11-22 12:26:14,645 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2650, loss[loss=0.06717, simple_loss=0.09066, pruned_loss=0.01161, audio_tagging_loss=0.01023, over 14419.00 frames. ], tot_loss[loss=0.0723, simple_loss=0.09568, pruned_loss=0.01541, audio_tagging_loss=0.009048, over 3054986.85 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:26:14,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1941460.0, ans=0.0
2023-11-22 12:26:25,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1941460.0, ans=0.1
2023-11-22 12:26:27,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1941526.6666666667, ans=0.0
2023-11-22 12:26:27,597 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.28 vs. limit=15.0
2023-11-22 12:26:53,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291250
2023-11-22 12:27:20,086 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2700, loss[loss=0.07286, simple_loss=0.09625, pruned_loss=0.01659, audio_tagging_loss=0.008148, over 15366.00 frames. ], tot_loss[loss=0.07199, simple_loss=0.0954, pruned_loss=0.01525, audio_tagging_loss=0.009035, over 3061938.43 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:27:50,983 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 7.955e+01 8.601e+01 9.459e+01 1.243e+02, threshold=1.720e+02, percent-clipped=0.0
2023-11-22 12:27:58,476 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291300
2023-11-22 12:28:25,090 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2750, loss[loss=0.03995, simple_loss=0.05151, pruned_loss=0.005204, audio_tagging_loss=0.008993, over 14111.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09436, pruned_loss=0.01503, audio_tagging_loss=0.009069, over 3050766.58 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:28:38,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1942193.3333333333, ans=0.07
2023-11-22 12:28:53,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1942260.0, ans=0.125
2023-11-22 12:28:59,786 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.78 vs. limit=10.0
2023-11-22 12:29:03,681 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291350
2023-11-22 12:29:09,135 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.09 vs. limit=10.0
2023-11-22 12:29:17,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1942393.3333333333, ans=0.1
2023-11-22 12:29:21,470 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:29:30,134 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2800, loss[loss=0.06525, simple_loss=0.08934, pruned_loss=0.009168, audio_tagging_loss=0.01142, over 15910.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09506, pruned_loss=0.01528, audio_tagging_loss=0.009082, over 3053132.25 frames. ], batch size: 61, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:29:39,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=1942460.0, ans=22.5
2023-11-22 12:30:01,637 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.537e+01 8.249e+01 8.938e+01 9.573e+01 1.382e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 12:30:07,959 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291400
2023-11-22 12:30:18,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1942660.0, ans=0.125
2023-11-22 12:30:36,177 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2850, loss[loss=0.08733, simple_loss=0.1205, pruned_loss=0.02132, audio_tagging_loss=0.005734, over 17378.00 frames. ], tot_loss[loss=0.07165, simple_loss=0.09499, pruned_loss=0.01517, audio_tagging_loss=0.008984, over 3057844.60 frames. ], batch size: 62, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:30:48,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1942860.0, ans=0.2
2023-11-22 12:30:50,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1942860.0, ans=0.125
2023-11-22 12:30:54,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1942860.0, ans=0.125
2023-11-22 12:30:56,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1942860.0, ans=0.0
2023-11-22 12:31:08,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=1942926.6666666667, ans=0.125
2023-11-22 12:31:13,565 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291450
2023-11-22 12:31:13,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1942993.3333333333, ans=0.125
2023-11-22 12:31:22,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1942993.3333333333, ans=0.0
2023-11-22 12:31:40,812 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2900, loss[loss=0.07235, simple_loss=0.09396, pruned_loss=0.01378, audio_tagging_loss=0.01159, over 16245.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09498, pruned_loss=0.01534, audio_tagging_loss=0.009196, over 3059948.89 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:31:47,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1943126.6666666667, ans=0.95
2023-11-22 12:32:06,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1943260.0, ans=0.125
2023-11-22 12:32:11,962 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.290e+01 8.892e+01 9.564e+01 1.220e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 12:32:19,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291500
2023-11-22 12:32:33,323 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.26 vs. limit=6.0
2023-11-22 12:32:45,619 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 2950, loss[loss=0.08155, simple_loss=0.1012, pruned_loss=0.02274, audio_tagging_loss=0.008193, over 14869.00 frames. ], tot_loss[loss=0.07233, simple_loss=0.09531, pruned_loss=0.01547, audio_tagging_loss=0.009205, over 3057036.65 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:33:04,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1943526.6666666667, ans=0.0
2023-11-22 12:33:06,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1943526.6666666667, ans=0.0
2023-11-22 12:33:12,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1943593.3333333333, ans=0.0
2023-11-22 12:33:12,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1943593.3333333333, ans=0.1
2023-11-22 12:33:13,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1943593.3333333333, ans=0.125
2023-11-22 12:33:23,476 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291550
2023-11-22 12:33:31,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1943660.0, ans=0.2
2023-11-22 12:33:39,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1943726.6666666667, ans=0.125
2023-11-22 12:33:50,767 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3000, loss[loss=0.07937, simple_loss=0.1042, pruned_loss=0.0191, audio_tagging_loss=0.008178, over 15633.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.09569, pruned_loss=0.01547, audio_tagging_loss=0.00925, over 3057302.40 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:33:50,770 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 12:34:12,164 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.6537, 5.3634, 4.9252, 5.3579], device='cuda:0')
2023-11-22 12:34:30,480 INFO [train_asr.py:1253] (0/4) Epoch 25, validation: loss=0.05876, simple_loss=0.05157, pruned_loss=0.005103, audio_tagging_loss=0.02788, over 4681554.00 frames. 
2023-11-22 12:34:30,481 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 12:34:37,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=1943793.3333333333, ans=0.025
2023-11-22 12:35:01,167 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.137e+01 8.344e+01 8.945e+01 9.581e+01 1.218e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-22 12:35:08,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291600
2023-11-22 12:35:32,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=1944060.0, ans=0.125
2023-11-22 12:35:35,438 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3050, loss[loss=0.07692, simple_loss=0.1019, pruned_loss=0.01613, audio_tagging_loss=0.009818, over 14955.00 frames. ], tot_loss[loss=0.07276, simple_loss=0.09604, pruned_loss=0.0155, audio_tagging_loss=0.009247, over 3055882.98 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:35:37,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=1944126.6666666667, ans=0.025
2023-11-22 12:36:10,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1944260.0, ans=0.0
2023-11-22 12:36:13,332 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:36:13,399 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291650
2023-11-22 12:36:15,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1944326.6666666667, ans=0.0
2023-11-22 12:36:34,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1944393.3333333333, ans=0.2
2023-11-22 12:36:39,982 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3100, loss[loss=0.06145, simple_loss=0.08216, pruned_loss=0.01374, audio_tagging_loss=0.006627, over 15119.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09522, pruned_loss=0.01522, audio_tagging_loss=0.009334, over 3051740.38 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:36:46,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1944460.0, ans=0.2
2023-11-22 12:36:55,313 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.32 vs. limit=10.0
2023-11-22 12:37:10,654 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.355e+01 8.957e+01 9.791e+01 1.144e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 12:37:17,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291700
2023-11-22 12:37:31,709 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.92 vs. limit=15.0
2023-11-22 12:37:44,668 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3150, loss[loss=0.07213, simple_loss=0.09655, pruned_loss=0.01368, audio_tagging_loss=0.01018, over 15338.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09422, pruned_loss=0.01512, audio_tagging_loss=0.009533, over 3050359.45 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:37:59,569 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.12 vs. limit=6.0
2023-11-22 12:38:21,546 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291750
2023-11-22 12:38:44,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1945060.0, ans=0.0
2023-11-22 12:38:45,167 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.41 vs. limit=6.0
2023-11-22 12:38:49,298 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3200, loss[loss=0.0801, simple_loss=0.1026, pruned_loss=0.01961, audio_tagging_loss=0.00917, over 14861.00 frames. ], tot_loss[loss=0.07229, simple_loss=0.0952, pruned_loss=0.01526, audio_tagging_loss=0.009433, over 3047517.67 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:39:20,212 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.187e+01 8.718e+01 9.345e+01 1.176e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-22 12:39:27,175 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291800
2023-11-22 12:39:33,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=1945326.6666666667, ans=0.05
2023-11-22 12:39:44,220 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.87 vs. limit=15.0
2023-11-22 12:39:53,940 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3250, loss[loss=0.06565, simple_loss=0.08332, pruned_loss=0.01278, audio_tagging_loss=0.01121, over 14676.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09488, pruned_loss=0.01521, audio_tagging_loss=0.009491, over 3040983.67 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:39:59,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1945460.0, ans=0.125
2023-11-22 12:40:08,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1945526.6666666667, ans=0.125
2023-11-22 12:40:30,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=7.89 vs. limit=12.0
2023-11-22 12:40:31,190 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291850
2023-11-22 12:40:58,265 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3300, loss[loss=0.06326, simple_loss=0.08246, pruned_loss=0.01282, audio_tagging_loss=0.009208, over 16710.00 frames. ], tot_loss[loss=0.07191, simple_loss=0.09417, pruned_loss=0.01529, audio_tagging_loss=0.009537, over 3041047.68 frames. ], batch size: 62, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:41:29,373 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.525e+01 8.169e+01 8.738e+01 9.487e+01 1.174e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 12:41:35,680 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291900
2023-11-22 12:42:03,242 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3350, loss[loss=0.08235, simple_loss=0.1009, pruned_loss=0.02083, audio_tagging_loss=0.01105, over 15342.00 frames. ], tot_loss[loss=0.07243, simple_loss=0.09498, pruned_loss=0.01551, audio_tagging_loss=0.009437, over 3039385.98 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:42:14,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1946126.6666666667, ans=0.2
2023-11-22 12:42:28,166 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.09 vs. limit=10.0
2023-11-22 12:42:33,691 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.41 vs. limit=22.5
2023-11-22 12:42:40,682 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 291950
2023-11-22 12:42:40,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1946326.6666666667, ans=0.0
2023-11-22 12:42:49,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.11 vs. limit=12.0
2023-11-22 12:42:57,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1946393.3333333333, ans=0.1
2023-11-22 12:43:08,069 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3400, loss[loss=0.05326, simple_loss=0.06682, pruned_loss=0.009806, audio_tagging_loss=0.01005, over 15019.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09552, pruned_loss=0.01548, audio_tagging_loss=0.009303, over 3045331.63 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 32.0
2023-11-22 12:43:32,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1946593.3333333333, ans=0.125
2023-11-22 12:43:38,762 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.227e+01 8.817e+01 9.429e+01 1.318e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 12:43:39,484 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.37 vs. limit=15.0
2023-11-22 12:43:45,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292000
2023-11-22 12:43:46,566 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-292000.pt
2023-11-22 12:43:56,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1946660.0, ans=0.125
2023-11-22 12:44:07,923 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.92 vs. limit=15.0
2023-11-22 12:44:15,299 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3450, loss[loss=0.09565, simple_loss=0.1408, pruned_loss=0.02028, audio_tagging_loss=0.004967, over 17115.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09527, pruned_loss=0.01533, audio_tagging_loss=0.009253, over 3040417.11 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:44:16,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1946793.3333333333, ans=0.1
2023-11-22 12:44:38,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1946860.0, ans=0.125
2023-11-22 12:44:46,169 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:44:51,958 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292050
2023-11-22 12:45:12,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1947060.0, ans=0.1
2023-11-22 12:45:12,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1947060.0, ans=0.125
2023-11-22 12:45:19,192 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3500, loss[loss=0.0645, simple_loss=0.08177, pruned_loss=0.01225, audio_tagging_loss=0.01137, over 15116.00 frames. ], tot_loss[loss=0.07227, simple_loss=0.09517, pruned_loss=0.0154, audio_tagging_loss=0.009291, over 3041839.89 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:45:21,206 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.93 vs. limit=6.0
2023-11-22 12:45:37,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=1947193.3333333333, ans=0.05
2023-11-22 12:45:51,049 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:45:52,128 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 7.990e+01 8.840e+01 9.495e+01 1.155e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 12:45:55,971 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292100
2023-11-22 12:46:01,206 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:46:03,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=1947326.6666666667, ans=0.125
2023-11-22 12:46:08,410 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:46:09,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1947393.3333333333, ans=0.125
2023-11-22 12:46:14,865 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.08 vs. limit=15.0
2023-11-22 12:46:23,361 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3550, loss[loss=0.07278, simple_loss=0.09363, pruned_loss=0.01597, audio_tagging_loss=0.01, over 15756.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09368, pruned_loss=0.01504, audio_tagging_loss=0.009324, over 3046832.93 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:46:28,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1947460.0, ans=0.125
2023-11-22 12:46:34,864 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.06 vs. limit=15.0
2023-11-22 12:46:35,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1947526.6666666667, ans=0.125
2023-11-22 12:46:37,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1947526.6666666667, ans=0.0
2023-11-22 12:46:48,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1947593.3333333333, ans=0.125
2023-11-22 12:47:00,217 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292150
2023-11-22 12:47:10,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1947660.0, ans=0.125
2023-11-22 12:47:16,281 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.36 vs. limit=15.0
2023-11-22 12:47:27,286 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3600, loss[loss=0.05687, simple_loss=0.07504, pruned_loss=0.01143, audio_tagging_loss=0.007917, over 15274.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09332, pruned_loss=0.01496, audio_tagging_loss=0.009228, over 3038010.94 frames. ], batch size: 57, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:47:31,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1947793.3333333333, ans=0.1
2023-11-22 12:47:34,141 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.22 vs. limit=15.0
2023-11-22 12:47:49,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1947860.0, ans=0.125
2023-11-22 12:48:00,887 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.791e+01 8.103e+01 8.938e+01 1.008e+02 1.423e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 12:48:04,709 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292200
2023-11-22 12:48:17,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.81 vs. limit=12.0
2023-11-22 12:48:23,489 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.31 vs. limit=15.0
2023-11-22 12:48:31,938 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3650, loss[loss=0.06764, simple_loss=0.08827, pruned_loss=0.01413, audio_tagging_loss=0.009376, over 15100.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09342, pruned_loss=0.01494, audio_tagging_loss=0.009249, over 3042053.35 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:49:09,592 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292250
2023-11-22 12:49:29,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.89 vs. limit=10.0
2023-11-22 12:49:37,040 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3700, loss[loss=0.08369, simple_loss=0.1167, pruned_loss=0.01764, audio_tagging_loss=0.007715, over 15046.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09385, pruned_loss=0.01502, audio_tagging_loss=0.009208, over 3050123.40 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:50:10,188 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.131e+01 8.890e+01 9.560e+01 1.123e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 12:50:13,999 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292300
2023-11-22 12:50:41,548 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3750, loss[loss=0.08847, simple_loss=0.111, pruned_loss=0.02337, audio_tagging_loss=0.009573, over 15359.00 frames. ], tot_loss[loss=0.07171, simple_loss=0.09478, pruned_loss=0.0152, audio_tagging_loss=0.009124, over 3055559.26 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:50:47,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=1948793.3333333333, ans=0.125
2023-11-22 12:50:53,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1948860.0, ans=0.07
2023-11-22 12:51:19,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292350
2023-11-22 12:51:25,168 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:51:37,461 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.41 vs. limit=15.0
2023-11-22 12:51:45,489 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3800, loss[loss=0.07318, simple_loss=0.09405, pruned_loss=0.0154, audio_tagging_loss=0.01076, over 15882.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09529, pruned_loss=0.01527, audio_tagging_loss=0.009206, over 3053445.61 frames. ], batch size: 59, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:51:59,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1949193.3333333333, ans=0.1
2023-11-22 12:52:13,876 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.30 vs. limit=12.0
2023-11-22 12:52:16,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=1949260.0, ans=0.125
2023-11-22 12:52:19,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.58 vs. limit=15.0
2023-11-22 12:52:20,460 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 12:52:21,350 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.073e+01 8.669e+01 9.455e+01 1.608e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-22 12:52:23,923 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292400
2023-11-22 12:52:43,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=1949393.3333333333, ans=0.0
2023-11-22 12:52:50,537 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3850, loss[loss=0.04322, simple_loss=0.04862, pruned_loss=0.005047, audio_tagging_loss=0.01386, over 13935.00 frames. ], tot_loss[loss=0.07173, simple_loss=0.09472, pruned_loss=0.01508, audio_tagging_loss=0.009292, over 3048610.49 frames. ], batch size: 54, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:53:19,360 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0
2023-11-22 12:53:28,235 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292450
2023-11-22 12:53:28,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1949660.0, ans=0.125
2023-11-22 12:53:38,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=1949660.0, ans=0.125
2023-11-22 12:53:50,089 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.22 vs. limit=15.0
2023-11-22 12:53:55,292 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3900, loss[loss=0.06989, simple_loss=0.09201, pruned_loss=0.01532, audio_tagging_loss=0.008563, over 14433.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09454, pruned_loss=0.01508, audio_tagging_loss=0.009329, over 3042840.75 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:54:30,352 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.059e+01 8.183e+01 8.833e+01 9.463e+01 1.290e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 12:54:33,674 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292500
2023-11-22 12:54:33,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1949993.3333333333, ans=0.125
2023-11-22 12:54:41,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.95 vs. limit=15.0
2023-11-22 12:55:00,189 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 3950, loss[loss=0.08762, simple_loss=0.1113, pruned_loss=0.02269, audio_tagging_loss=0.009271, over 14203.00 frames. ], tot_loss[loss=0.07182, simple_loss=0.09457, pruned_loss=0.01507, audio_tagging_loss=0.009465, over 3045322.48 frames. ], batch size: 53, lr: 2.74e-03, grad_scale: 8.0
2023-11-22 12:55:12,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1950193.3333333333, ans=0.07
2023-11-22 12:55:21,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=1950193.3333333333, ans=0.09899494936611666
2023-11-22 12:55:31,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1950260.0, ans=0.125
2023-11-22 12:55:38,545 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292550
2023-11-22 12:55:38,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1950326.6666666667, ans=0.035
2023-11-22 12:55:40,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1950326.6666666667, ans=0.1
2023-11-22 12:56:04,982 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4000, loss[loss=0.07985, simple_loss=0.1035, pruned_loss=0.01669, audio_tagging_loss=0.01142, over 15023.00 frames. ], tot_loss[loss=0.0719, simple_loss=0.09436, pruned_loss=0.01513, audio_tagging_loss=0.009592, over 3047062.37 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:56:11,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1950460.0, ans=0.125
2023-11-22 12:56:29,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1950526.6666666667, ans=0.0
2023-11-22 12:56:30,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1950593.3333333333, ans=0.125
2023-11-22 12:56:33,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1950593.3333333333, ans=0.125
2023-11-22 12:56:40,612 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.359e+01 8.523e+01 9.261e+01 9.950e+01 1.651e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-22 12:56:43,808 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292600
2023-11-22 12:56:49,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=1950660.0, ans=0.0
2023-11-22 12:57:04,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1950726.6666666667, ans=0.125
2023-11-22 12:57:09,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1950726.6666666667, ans=0.0
2023-11-22 12:57:11,647 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4050, loss[loss=0.07893, simple_loss=0.1154, pruned_loss=0.01156, audio_tagging_loss=0.009691, over 15279.00 frames. ], tot_loss[loss=0.07294, simple_loss=0.09595, pruned_loss=0.01541, audio_tagging_loss=0.00955, over 3045616.07 frames. ], batch size: 55, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:57:14,244 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 12:57:26,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.36 vs. limit=22.5
2023-11-22 12:57:49,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292650
2023-11-22 12:57:49,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1950993.3333333333, ans=0.2
2023-11-22 12:57:52,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1950993.3333333333, ans=0.1
2023-11-22 12:58:05,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1951060.0, ans=0.125
2023-11-22 12:58:11,699 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.51 vs. limit=6.0
2023-11-22 12:58:16,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1951126.6666666667, ans=0.1
2023-11-22 12:58:17,017 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4100, loss[loss=0.05665, simple_loss=0.0698, pruned_loss=0.01054, audio_tagging_loss=0.01121, over 15306.00 frames. ], tot_loss[loss=0.07256, simple_loss=0.0954, pruned_loss=0.01534, audio_tagging_loss=0.009518, over 3044134.92 frames. ], batch size: 60, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:58:52,136 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.594e+01 8.488e+01 8.924e+01 9.565e+01 1.327e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 12:58:54,718 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292700
2023-11-22 12:59:19,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=1951393.3333333333, ans=0.0
2023-11-22 12:59:22,207 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4150, loss[loss=0.05854, simple_loss=0.0752, pruned_loss=0.01098, audio_tagging_loss=0.009953, over 15612.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09516, pruned_loss=0.01524, audio_tagging_loss=0.0093, over 3046164.35 frames. ], batch size: 61, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 12:59:25,433 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.11 vs. limit=10.0
2023-11-22 12:59:37,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1951526.6666666667, ans=0.125
2023-11-22 12:59:44,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.16 vs. limit=15.0
2023-11-22 12:59:49,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1951593.3333333333, ans=10.0
2023-11-22 12:59:59,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1951593.3333333333, ans=0.0
2023-11-22 13:00:00,033 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292750
2023-11-22 13:00:09,154 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 13:00:09,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1951660.0, ans=0.0
2023-11-22 13:00:16,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1951726.6666666667, ans=0.0
2023-11-22 13:00:26,778 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4200, loss[loss=0.07772, simple_loss=0.1077, pruned_loss=0.0159, audio_tagging_loss=0.007978, over 15933.00 frames. ], tot_loss[loss=0.07185, simple_loss=0.09495, pruned_loss=0.01518, audio_tagging_loss=0.009195, over 3049908.25 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 13:00:38,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1951793.3333333333, ans=0.125
2023-11-22 13:01:01,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1951926.6666666667, ans=0.125
2023-11-22 13:01:02,469 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.208e+01 8.744e+01 9.952e+01 1.323e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 13:01:05,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292800
2023-11-22 13:01:33,213 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4250, loss[loss=0.07141, simple_loss=0.1029, pruned_loss=0.01368, audio_tagging_loss=0.00629, over 14897.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09549, pruned_loss=0.01534, audio_tagging_loss=0.009132, over 3045792.27 frames. ], batch size: 56, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 13:01:51,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1952193.3333333333, ans=0.125
2023-11-22 13:02:07,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1952260.0, ans=0.0
2023-11-22 13:02:09,947 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292850
2023-11-22 13:02:10,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1952326.6666666667, ans=0.0
2023-11-22 13:02:15,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=1952326.6666666667, ans=0.015
2023-11-22 13:02:29,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=1952393.3333333333, ans=0.125
2023-11-22 13:02:37,483 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4300, loss[loss=0.06464, simple_loss=0.08298, pruned_loss=0.01323, audio_tagging_loss=0.009916, over 14927.00 frames. ], tot_loss[loss=0.07232, simple_loss=0.09573, pruned_loss=0.01535, audio_tagging_loss=0.009104, over 3043978.68 frames. ], batch size: 58, lr: 2.74e-03, grad_scale: 16.0
2023-11-22 13:02:42,716 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:02:48,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=1952460.0, ans=0.125
2023-11-22 13:02:51,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1952526.6666666667, ans=0.125
2023-11-22 13:03:11,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1952593.3333333333, ans=0.125
2023-11-22 13:03:12,146 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.605e+01 8.363e+01 9.136e+01 9.974e+01 1.277e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-22 13:03:14,739 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292900
2023-11-22 13:03:40,081 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.38 vs. limit=10.0
2023-11-22 13:03:40,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1952793.3333333333, ans=0.1
2023-11-22 13:03:41,717 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4350, loss[loss=0.08403, simple_loss=0.1063, pruned_loss=0.02188, audio_tagging_loss=0.008993, over 14043.00 frames. ], tot_loss[loss=0.07242, simple_loss=0.09574, pruned_loss=0.01537, audio_tagging_loss=0.009176, over 3043997.86 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:03:47,552 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.56 vs. limit=10.0
2023-11-22 13:03:50,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1952793.3333333333, ans=0.1
2023-11-22 13:03:58,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1952860.0, ans=0.125
2023-11-22 13:04:12,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1952926.6666666667, ans=0.0
2023-11-22 13:04:13,781 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:04:19,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 292950
2023-11-22 13:04:19,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1952993.3333333333, ans=0.125
2023-11-22 13:04:23,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=1952993.3333333333, ans=0.025
2023-11-22 13:04:26,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1952993.3333333333, ans=0.0
2023-11-22 13:04:32,956 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:04:43,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1953060.0, ans=0.2
2023-11-22 13:04:46,744 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4400, loss[loss=0.04958, simple_loss=0.06917, pruned_loss=0.008885, audio_tagging_loss=0.00611, over 16349.00 frames. ], tot_loss[loss=0.07214, simple_loss=0.09539, pruned_loss=0.01535, audio_tagging_loss=0.00909, over 3045481.74 frames. ], batch size: 64, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:04:51,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1953126.6666666667, ans=0.125
2023-11-22 13:05:03,831 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.61 vs. limit=5.0
2023-11-22 13:05:13,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=1953260.0, ans=0.95
2023-11-22 13:05:21,492 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 8.093e+01 8.881e+01 9.679e+01 1.352e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-22 13:05:24,145 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293000
2023-11-22 13:05:33,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1953326.6666666667, ans=0.125
2023-11-22 13:05:44,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1953393.3333333333, ans=0.125
2023-11-22 13:05:51,676 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4450, loss[loss=0.04794, simple_loss=0.05726, pruned_loss=0.007973, audio_tagging_loss=0.01133, over 14346.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09539, pruned_loss=0.01535, audio_tagging_loss=0.009049, over 3051750.74 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:05:56,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1953460.0, ans=0.2
2023-11-22 13:05:58,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1953460.0, ans=0.125
2023-11-22 13:05:59,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1953460.0, ans=0.125
2023-11-22 13:06:01,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1953460.0, ans=0.0
2023-11-22 13:06:21,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1953593.3333333333, ans=0.0
2023-11-22 13:06:28,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293050
2023-11-22 13:06:28,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1953660.0, ans=0.125
2023-11-22 13:06:32,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.92 vs. limit=22.5
2023-11-22 13:06:55,434 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4500, loss[loss=0.07761, simple_loss=0.1026, pruned_loss=0.01799, audio_tagging_loss=0.008316, over 16240.00 frames. ], tot_loss[loss=0.07158, simple_loss=0.0943, pruned_loss=0.01525, audio_tagging_loss=0.009175, over 3049551.53 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:07:14,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1953860.0, ans=0.125
2023-11-22 13:07:21,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1953926.6666666667, ans=0.125
2023-11-22 13:07:31,537 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.144e+01 8.843e+01 9.621e+01 1.771e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 13:07:32,440 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.03 vs. limit=15.0
2023-11-22 13:07:32,913 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293100
2023-11-22 13:07:36,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1953993.3333333333, ans=0.125
2023-11-22 13:07:44,134 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.47 vs. limit=10.0
2023-11-22 13:07:44,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=1953993.3333333333, ans=0.0
2023-11-22 13:07:49,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1954060.0, ans=0.0
2023-11-22 13:07:58,964 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4550, loss[loss=0.07203, simple_loss=0.0942, pruned_loss=0.0169, audio_tagging_loss=0.008028, over 15024.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09508, pruned_loss=0.01534, audio_tagging_loss=0.009125, over 3054715.62 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:08:19,653 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.47 vs. limit=22.5
2023-11-22 13:08:21,115 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.37 vs. limit=22.5
2023-11-22 13:08:23,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=1954193.3333333333, ans=0.2
2023-11-22 13:08:37,160 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293150
2023-11-22 13:08:44,929 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.09 vs. limit=22.5
2023-11-22 13:08:45,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1954326.6666666667, ans=0.04949747468305833
2023-11-22 13:08:48,030 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 13:08:51,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1954393.3333333333, ans=0.125
2023-11-22 13:09:03,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=1954460.0, ans=0.0
2023-11-22 13:09:03,947 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4600, loss[loss=0.06011, simple_loss=0.08221, pruned_loss=0.01139, audio_tagging_loss=0.007607, over 15984.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.0955, pruned_loss=0.01531, audio_tagging_loss=0.009164, over 3059545.98 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:09:16,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=1954526.6666666667, ans=0.2
2023-11-22 13:09:20,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1954526.6666666667, ans=0.2
2023-11-22 13:09:20,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=1954526.6666666667, ans=0.5
2023-11-22 13:09:29,949 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:09:40,091 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.472e+01 8.168e+01 8.767e+01 9.419e+01 1.196e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 13:09:41,494 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293200
2023-11-22 13:10:00,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-22 13:10:05,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1954726.6666666667, ans=0.2
2023-11-22 13:10:07,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.76 vs. limit=6.0
2023-11-22 13:10:09,629 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4650, loss[loss=0.08284, simple_loss=0.1054, pruned_loss=0.01971, audio_tagging_loss=0.01041, over 14785.00 frames. ], tot_loss[loss=0.07165, simple_loss=0.09423, pruned_loss=0.01515, audio_tagging_loss=0.009385, over 3053394.35 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:10:46,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293250
2023-11-22 13:10:56,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=1954993.3333333333, ans=0.125
2023-11-22 13:11:01,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1955060.0, ans=0.125
2023-11-22 13:11:13,276 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4700, loss[loss=0.08645, simple_loss=0.12, pruned_loss=0.02, audio_tagging_loss=0.006477, over 15441.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09403, pruned_loss=0.01501, audio_tagging_loss=0.009504, over 3054867.70 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:11:25,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1955193.3333333333, ans=0.125
2023-11-22 13:11:35,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1955193.3333333333, ans=0.0
2023-11-22 13:11:49,579 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.837e+01 8.061e+01 8.708e+01 9.431e+01 1.267e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 13:11:50,505 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.72 vs. limit=10.0
2023-11-22 13:11:50,919 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293300
2023-11-22 13:11:55,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1955326.6666666667, ans=0.2
2023-11-22 13:12:04,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1955393.3333333333, ans=0.125
2023-11-22 13:12:17,209 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4750, loss[loss=0.0764, simple_loss=0.09052, pruned_loss=0.01409, audio_tagging_loss=0.01704, over 14122.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09356, pruned_loss=0.01504, audio_tagging_loss=0.009549, over 3051454.97 frames. ], batch size: 54, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:12:30,967 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=5.142e-03
2023-11-22 13:12:39,630 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.62 vs. limit=10.0
2023-11-22 13:12:54,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293350
2023-11-22 13:13:03,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1955660.0, ans=0.125
2023-11-22 13:13:23,053 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4800, loss[loss=0.07926, simple_loss=0.09743, pruned_loss=0.01984, audio_tagging_loss=0.0107, over 14891.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09396, pruned_loss=0.01506, audio_tagging_loss=0.009632, over 3056987.74 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:13:29,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1955793.3333333333, ans=0.0
2023-11-22 13:13:29,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1955793.3333333333, ans=0.125
2023-11-22 13:13:47,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1955926.6666666667, ans=0.2
2023-11-22 13:13:52,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.11 vs. limit=15.0
2023-11-22 13:13:58,939 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.542e+01 8.018e+01 8.726e+01 9.477e+01 1.215e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 13:14:00,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293400
2023-11-22 13:14:11,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1955993.3333333333, ans=0.1
2023-11-22 13:14:16,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=1956060.0, ans=0.0
2023-11-22 13:14:23,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.19 vs. limit=15.0
2023-11-22 13:14:28,250 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4850, loss[loss=0.08417, simple_loss=0.1078, pruned_loss=0.0183, audio_tagging_loss=0.01198, over 15760.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09443, pruned_loss=0.0152, audio_tagging_loss=0.009601, over 3054119.25 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:14:48,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1956193.3333333333, ans=0.2
2023-11-22 13:15:06,497 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293450
2023-11-22 13:15:17,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1956326.6666666667, ans=0.125
2023-11-22 13:15:33,147 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4900, loss[loss=0.07839, simple_loss=0.1102, pruned_loss=0.0146, audio_tagging_loss=0.008683, over 15670.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09436, pruned_loss=0.01511, audio_tagging_loss=0.009593, over 3050568.41 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:15:42,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1956460.0, ans=0.0
2023-11-22 13:15:45,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1956526.6666666667, ans=0.1
2023-11-22 13:15:45,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1956526.6666666667, ans=0.1
2023-11-22 13:15:57,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1956526.6666666667, ans=0.0
2023-11-22 13:16:04,446 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.30 vs. limit=10.0
2023-11-22 13:16:10,097 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.038e+01 8.107e+01 8.809e+01 9.600e+01 1.365e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 13:16:11,508 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293500
2023-11-22 13:16:19,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1956660.0, ans=0.1
2023-11-22 13:16:31,889 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.89 vs. limit=6.0
2023-11-22 13:16:34,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.91 vs. limit=12.0
2023-11-22 13:16:35,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=1956726.6666666667, ans=0.0
2023-11-22 13:16:38,730 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 4950, loss[loss=0.07699, simple_loss=0.09919, pruned_loss=0.0184, audio_tagging_loss=0.008988, over 15509.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09353, pruned_loss=0.01489, audio_tagging_loss=0.00963, over 3057329.39 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:16:39,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1956793.3333333333, ans=0.0
2023-11-22 13:16:43,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1956793.3333333333, ans=0.1
2023-11-22 13:16:50,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1956793.3333333333, ans=0.125
2023-11-22 13:16:56,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1956860.0, ans=0.125
2023-11-22 13:16:56,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1956860.0, ans=0.1
2023-11-22 13:17:05,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_na.min_abs, batch_count=1956926.6666666667, ans=0.02
2023-11-22 13:17:12,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=1956926.6666666667, ans=0.0
2023-11-22 13:17:16,573 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293550
2023-11-22 13:17:32,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1957060.0, ans=0.2
2023-11-22 13:17:44,346 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5000, loss[loss=0.06131, simple_loss=0.07861, pruned_loss=0.01225, audio_tagging_loss=0.009753, over 14864.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09319, pruned_loss=0.01479, audio_tagging_loss=0.00949, over 3051323.75 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:17:57,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1957193.3333333333, ans=0.125
2023-11-22 13:18:03,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1957193.3333333333, ans=0.125
2023-11-22 13:18:08,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=1957193.3333333333, ans=0.0
2023-11-22 13:18:20,735 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.208e+01 8.762e+01 9.402e+01 1.160e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 13:18:22,769 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293600
2023-11-22 13:18:33,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1957326.6666666667, ans=0.2
2023-11-22 13:18:46,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=1957393.3333333333, ans=0.125
2023-11-22 13:18:49,674 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5050, loss[loss=0.05495, simple_loss=0.06725, pruned_loss=0.008091, audio_tagging_loss=0.01324, over 14757.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09312, pruned_loss=0.01457, audio_tagging_loss=0.009368, over 3053600.42 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:18:53,895 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.52 vs. limit=12.0
2023-11-22 13:19:00,618 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0
2023-11-22 13:19:00,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.22 vs. limit=10.0
2023-11-22 13:19:25,226 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.86 vs. limit=15.0
2023-11-22 13:19:27,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293650
2023-11-22 13:19:45,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1957726.6666666667, ans=0.2
2023-11-22 13:19:53,999 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5100, loss[loss=0.04475, simple_loss=0.05801, pruned_loss=0.005055, audio_tagging_loss=0.01069, over 15436.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09328, pruned_loss=0.01469, audio_tagging_loss=0.009326, over 3051180.03 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:19:54,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1957793.3333333333, ans=0.125
2023-11-22 13:20:21,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.44 vs. limit=15.0
2023-11-22 13:20:28,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1957926.6666666667, ans=0.0
2023-11-22 13:20:30,027 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.490e+01 8.194e+01 8.730e+01 9.544e+01 1.434e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 13:20:31,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293700
2023-11-22 13:20:33,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys.whitening_limit, batch_count=1957993.3333333333, ans=6.0
2023-11-22 13:20:58,904 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5150, loss[loss=0.08719, simple_loss=0.1226, pruned_loss=0.01747, audio_tagging_loss=0.008442, over 16178.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09303, pruned_loss=0.01478, audio_tagging_loss=0.009356, over 3058624.66 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:21:13,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=1958193.3333333333, ans=0.2
2023-11-22 13:21:22,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.10 vs. limit=15.0
2023-11-22 13:21:32,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=1958260.0, ans=10.0
2023-11-22 13:21:35,845 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293750
2023-11-22 13:21:39,446 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.15 vs. limit=6.0
2023-11-22 13:21:41,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1958326.6666666667, ans=0.1
2023-11-22 13:21:48,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1958326.6666666667, ans=0.125
2023-11-22 13:21:50,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1958393.3333333333, ans=0.125
2023-11-22 13:21:55,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1958393.3333333333, ans=0.1
2023-11-22 13:22:03,625 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5200, loss[loss=0.07699, simple_loss=0.1048, pruned_loss=0.01859, audio_tagging_loss=0.005981, over 15101.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09375, pruned_loss=0.01497, audio_tagging_loss=0.009137, over 3052480.33 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:22:14,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.whiten.whitening_limit, batch_count=1958460.0, ans=12.0
2023-11-22 13:22:39,709 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.666e+01 8.256e+01 9.056e+01 9.741e+01 1.453e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 13:22:41,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293800
2023-11-22 13:23:08,261 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5250, loss[loss=0.05663, simple_loss=0.06176, pruned_loss=0.01268, audio_tagging_loss=0.01306, over 15179.00 frames. ], tot_loss[loss=0.07148, simple_loss=0.0948, pruned_loss=0.01507, audio_tagging_loss=0.009015, over 3057345.40 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:23:11,605 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.59 vs. limit=15.0
2023-11-22 13:23:34,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1958926.6666666667, ans=0.125
2023-11-22 13:23:42,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1958926.6666666667, ans=0.125
2023-11-22 13:23:46,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293850
2023-11-22 13:24:08,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=1959060.0, ans=0.0
2023-11-22 13:24:13,232 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5300, loss[loss=0.06305, simple_loss=0.08587, pruned_loss=0.01165, audio_tagging_loss=0.008465, over 14173.00 frames. ], tot_loss[loss=0.07219, simple_loss=0.09571, pruned_loss=0.01538, audio_tagging_loss=0.00896, over 3056674.07 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:24:27,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1959193.3333333333, ans=0.0
2023-11-22 13:24:37,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1959193.3333333333, ans=0.1
2023-11-22 13:24:38,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1959260.0, ans=0.0
2023-11-22 13:24:49,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1959260.0, ans=0.0
2023-11-22 13:24:50,534 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.56 vs. limit=22.5
2023-11-22 13:24:51,125 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.505e+01 9.104e+01 9.788e+01 1.254e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-22 13:24:51,294 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293900
2023-11-22 13:24:52,995 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.23 vs. limit=15.0
2023-11-22 13:25:18,128 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5350, loss[loss=0.08405, simple_loss=0.1063, pruned_loss=0.0185, audio_tagging_loss=0.01242, over 15314.00 frames. ], tot_loss[loss=0.0728, simple_loss=0.09648, pruned_loss=0.0157, audio_tagging_loss=0.008858, over 3049639.31 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:25:21,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1959460.0, ans=0.1
2023-11-22 13:25:21,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1959460.0, ans=0.1
2023-11-22 13:25:35,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1959526.6666666667, ans=0.2
2023-11-22 13:25:38,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1959526.6666666667, ans=0.125
2023-11-22 13:25:55,586 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 293950
2023-11-22 13:26:06,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1959660.0, ans=0.09899494936611666
2023-11-22 13:26:09,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1959726.6666666667, ans=0.125
2023-11-22 13:26:16,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1959726.6666666667, ans=0.1
2023-11-22 13:26:23,251 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5400, loss[loss=0.07965, simple_loss=0.1093, pruned_loss=0.01672, audio_tagging_loss=0.008284, over 16714.00 frames. ], tot_loss[loss=0.07244, simple_loss=0.09578, pruned_loss=0.01552, audio_tagging_loss=0.009028, over 3049989.69 frames. ], batch size: 61, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:26:59,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1959926.6666666667, ans=0.0
2023-11-22 13:27:00,601 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.841e+01 8.190e+01 8.796e+01 9.372e+01 1.218e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 13:27:00,770 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294000
2023-11-22 13:27:07,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.49 vs. limit=15.0
2023-11-22 13:27:15,719 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.06 vs. limit=22.5
2023-11-22 13:27:26,921 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:27:28,022 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5450, loss[loss=0.06139, simple_loss=0.07483, pruned_loss=0.01311, audio_tagging_loss=0.01087, over 15746.00 frames. ], tot_loss[loss=0.07221, simple_loss=0.09506, pruned_loss=0.01552, audio_tagging_loss=0.009167, over 3048110.20 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:27:40,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1960193.3333333333, ans=0.2
2023-11-22 13:28:06,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294050
2023-11-22 13:28:06,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1960326.6666666667, ans=0.1
2023-11-22 13:28:30,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1960393.3333333333, ans=0.125
2023-11-22 13:28:32,362 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5500, loss[loss=0.07202, simple_loss=0.09206, pruned_loss=0.01456, audio_tagging_loss=0.01142, over 15663.00 frames. ], tot_loss[loss=0.07235, simple_loss=0.09509, pruned_loss=0.01553, audio_tagging_loss=0.009276, over 3045801.41 frames. ], batch size: 59, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:28:37,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1960460.0, ans=0.2
2023-11-22 13:28:37,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.38 vs. limit=22.5
2023-11-22 13:28:42,858 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.09 vs. limit=15.0
2023-11-22 13:28:46,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1960526.6666666667, ans=0.05
2023-11-22 13:28:50,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1960526.6666666667, ans=0.2
2023-11-22 13:29:05,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1960593.3333333333, ans=0.125
2023-11-22 13:29:10,072 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.609e+01 8.387e+01 8.932e+01 9.464e+01 1.203e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 13:29:10,242 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294100
2023-11-22 13:29:12,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.70 vs. limit=6.0
2023-11-22 13:29:16,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1960660.0, ans=0.0
2023-11-22 13:29:17,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=1960660.0, ans=0.125
2023-11-22 13:29:23,277 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.30 vs. limit=10.0
2023-11-22 13:29:33,750 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:29:37,768 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5550, loss[loss=0.06308, simple_loss=0.08001, pruned_loss=0.01204, audio_tagging_loss=0.01103, over 14682.00 frames. ], tot_loss[loss=0.07263, simple_loss=0.09515, pruned_loss=0.0156, audio_tagging_loss=0.009465, over 3043309.62 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:29:41,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1960793.3333333333, ans=0.0
2023-11-22 13:30:10,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1960926.6666666667, ans=0.125
2023-11-22 13:30:14,768 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294150
2023-11-22 13:30:19,953 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.21 vs. limit=15.0
2023-11-22 13:30:23,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1960993.3333333333, ans=0.125
2023-11-22 13:30:35,037 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.70 vs. limit=15.0
2023-11-22 13:30:42,365 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5600, loss[loss=0.0611, simple_loss=0.07266, pruned_loss=0.01489, audio_tagging_loss=0.009882, over 16490.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09434, pruned_loss=0.01536, audio_tagging_loss=0.009691, over 3039909.28 frames. ], batch size: 64, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:30:44,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1961126.6666666667, ans=0.125
2023-11-22 13:31:06,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=1961260.0, ans=0.0
2023-11-22 13:31:09,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=1961260.0, ans=0.2
2023-11-22 13:31:13,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1961260.0, ans=0.2
2023-11-22 13:31:19,930 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294200
2023-11-22 13:31:21,013 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.888e+01 8.047e+01 8.683e+01 9.396e+01 1.171e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-22 13:31:25,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1961326.6666666667, ans=0.0
2023-11-22 13:31:27,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1961326.6666666667, ans=0.2
2023-11-22 13:31:28,724 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 13:31:29,412 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.36 vs. limit=15.0
2023-11-22 13:31:46,609 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5650, loss[loss=0.07368, simple_loss=0.09068, pruned_loss=0.01816, audio_tagging_loss=0.01018, over 16447.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09308, pruned_loss=0.01499, audio_tagging_loss=0.00976, over 3051556.29 frames. ], batch size: 61, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:31:51,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1961460.0, ans=0.2
2023-11-22 13:32:06,408 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.78 vs. limit=22.5
2023-11-22 13:32:24,605 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294250
2023-11-22 13:32:34,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1961660.0, ans=0.125
2023-11-22 13:32:37,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1961726.6666666667, ans=0.125
2023-11-22 13:32:40,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1961726.6666666667, ans=0.1
2023-11-22 13:32:51,473 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5700, loss[loss=0.08966, simple_loss=0.1162, pruned_loss=0.02306, audio_tagging_loss=0.008497, over 15280.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09355, pruned_loss=0.0151, audio_tagging_loss=0.009779, over 3057303.35 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:33:13,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1961860.0, ans=0.04949747468305833
2023-11-22 13:33:28,465 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294300
2023-11-22 13:33:29,497 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.416e+01 9.012e+01 9.956e+01 1.838e+02, threshold=1.802e+02, percent-clipped=1.0
2023-11-22 13:33:41,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1962060.0, ans=0.0
2023-11-22 13:33:48,835 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:33:50,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1962060.0, ans=0.125
2023-11-22 13:33:51,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1962060.0, ans=0.125
2023-11-22 13:33:51,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1962060.0, ans=0.125
2023-11-22 13:33:55,987 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5750, loss[loss=0.05631, simple_loss=0.07104, pruned_loss=0.01107, audio_tagging_loss=0.00972, over 14018.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09247, pruned_loss=0.01486, audio_tagging_loss=0.009702, over 3048362.08 frames. ], batch size: 54, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:34:00,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1962126.6666666667, ans=0.1
2023-11-22 13:34:08,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1962193.3333333333, ans=0.0
2023-11-22 13:34:10,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1962193.3333333333, ans=0.125
2023-11-22 13:34:18,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1962193.3333333333, ans=0.0
2023-11-22 13:34:18,497 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.46 vs. limit=12.0
2023-11-22 13:34:23,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0
2023-11-22 13:34:33,960 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294350
2023-11-22 13:34:39,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1962326.6666666667, ans=0.2
2023-11-22 13:34:51,508 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.70 vs. limit=15.0
2023-11-22 13:35:00,428 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5800, loss[loss=0.08304, simple_loss=0.1065, pruned_loss=0.01664, audio_tagging_loss=0.01315, over 15697.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09331, pruned_loss=0.01495, audio_tagging_loss=0.00949, over 3051018.23 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:35:31,288 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=15.37 vs. limit=15.0
2023-11-22 13:35:38,648 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294400
2023-11-22 13:35:39,735 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.175e+01 8.400e+01 8.933e+01 9.518e+01 1.972e+02, threshold=1.787e+02, percent-clipped=1.0
2023-11-22 13:35:56,755 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.86 vs. limit=15.0
2023-11-22 13:36:05,426 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5850, loss[loss=0.0636, simple_loss=0.08956, pruned_loss=0.01213, audio_tagging_loss=0.006686, over 14509.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09336, pruned_loss=0.01491, audio_tagging_loss=0.00937, over 3052456.94 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:36:42,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294450
2023-11-22 13:36:43,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=1962993.3333333333, ans=0.2
2023-11-22 13:37:07,359 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.57 vs. limit=10.0
2023-11-22 13:37:10,410 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5900, loss[loss=0.06924, simple_loss=0.09223, pruned_loss=0.0134, audio_tagging_loss=0.009727, over 14744.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09353, pruned_loss=0.01502, audio_tagging_loss=0.009331, over 3043144.74 frames. ], batch size: 56, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:37:21,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.62 vs. limit=22.5
2023-11-22 13:37:22,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1963193.3333333333, ans=0.1
2023-11-22 13:37:24,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1963193.3333333333, ans=0.07
2023-11-22 13:37:47,460 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294500
2023-11-22 13:37:49,047 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.759e+01 8.196e+01 8.876e+01 9.676e+01 1.604e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 13:37:51,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1963326.6666666667, ans=0.125
2023-11-22 13:37:57,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1963326.6666666667, ans=0.2
2023-11-22 13:38:03,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1963393.3333333333, ans=0.0
2023-11-22 13:38:14,678 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 5950, loss[loss=0.06706, simple_loss=0.08229, pruned_loss=0.01298, audio_tagging_loss=0.01293, over 15167.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.0932, pruned_loss=0.01492, audio_tagging_loss=0.009307, over 3039276.07 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:38:16,859 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.41 vs. limit=8.0
2023-11-22 13:38:20,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1963460.0, ans=0.2
2023-11-22 13:38:24,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1963460.0, ans=0.0
2023-11-22 13:38:40,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1963593.3333333333, ans=0.0
2023-11-22 13:38:52,682 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294550
2023-11-22 13:39:02,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1963660.0, ans=0.1
2023-11-22 13:39:07,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1963726.6666666667, ans=0.125
2023-11-22 13:39:10,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1963726.6666666667, ans=0.0
2023-11-22 13:39:19,054 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6000, loss[loss=0.06765, simple_loss=0.09728, pruned_loss=0.01049, audio_tagging_loss=0.008514, over 15988.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09398, pruned_loss=0.01487, audio_tagging_loss=0.009193, over 3042447.89 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:39:19,057 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 13:39:38,240 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.9594, 5.4677, 5.8321, 5.2290], device='cuda:0')
2023-11-22 13:40:01,010 INFO [train_asr.py:1253] (0/4) Epoch 25, validation: loss=0.05896, simple_loss=0.05155, pruned_loss=0.00512, audio_tagging_loss=0.02806, over 4681554.00 frames. 
2023-11-22 13:40:01,010 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 13:40:07,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1963793.3333333333, ans=0.125
2023-11-22 13:40:29,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1963926.6666666667, ans=0.1
2023-11-22 13:40:38,170 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294600
2023-11-22 13:40:39,175 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.185e+01 8.770e+01 9.296e+01 1.374e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 13:40:46,818 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 13:41:04,627 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6050, loss[loss=0.07175, simple_loss=0.09806, pruned_loss=0.01485, audio_tagging_loss=0.007872, over 14271.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09375, pruned_loss=0.01484, audio_tagging_loss=0.009132, over 3038417.30 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:41:22,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1964193.3333333333, ans=0.125
2023-11-22 13:41:31,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=22.91 vs. limit=22.5
2023-11-22 13:41:42,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294650
2023-11-22 13:42:09,287 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6100, loss[loss=0.06968, simple_loss=0.09267, pruned_loss=0.01533, audio_tagging_loss=0.008015, over 14780.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09276, pruned_loss=0.01474, audio_tagging_loss=0.009163, over 3031651.43 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:42:38,552 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.19 vs. limit=15.0
2023-11-22 13:42:43,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1964593.3333333333, ans=0.125
2023-11-22 13:42:46,779 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294700
2023-11-22 13:42:47,812 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.151e+01 8.939e+01 9.703e+01 1.163e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 13:42:48,239 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:42:52,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=1964660.0, ans=0.125
2023-11-22 13:43:13,357 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6150, loss[loss=0.06702, simple_loss=0.09576, pruned_loss=0.0118, audio_tagging_loss=0.00734, over 16512.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09267, pruned_loss=0.01467, audio_tagging_loss=0.00923, over 3035773.46 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:43:40,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1964926.6666666667, ans=0.125
2023-11-22 13:43:40,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1964926.6666666667, ans=0.125
2023-11-22 13:43:40,731 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.53 vs. limit=12.0
2023-11-22 13:43:42,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1964926.6666666667, ans=0.2
2023-11-22 13:43:42,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1964926.6666666667, ans=0.125
2023-11-22 13:43:44,182 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.46 vs. limit=22.5
2023-11-22 13:43:49,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1964926.6666666667, ans=0.125
2023-11-22 13:43:50,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294750
2023-11-22 13:44:08,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1965060.0, ans=0.125
2023-11-22 13:44:13,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=1965060.0, ans=15.0
2023-11-22 13:44:13,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1965060.0, ans=0.125
2023-11-22 13:44:15,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1965060.0, ans=0.0
2023-11-22 13:44:18,495 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6200, loss[loss=0.05715, simple_loss=0.07148, pruned_loss=0.01076, audio_tagging_loss=0.01065, over 14951.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09284, pruned_loss=0.01493, audio_tagging_loss=0.009357, over 3039420.94 frames. ], batch size: 58, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:44:26,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=1965126.6666666667, ans=0.07
2023-11-22 13:44:30,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1965193.3333333333, ans=0.0
2023-11-22 13:44:30,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1965193.3333333333, ans=0.125
2023-11-22 13:44:32,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1965193.3333333333, ans=0.125
2023-11-22 13:44:40,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=1965193.3333333333, ans=0.125
2023-11-22 13:44:42,578 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.26 vs. limit=15.0
2023-11-22 13:44:56,160 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294800
2023-11-22 13:44:57,267 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 7.998e+01 8.582e+01 9.232e+01 1.087e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-22 13:45:02,357 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.26 vs. limit=10.0
2023-11-22 13:45:23,772 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6250, loss[loss=0.07979, simple_loss=0.09988, pruned_loss=0.0196, audio_tagging_loss=0.01025, over 16997.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09307, pruned_loss=0.01493, audio_tagging_loss=0.009421, over 3048427.51 frames. ], batch size: 63, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:45:29,484 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.78 vs. limit=15.0
2023-11-22 13:45:41,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1965526.6666666667, ans=0.125
2023-11-22 13:45:48,712 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.54 vs. limit=15.0
2023-11-22 13:46:01,151 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294850
2023-11-22 13:46:01,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1965660.0, ans=0.0
2023-11-22 13:46:02,791 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.26 vs. limit=10.0
2023-11-22 13:46:12,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1965660.0, ans=0.0
2023-11-22 13:46:23,561 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:46:28,194 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6300, loss[loss=0.05651, simple_loss=0.07387, pruned_loss=0.005954, audio_tagging_loss=0.01362, over 14395.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09391, pruned_loss=0.01511, audio_tagging_loss=0.009553, over 3049303.51 frames. ], batch size: 55, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:46:49,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1965860.0, ans=0.1
2023-11-22 13:46:59,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1965926.6666666667, ans=0.125
2023-11-22 13:47:03,991 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.58 vs. limit=22.5
2023-11-22 13:47:05,625 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294900
2023-11-22 13:47:07,901 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.416e+01 8.996e+01 9.619e+01 1.207e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 13:47:09,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1965993.3333333333, ans=0.0
2023-11-22 13:47:12,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1965993.3333333333, ans=0.0
2023-11-22 13:47:32,520 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6350, loss[loss=0.05791, simple_loss=0.08254, pruned_loss=0.007313, audio_tagging_loss=0.009331, over 15703.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09315, pruned_loss=0.015, audio_tagging_loss=0.009622, over 3041090.52 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 16.0
2023-11-22 13:47:34,613 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.37 vs. limit=22.5
2023-11-22 13:47:51,684 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.90 vs. limit=15.0
2023-11-22 13:48:10,980 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 294950
2023-11-22 13:48:30,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1966393.3333333333, ans=0.125
2023-11-22 13:48:38,029 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6400, loss[loss=0.06617, simple_loss=0.08477, pruned_loss=0.01162, audio_tagging_loss=0.01216, over 14912.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.09296, pruned_loss=0.01486, audio_tagging_loss=0.009706, over 3040137.04 frames. ], batch size: 57, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:48:50,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1966526.6666666667, ans=0.1
2023-11-22 13:48:59,205 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.57 vs. limit=15.0
2023-11-22 13:49:09,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1966593.3333333333, ans=0.2
2023-11-22 13:49:15,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295000
2023-11-22 13:49:17,967 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.103e+01 8.926e+01 9.580e+01 1.211e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 13:49:43,155 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6450, loss[loss=0.0813, simple_loss=0.1067, pruned_loss=0.01842, audio_tagging_loss=0.009546, over 16126.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09283, pruned_loss=0.01476, audio_tagging_loss=0.009756, over 3038595.14 frames. ], batch size: 60, lr: 2.73e-03, grad_scale: 32.0
2023-11-22 13:49:48,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1966793.3333333333, ans=0.2
2023-11-22 13:50:19,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1966926.6666666667, ans=0.125
2023-11-22 13:50:20,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295050
2023-11-22 13:50:21,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1966993.3333333333, ans=0.125
2023-11-22 13:50:47,838 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6500, loss[loss=0.07749, simple_loss=0.1003, pruned_loss=0.01735, audio_tagging_loss=0.01001, over 14739.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09299, pruned_loss=0.0148, audio_tagging_loss=0.009721, over 3042722.95 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:50:52,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1967126.6666666667, ans=0.1
2023-11-22 13:51:03,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1967193.3333333333, ans=0.125
2023-11-22 13:51:06,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1967193.3333333333, ans=0.1
2023-11-22 13:51:23,391 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.41 vs. limit=15.0
2023-11-22 13:51:25,065 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295100
2023-11-22 13:51:27,953 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.089e+01 8.137e+01 8.907e+01 9.781e+01 1.377e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-22 13:51:43,335 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.15 vs. limit=22.5
2023-11-22 13:51:51,248 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6550, loss[loss=0.07026, simple_loss=0.1028, pruned_loss=0.01125, audio_tagging_loss=0.0076, over 14841.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09349, pruned_loss=0.01496, audio_tagging_loss=0.009622, over 3040183.16 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:51:53,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1967460.0, ans=0.125
2023-11-22 13:52:02,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1967460.0, ans=0.1
2023-11-22 13:52:06,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1967526.6666666667, ans=0.2
2023-11-22 13:52:12,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1967526.6666666667, ans=0.1
2023-11-22 13:52:29,182 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295150
2023-11-22 13:52:50,773 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=17.99 vs. limit=22.5
2023-11-22 13:52:53,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=1967726.6666666667, ans=0.0
2023-11-22 13:52:56,823 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6600, loss[loss=0.06999, simple_loss=0.08583, pruned_loss=0.01662, audio_tagging_loss=0.01046, over 15670.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09332, pruned_loss=0.01484, audio_tagging_loss=0.009504, over 3040894.73 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:52:58,641 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.92 vs. limit=22.5
2023-11-22 13:53:24,947 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 13:53:29,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1967926.6666666667, ans=0.0
2023-11-22 13:53:34,490 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295200
2023-11-22 13:53:37,136 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.628e+01 8.017e+01 8.525e+01 9.445e+01 1.151e+02, threshold=1.705e+02, percent-clipped=0.0
2023-11-22 13:53:38,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1967993.3333333333, ans=10.0
2023-11-22 13:53:41,460 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.53 vs. limit=15.0
2023-11-22 13:53:45,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1967993.3333333333, ans=0.125
2023-11-22 13:54:02,052 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6650, loss[loss=0.09735, simple_loss=0.1435, pruned_loss=0.01977, audio_tagging_loss=0.005822, over 16408.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09436, pruned_loss=0.01513, audio_tagging_loss=0.009382, over 3045200.24 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:54:05,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1968126.6666666667, ans=0.125
2023-11-22 13:54:13,679 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.50 vs. limit=15.0
2023-11-22 13:54:14,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=1968193.3333333333, ans=0.2
2023-11-22 13:54:39,556 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295250
2023-11-22 13:54:40,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1968326.6666666667, ans=0.125
2023-11-22 13:54:43,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1968326.6666666667, ans=0.1
2023-11-22 13:54:47,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1968326.6666666667, ans=0.0
2023-11-22 13:54:52,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1968393.3333333333, ans=0.125
2023-11-22 13:55:01,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1968393.3333333333, ans=0.0
2023-11-22 13:55:04,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1968460.0, ans=0.125
2023-11-22 13:55:05,853 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6700, loss[loss=0.06779, simple_loss=0.09645, pruned_loss=0.01097, audio_tagging_loss=0.0086, over 15168.00 frames. ], tot_loss[loss=0.07241, simple_loss=0.09558, pruned_loss=0.01535, audio_tagging_loss=0.009268, over 3051562.24 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:55:24,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.35 vs. limit=15.0
2023-11-22 13:55:34,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1968593.3333333333, ans=0.07
2023-11-22 13:55:44,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295300
2023-11-22 13:55:45,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1968660.0, ans=0.125
2023-11-22 13:55:46,361 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.891e+01 8.344e+01 8.911e+01 9.672e+01 1.240e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 13:56:11,715 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6750, loss[loss=0.09737, simple_loss=0.1323, pruned_loss=0.02096, audio_tagging_loss=0.01027, over 15918.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09391, pruned_loss=0.01493, audio_tagging_loss=0.009291, over 3057529.15 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:56:24,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.12 vs. limit=22.5
2023-11-22 13:56:33,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1968860.0, ans=0.125
2023-11-22 13:56:47,803 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295350
2023-11-22 13:57:01,260 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.32 vs. limit=15.0
2023-11-22 13:57:04,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1969060.0, ans=0.0
2023-11-22 13:57:10,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1969060.0, ans=0.1
2023-11-22 13:57:14,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=1969126.6666666667, ans=0.125
2023-11-22 13:57:15,702 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6800, loss[loss=0.08234, simple_loss=0.1029, pruned_loss=0.02105, audio_tagging_loss=0.009844, over 14709.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09337, pruned_loss=0.01496, audio_tagging_loss=0.009299, over 3047463.90 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:57:16,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1969126.6666666667, ans=0.0
2023-11-22 13:57:53,505 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295400
2023-11-22 13:57:56,108 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.834e+01 8.041e+01 8.623e+01 9.531e+01 1.254e+02, threshold=1.725e+02, percent-clipped=0.0
2023-11-22 13:58:06,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1969393.3333333333, ans=0.125
2023-11-22 13:58:20,152 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6850, loss[loss=0.06149, simple_loss=0.0777, pruned_loss=0.009841, audio_tagging_loss=0.0128, over 14591.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.0928, pruned_loss=0.01483, audio_tagging_loss=0.009345, over 3050737.43 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:58:32,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1969526.6666666667, ans=0.125
2023-11-22 13:58:34,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=1969526.6666666667, ans=0.125
2023-11-22 13:58:51,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1969593.3333333333, ans=0.1
2023-11-22 13:58:57,996 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295450
2023-11-22 13:59:24,615 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6900, loss[loss=0.07797, simple_loss=0.09658, pruned_loss=0.01937, audio_tagging_loss=0.01031, over 16237.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09273, pruned_loss=0.0147, audio_tagging_loss=0.00934, over 3055364.78 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 13:59:26,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=1969793.3333333333, ans=0.0
2023-11-22 13:59:53,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1969926.6666666667, ans=0.1
2023-11-22 14:00:02,181 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295500
2023-11-22 14:00:04,519 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.094e+01 8.795e+01 9.359e+01 1.337e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 14:00:15,063 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:00:17,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=1970060.0, ans=0.0
2023-11-22 14:00:30,087 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 6950, loss[loss=0.102, simple_loss=0.1352, pruned_loss=0.02828, audio_tagging_loss=0.006067, over 15000.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09367, pruned_loss=0.01481, audio_tagging_loss=0.009225, over 3053311.40 frames. ], batch size: 52, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:00:32,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1970126.6666666667, ans=0.125
2023-11-22 14:00:57,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=1970260.0, ans=0.125
2023-11-22 14:01:05,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=1970260.0, ans=0.0
2023-11-22 14:01:06,998 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295550
2023-11-22 14:01:17,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1970326.6666666667, ans=0.0
2023-11-22 14:01:31,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=1970393.3333333333, ans=0.2
2023-11-22 14:01:33,778 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7000, loss[loss=0.1043, simple_loss=0.136, pruned_loss=0.02969, audio_tagging_loss=0.006645, over 15299.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09387, pruned_loss=0.01492, audio_tagging_loss=0.009268, over 3050760.75 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:01:36,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1970460.0, ans=0.0
2023-11-22 14:01:55,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=1970526.6666666667, ans=0.0
2023-11-22 14:02:09,326 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.56 vs. limit=15.0
2023-11-22 14:02:12,252 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295600
2023-11-22 14:02:14,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1970660.0, ans=15.0
2023-11-22 14:02:14,864 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.282e+01 8.833e+01 9.554e+01 1.299e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 14:02:32,118 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:02:38,783 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7050, loss[loss=0.08248, simple_loss=0.1099, pruned_loss=0.01791, audio_tagging_loss=0.009612, over 14959.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09427, pruned_loss=0.01516, audio_tagging_loss=0.00928, over 3048096.11 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:02:48,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1970793.3333333333, ans=0.0
2023-11-22 14:02:51,946 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:03:03,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1970860.0, ans=0.1
2023-11-22 14:03:16,326 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295650
2023-11-22 14:03:28,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=1970993.3333333333, ans=0.125
2023-11-22 14:03:33,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=1971060.0, ans=0.04949747468305833
2023-11-22 14:03:37,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1971060.0, ans=0.0
2023-11-22 14:03:40,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=1971060.0, ans=0.125
2023-11-22 14:03:43,542 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7100, loss[loss=0.08502, simple_loss=0.1169, pruned_loss=0.01851, audio_tagging_loss=0.00806, over 15859.00 frames. ], tot_loss[loss=0.0715, simple_loss=0.09397, pruned_loss=0.01514, audio_tagging_loss=0.009372, over 3054751.73 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:03:51,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=1971126.6666666667, ans=0.2
2023-11-22 14:04:04,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=1971193.3333333333, ans=0.125
2023-11-22 14:04:06,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1971193.3333333333, ans=0.04949747468305833
2023-11-22 14:04:18,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1971260.0, ans=0.125
2023-11-22 14:04:18,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1971260.0, ans=0.1
2023-11-22 14:04:21,115 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295700
2023-11-22 14:04:24,053 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 8.394e+01 8.961e+01 9.557e+01 1.218e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 14:04:47,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.45 vs. limit=22.5
2023-11-22 14:04:47,939 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7150, loss[loss=0.09146, simple_loss=0.1204, pruned_loss=0.02008, audio_tagging_loss=0.01118, over 15873.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09456, pruned_loss=0.01526, audio_tagging_loss=0.009379, over 3051698.92 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:05:03,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1971526.6666666667, ans=0.125
2023-11-22 14:05:09,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1971526.6666666667, ans=0.125
2023-11-22 14:05:17,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1971593.3333333333, ans=0.2
2023-11-22 14:05:22,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1971593.3333333333, ans=0.2
2023-11-22 14:05:26,327 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295750
2023-11-22 14:05:26,813 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.16 vs. limit=15.0
2023-11-22 14:05:41,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=1971726.6666666667, ans=0.125
2023-11-22 14:05:51,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=1971793.3333333333, ans=0.0
2023-11-22 14:05:52,792 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7200, loss[loss=0.07268, simple_loss=0.09523, pruned_loss=0.01691, audio_tagging_loss=0.008165, over 15865.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09389, pruned_loss=0.01512, audio_tagging_loss=0.009488, over 3052762.65 frames. ], batch size: 61, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:05:53,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1971793.3333333333, ans=0.125
2023-11-22 14:06:30,680 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295800
2023-11-22 14:06:33,411 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.672e+01 8.039e+01 8.548e+01 9.116e+01 1.085e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 14:06:39,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=1971993.3333333333, ans=10.0
2023-11-22 14:06:49,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1972060.0, ans=0.125
2023-11-22 14:06:58,273 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7250, loss[loss=0.05975, simple_loss=0.07995, pruned_loss=0.009405, audio_tagging_loss=0.01037, over 14914.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09408, pruned_loss=0.01506, audio_tagging_loss=0.009518, over 3045145.48 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:07:00,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=1972126.6666666667, ans=0.05
2023-11-22 14:07:11,239 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.40 vs. limit=15.0
2023-11-22 14:07:16,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.34 vs. limit=22.5
2023-11-22 14:07:20,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1972193.3333333333, ans=0.2
2023-11-22 14:07:36,055 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295850
2023-11-22 14:08:03,635 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7300, loss[loss=0.08137, simple_loss=0.1139, pruned_loss=0.01802, audio_tagging_loss=0.006397, over 16188.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09447, pruned_loss=0.0151, audio_tagging_loss=0.009351, over 3049488.31 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:08:19,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=1972526.6666666667, ans=0.2
2023-11-22 14:08:21,084 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.75 vs. limit=8.0
2023-11-22 14:08:40,779 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295900
2023-11-22 14:08:44,859 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.276e+01 8.894e+01 9.525e+01 1.269e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 14:08:48,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.78 vs. limit=15.0
2023-11-22 14:08:50,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.77 vs. limit=10.0
2023-11-22 14:09:05,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1972726.6666666667, ans=0.125
2023-11-22 14:09:08,256 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7350, loss[loss=0.06689, simple_loss=0.0907, pruned_loss=0.01476, audio_tagging_loss=0.006776, over 14729.00 frames. ], tot_loss[loss=0.07116, simple_loss=0.09407, pruned_loss=0.01493, audio_tagging_loss=0.009194, over 3049700.36 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:09:26,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=1972860.0, ans=0.04949747468305833
2023-11-22 14:09:31,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1972860.0, ans=0.125
2023-11-22 14:09:32,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=1972926.6666666667, ans=0.025
2023-11-22 14:09:40,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1972926.6666666667, ans=0.1
2023-11-22 14:09:45,381 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 295950
2023-11-22 14:09:45,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1972993.3333333333, ans=0.0
2023-11-22 14:10:09,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=1973060.0, ans=0.0
2023-11-22 14:10:12,136 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7400, loss[loss=0.06452, simple_loss=0.0821, pruned_loss=0.01301, audio_tagging_loss=0.01046, over 15791.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09431, pruned_loss=0.01504, audio_tagging_loss=0.00917, over 3052472.02 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:10:25,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1973193.3333333333, ans=0.0
2023-11-22 14:10:41,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1973260.0, ans=0.125
2023-11-22 14:10:44,341 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:10:45,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1973260.0, ans=0.125
2023-11-22 14:10:49,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296000
2023-11-22 14:10:51,102 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-296000.pt
2023-11-22 14:10:55,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1973326.6666666667, ans=0.1
2023-11-22 14:10:56,596 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.500e+01 8.081e+01 8.804e+01 9.585e+01 1.209e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 14:11:19,954 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7450, loss[loss=0.09554, simple_loss=0.138, pruned_loss=0.02393, audio_tagging_loss=0.002625, over 16586.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09386, pruned_loss=0.01496, audio_tagging_loss=0.00904, over 3050733.83 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:11:25,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1973460.0, ans=0.0
2023-11-22 14:11:37,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=1973526.6666666667, ans=0.2
2023-11-22 14:11:56,981 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.03 vs. limit=15.0
2023-11-22 14:11:57,595 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296050
2023-11-22 14:11:57,822 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:11:58,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1973660.0, ans=0.0
2023-11-22 14:12:19,178 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.24 vs. limit=22.5
2023-11-22 14:12:24,573 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7500, loss[loss=0.08608, simple_loss=0.1111, pruned_loss=0.02093, audio_tagging_loss=0.00961, over 14844.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09279, pruned_loss=0.01484, audio_tagging_loss=0.009113, over 3045783.79 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:12:35,580 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.03 vs. limit=22.5
2023-11-22 14:12:43,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1973860.0, ans=0.0
2023-11-22 14:13:02,168 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296100
2023-11-22 14:13:06,437 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.270e+01 8.171e+01 8.790e+01 9.530e+01 1.185e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 14:13:16,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1974060.0, ans=0.0
2023-11-22 14:13:19,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1974060.0, ans=0.2
2023-11-22 14:13:25,023 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.92 vs. limit=15.0
2023-11-22 14:13:29,563 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7550, loss[loss=0.05353, simple_loss=0.06522, pruned_loss=0.01096, audio_tagging_loss=0.009954, over 14222.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09371, pruned_loss=0.01504, audio_tagging_loss=0.009054, over 3048864.80 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:13:46,849 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.94 vs. limit=12.0
2023-11-22 14:14:07,111 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296150
2023-11-22 14:14:32,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1974460.0, ans=0.125
2023-11-22 14:14:33,783 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7600, loss[loss=0.07763, simple_loss=0.104, pruned_loss=0.01734, audio_tagging_loss=0.0083, over 16240.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09329, pruned_loss=0.01487, audio_tagging_loss=0.009044, over 3049331.90 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:15:06,317 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.81 vs. limit=22.5
2023-11-22 14:15:11,964 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296200
2023-11-22 14:15:15,783 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.584e+01 8.244e+01 8.772e+01 9.413e+01 1.274e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 14:15:20,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1974660.0, ans=0.0
2023-11-22 14:15:21,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1974660.0, ans=0.125
2023-11-22 14:15:24,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=1974660.0, ans=0.125
2023-11-22 14:15:39,244 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7650, loss[loss=0.05689, simple_loss=0.07696, pruned_loss=0.009672, audio_tagging_loss=0.008739, over 15144.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09202, pruned_loss=0.01457, audio_tagging_loss=0.009144, over 3045911.38 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:15:55,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=1974860.0, ans=0.0
2023-11-22 14:16:11,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1974926.6666666667, ans=0.1
2023-11-22 14:16:17,253 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296250
2023-11-22 14:16:37,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1975060.0, ans=0.125
2023-11-22 14:16:40,019 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.12 vs. limit=15.0
2023-11-22 14:16:44,142 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7700, loss[loss=0.0661, simple_loss=0.09165, pruned_loss=0.01069, audio_tagging_loss=0.009585, over 15250.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09266, pruned_loss=0.01469, audio_tagging_loss=0.009067, over 3042799.10 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:17:21,597 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296300
2023-11-22 14:17:25,652 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.979e+01 8.266e+01 8.985e+01 9.649e+01 1.326e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 14:17:26,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1975326.6666666667, ans=0.125
2023-11-22 14:17:38,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1975393.3333333333, ans=0.125
2023-11-22 14:17:48,834 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7750, loss[loss=0.07541, simple_loss=0.09499, pruned_loss=0.01194, audio_tagging_loss=0.01598, over 14168.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09198, pruned_loss=0.01461, audio_tagging_loss=0.009215, over 3034563.26 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 32.0
2023-11-22 14:17:49,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1975460.0, ans=0.0
2023-11-22 14:17:49,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1975460.0, ans=0.1
2023-11-22 14:17:49,127 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:18:04,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1975526.6666666667, ans=0.1
2023-11-22 14:18:09,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=1975526.6666666667, ans=10.0
2023-11-22 14:18:09,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1975526.6666666667, ans=0.0
2023-11-22 14:18:26,540 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296350
2023-11-22 14:18:28,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=1975660.0, ans=0.125
2023-11-22 14:18:45,226 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.68 vs. limit=15.0
2023-11-22 14:18:50,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=1975726.6666666667, ans=0.125
2023-11-22 14:18:52,719 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7800, loss[loss=0.07854, simple_loss=0.1073, pruned_loss=0.0142, audio_tagging_loss=0.0107, over 15693.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09267, pruned_loss=0.01469, audio_tagging_loss=0.009239, over 3040518.28 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:19:05,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=1975860.0, ans=0.0
2023-11-22 14:19:30,995 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296400
2023-11-22 14:19:36,047 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 7.916e+01 8.644e+01 9.515e+01 1.205e+02, threshold=1.729e+02, percent-clipped=0.0
2023-11-22 14:19:48,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1976060.0, ans=0.0
2023-11-22 14:19:49,097 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.23 vs. limit=15.0
2023-11-22 14:19:58,145 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7850, loss[loss=0.06073, simple_loss=0.07536, pruned_loss=0.01054, audio_tagging_loss=0.01251, over 15565.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09292, pruned_loss=0.01474, audio_tagging_loss=0.009233, over 3044841.41 frames. ], batch size: 61, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:20:06,364 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:20:26,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1976260.0, ans=0.1
2023-11-22 14:20:28,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1976260.0, ans=0.1
2023-11-22 14:20:29,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.88 vs. limit=10.0
2023-11-22 14:20:30,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1976260.0, ans=0.125
2023-11-22 14:20:34,923 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296450
2023-11-22 14:20:42,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1976326.6666666667, ans=0.0
2023-11-22 14:20:43,471 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:20:43,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1976326.6666666667, ans=0.0
2023-11-22 14:20:50,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1976393.3333333333, ans=0.125
2023-11-22 14:20:57,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=1976393.3333333333, ans=0.125
2023-11-22 14:21:01,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1976460.0, ans=0.125
2023-11-22 14:21:02,757 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7900, loss[loss=0.07553, simple_loss=0.1036, pruned_loss=0.01464, audio_tagging_loss=0.009114, over 14756.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.0935, pruned_loss=0.01482, audio_tagging_loss=0.009317, over 3047856.79 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:21:39,908 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296500
2023-11-22 14:21:40,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1976660.0, ans=0.0
2023-11-22 14:21:44,986 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.338e+01 8.490e+01 8.875e+01 9.857e+01 1.314e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 14:22:01,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1976726.6666666667, ans=0.1
2023-11-22 14:22:06,363 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 7950, loss[loss=0.05846, simple_loss=0.08108, pruned_loss=0.009102, audio_tagging_loss=0.008822, over 15650.00 frames. ], tot_loss[loss=0.07172, simple_loss=0.09438, pruned_loss=0.01514, audio_tagging_loss=0.009388, over 3044790.27 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 8.0
2023-11-22 14:22:21,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.90 vs. limit=15.0
2023-11-22 14:22:23,728 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:22:44,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296550
2023-11-22 14:23:04,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1977060.0, ans=0.2
2023-11-22 14:23:10,972 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8000, loss[loss=0.08718, simple_loss=0.1124, pruned_loss=0.02069, audio_tagging_loss=0.01027, over 15263.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09373, pruned_loss=0.01507, audio_tagging_loss=0.009455, over 3043816.72 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:23:11,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1977126.6666666667, ans=0.125
2023-11-22 14:23:31,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1977193.3333333333, ans=0.125
2023-11-22 14:23:47,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1977260.0, ans=0.0
2023-11-22 14:23:48,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296600
2023-11-22 14:23:55,288 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.359e+01 8.469e+01 8.936e+01 9.726e+01 1.360e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 14:24:05,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=1977393.3333333333, ans=0.125
2023-11-22 14:24:11,696 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:24:14,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=1977393.3333333333, ans=0.05
2023-11-22 14:24:16,310 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8050, loss[loss=0.06875, simple_loss=0.07252, pruned_loss=0.02155, audio_tagging_loss=0.01094, over 15103.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.0933, pruned_loss=0.01503, audio_tagging_loss=0.009546, over 3043703.57 frames. ], batch size: 59, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:24:21,214 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.07 vs. limit=15.0
2023-11-22 14:24:35,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=1977526.6666666667, ans=0.2
2023-11-22 14:24:38,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=1977526.6666666667, ans=0.0
2023-11-22 14:24:52,945 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296650
2023-11-22 14:25:08,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1977726.6666666667, ans=0.1
2023-11-22 14:25:20,016 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8100, loss[loss=0.06743, simple_loss=0.09332, pruned_loss=0.01117, audio_tagging_loss=0.009601, over 14114.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09367, pruned_loss=0.01508, audio_tagging_loss=0.009417, over 3039848.49 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:25:30,450 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.94 vs. limit=15.0
2023-11-22 14:25:31,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1977860.0, ans=0.1
2023-11-22 14:25:34,945 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.75 vs. limit=15.0
2023-11-22 14:25:57,408 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296700
2023-11-22 14:25:57,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1977993.3333333333, ans=0.125
2023-11-22 14:26:03,383 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.691e+01 8.263e+01 8.996e+01 9.612e+01 1.203e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 14:26:17,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1978060.0, ans=0.1
2023-11-22 14:26:18,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=1978060.0, ans=0.0
2023-11-22 14:26:23,621 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8150, loss[loss=0.07294, simple_loss=0.09995, pruned_loss=0.01438, audio_tagging_loss=0.008588, over 15154.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09394, pruned_loss=0.01512, audio_tagging_loss=0.009371, over 3034095.91 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:26:38,367 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.01 vs. limit=15.0
2023-11-22 14:26:39,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1978193.3333333333, ans=0.2
2023-11-22 14:27:00,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296750
2023-11-22 14:27:23,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1978393.3333333333, ans=0.0
2023-11-22 14:27:27,877 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8200, loss[loss=0.09785, simple_loss=0.1277, pruned_loss=0.02542, audio_tagging_loss=0.008561, over 16895.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09318, pruned_loss=0.01495, audio_tagging_loss=0.009422, over 3037275.36 frames. ], batch size: 61, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:27:28,409 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.07 vs. limit=15.0
2023-11-22 14:27:29,138 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:27:36,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=1978460.0, ans=0.0
2023-11-22 14:27:37,631 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.29 vs. limit=12.0
2023-11-22 14:28:04,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296800
2023-11-22 14:28:08,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=1978660.0, ans=0.0
2023-11-22 14:28:10,694 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 7.904e+01 8.816e+01 9.431e+01 1.476e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 14:28:12,817 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:28:14,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1978660.0, ans=0.1
2023-11-22 14:28:18,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1978726.6666666667, ans=0.04949747468305833
2023-11-22 14:28:19,522 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:28:22,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=1978726.6666666667, ans=0.2
2023-11-22 14:28:32,354 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8250, loss[loss=0.07366, simple_loss=0.09938, pruned_loss=0.01464, audio_tagging_loss=0.009333, over 14385.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09319, pruned_loss=0.01499, audio_tagging_loss=0.009397, over 3043721.46 frames. ], batch size: 54, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:28:37,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1978793.3333333333, ans=0.125
2023-11-22 14:28:38,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=1978793.3333333333, ans=0.0
2023-11-22 14:28:43,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0
2023-11-22 14:28:46,425 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.59 vs. limit=12.0
2023-11-22 14:29:09,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296850
2023-11-22 14:29:11,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1978993.3333333333, ans=0.0
2023-11-22 14:29:20,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1978993.3333333333, ans=0.0
2023-11-22 14:29:24,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=1979060.0, ans=0.0
2023-11-22 14:29:30,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1979060.0, ans=0.0
2023-11-22 14:29:35,947 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8300, loss[loss=0.07366, simple_loss=0.1021, pruned_loss=0.01266, audio_tagging_loss=0.009944, over 14914.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09351, pruned_loss=0.01487, audio_tagging_loss=0.009296, over 3043580.60 frames. ], batch size: 55, lr: 2.72e-03, grad_scale: 8.0
2023-11-22 14:29:42,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1979126.6666666667, ans=0.125
2023-11-22 14:30:03,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1979260.0, ans=0.125
2023-11-22 14:30:08,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1979260.0, ans=0.0
2023-11-22 14:30:13,327 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296900
2023-11-22 14:30:20,395 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.061e+01 8.620e+01 9.639e+01 1.216e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-22 14:30:27,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1979393.3333333333, ans=0.2
2023-11-22 14:30:39,599 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8350, loss[loss=0.05633, simple_loss=0.06583, pruned_loss=0.009613, audio_tagging_loss=0.0138, over 14232.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09416, pruned_loss=0.01493, audio_tagging_loss=0.009289, over 3047630.74 frames. ], batch size: 60, lr: 2.72e-03, grad_scale: 8.0
2023-11-22 14:30:39,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=1979460.0, ans=0.0
2023-11-22 14:30:50,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1979460.0, ans=0.1
2023-11-22 14:30:58,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1979526.6666666667, ans=0.125
2023-11-22 14:31:17,057 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 296950
2023-11-22 14:31:28,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1979660.0, ans=0.0
2023-11-22 14:31:32,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1979726.6666666667, ans=0.0
2023-11-22 14:31:40,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1979726.6666666667, ans=0.0
2023-11-22 14:31:42,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1979793.3333333333, ans=0.1
2023-11-22 14:31:43,421 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8400, loss[loss=0.06331, simple_loss=0.07634, pruned_loss=0.01271, audio_tagging_loss=0.01243, over 16302.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09274, pruned_loss=0.01466, audio_tagging_loss=0.009325, over 3051743.82 frames. ], batch size: 62, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:31:51,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=1979793.3333333333, ans=0.0
2023-11-22 14:31:57,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1979860.0, ans=0.09899494936611666
2023-11-22 14:32:05,319 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.72 vs. limit=6.0
2023-11-22 14:32:07,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1979926.6666666667, ans=0.1
2023-11-22 14:32:20,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297000
2023-11-22 14:32:29,036 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.093e+01 8.742e+01 9.671e+01 1.399e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 14:32:31,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1979993.3333333333, ans=0.125
2023-11-22 14:32:42,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=1980060.0, ans=0.125
2023-11-22 14:32:47,271 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8450, loss[loss=0.07465, simple_loss=0.09746, pruned_loss=0.01624, audio_tagging_loss=0.009677, over 14202.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09279, pruned_loss=0.01464, audio_tagging_loss=0.009256, over 3045805.59 frames. ], batch size: 53, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:32:51,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1980126.6666666667, ans=0.2
2023-11-22 14:33:19,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=1980260.0, ans=0.2
2023-11-22 14:33:20,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1980260.0, ans=0.0
2023-11-22 14:33:25,295 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297050
2023-11-22 14:33:50,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1980460.0, ans=0.2
2023-11-22 14:33:52,046 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8500, loss[loss=0.06796, simple_loss=0.08692, pruned_loss=0.01373, audio_tagging_loss=0.01077, over 15457.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09298, pruned_loss=0.01467, audio_tagging_loss=0.009245, over 3045761.04 frames. ], batch size: 57, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:34:00,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=1980460.0, ans=0.07
2023-11-22 14:34:08,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1980526.6666666667, ans=0.1
2023-11-22 14:34:20,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.29 vs. limit=15.0
2023-11-22 14:34:29,171 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297100
2023-11-22 14:34:34,614 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.35 vs. limit=15.0
2023-11-22 14:34:36,241 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.259e+01 8.972e+01 9.645e+01 1.298e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 14:34:36,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=1980660.0, ans=15.0
2023-11-22 14:34:55,750 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8550, loss[loss=0.06281, simple_loss=0.07807, pruned_loss=0.01357, audio_tagging_loss=0.0102, over 16701.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.0928, pruned_loss=0.01462, audio_tagging_loss=0.009292, over 3047135.35 frames. ], batch size: 62, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:34:56,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=1980793.3333333333, ans=0.95
2023-11-22 14:35:00,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1980793.3333333333, ans=0.125
2023-11-22 14:35:33,180 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297150
2023-11-22 14:35:42,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1980993.3333333333, ans=0.125
2023-11-22 14:35:44,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1980993.3333333333, ans=0.125
2023-11-22 14:35:51,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1981060.0, ans=0.125
2023-11-22 14:35:51,722 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.650e-03
2023-11-22 14:35:59,866 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8600, loss[loss=0.0792, simple_loss=0.1022, pruned_loss=0.01847, audio_tagging_loss=0.009622, over 14887.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09212, pruned_loss=0.01459, audio_tagging_loss=0.009353, over 3044161.68 frames. ], batch size: 56, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:36:26,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1981260.0, ans=0.125
2023-11-22 14:36:32,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1981260.0, ans=0.2
2023-11-22 14:36:35,198 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.54 vs. limit=15.0
2023-11-22 14:36:37,045 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297200
2023-11-22 14:36:39,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=1981326.6666666667, ans=0.04949747468305833
2023-11-22 14:36:44,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1981326.6666666667, ans=0.125
2023-11-22 14:36:45,255 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.215e+01 8.334e+01 9.004e+01 9.545e+01 1.536e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 14:36:55,068 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:36:58,896 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.65 vs. limit=12.0
2023-11-22 14:36:59,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=1981393.3333333333, ans=0.5
2023-11-22 14:37:02,642 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=12.0
2023-11-22 14:37:03,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=1981460.0, ans=0.125
2023-11-22 14:37:03,621 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.72 vs. limit=22.5
2023-11-22 14:37:04,311 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8650, loss[loss=0.0746, simple_loss=0.09307, pruned_loss=0.01779, audio_tagging_loss=0.01027, over 15239.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09286, pruned_loss=0.0146, audio_tagging_loss=0.009407, over 3049131.08 frames. ], batch size: 58, lr: 2.72e-03, grad_scale: 16.0
2023-11-22 14:37:14,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1981460.0, ans=0.035
2023-11-22 14:37:41,397 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297250
2023-11-22 14:37:48,695 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.84 vs. limit=15.0
2023-11-22 14:37:53,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1981660.0, ans=0.0
2023-11-22 14:37:58,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1981726.6666666667, ans=0.2
2023-11-22 14:38:08,737 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8700, loss[loss=0.08019, simple_loss=0.09922, pruned_loss=0.02053, audio_tagging_loss=0.01005, over 15084.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09315, pruned_loss=0.01465, audio_tagging_loss=0.009471, over 3053424.43 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:38:12,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1981793.3333333333, ans=0.125
2023-11-22 14:38:20,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1981860.0, ans=0.2
2023-11-22 14:38:29,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1981860.0, ans=0.1
2023-11-22 14:38:39,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1981926.6666666667, ans=0.125
2023-11-22 14:38:43,106 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:38:45,893 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297300
2023-11-22 14:38:48,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.47 vs. limit=12.0
2023-11-22 14:38:53,652 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.337e+01 8.929e+01 9.655e+01 1.210e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 14:38:56,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=1981993.3333333333, ans=0.2
2023-11-22 14:39:12,695 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8750, loss[loss=0.07, simple_loss=0.09358, pruned_loss=0.01444, audio_tagging_loss=0.008769, over 14322.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09349, pruned_loss=0.01479, audio_tagging_loss=0.009451, over 3052632.52 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:39:17,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1982126.6666666667, ans=0.125
2023-11-22 14:39:35,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1982193.3333333333, ans=0.125
2023-11-22 14:39:37,326 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.51 vs. limit=15.0
2023-11-22 14:39:43,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1982260.0, ans=0.1
2023-11-22 14:39:49,732 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297350
2023-11-22 14:39:56,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1982326.6666666667, ans=0.035
2023-11-22 14:40:11,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=1982393.3333333333, ans=0.125
2023-11-22 14:40:15,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1982460.0, ans=0.125
2023-11-22 14:40:16,885 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8800, loss[loss=0.09546, simple_loss=0.1316, pruned_loss=0.02117, audio_tagging_loss=0.008501, over 15843.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09432, pruned_loss=0.01498, audio_tagging_loss=0.009423, over 3047878.33 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:40:43,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1982593.3333333333, ans=0.125
2023-11-22 14:40:44,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1982593.3333333333, ans=0.0
2023-11-22 14:40:47,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1982593.3333333333, ans=0.0
2023-11-22 14:40:48,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=1982593.3333333333, ans=0.2
2023-11-22 14:40:53,932 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297400
2023-11-22 14:40:57,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.32 vs. limit=22.5
2023-11-22 14:41:04,202 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.22 vs. limit=15.0
2023-11-22 14:41:04,594 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.228e+01 8.972e+01 9.857e+01 1.350e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 14:41:05,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=1982660.0, ans=0.07
2023-11-22 14:41:10,159 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.13 vs. limit=22.5
2023-11-22 14:41:16,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1982726.6666666667, ans=0.2
2023-11-22 14:41:21,481 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8850, loss[loss=0.07238, simple_loss=0.09696, pruned_loss=0.01488, audio_tagging_loss=0.009018, over 16084.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09376, pruned_loss=0.01477, audio_tagging_loss=0.009477, over 3055508.98 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:41:32,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1982860.0, ans=0.1
2023-11-22 14:41:33,597 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:41:44,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=1982860.0, ans=0.0
2023-11-22 14:41:51,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1982926.6666666667, ans=0.125
2023-11-22 14:41:51,217 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.00 vs. limit=22.5
2023-11-22 14:41:58,025 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297450
2023-11-22 14:42:10,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1982993.3333333333, ans=0.125
2023-11-22 14:42:20,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1983060.0, ans=0.1
2023-11-22 14:42:24,253 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8900, loss[loss=0.07984, simple_loss=0.1119, pruned_loss=0.01634, audio_tagging_loss=0.007574, over 14870.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09435, pruned_loss=0.01493, audio_tagging_loss=0.009323, over 3053230.20 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:42:30,430 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:42:42,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1983193.3333333333, ans=0.1
2023-11-22 14:42:58,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1983260.0, ans=0.1
2023-11-22 14:43:01,953 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297500
2023-11-22 14:43:07,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=1983326.6666666667, ans=0.0
2023-11-22 14:43:11,665 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.250e+01 8.702e+01 9.316e+01 1.378e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 14:43:13,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=1983326.6666666667, ans=0.125
2023-11-22 14:43:16,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1983393.3333333333, ans=0.0
2023-11-22 14:43:19,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=1983393.3333333333, ans=0.025
2023-11-22 14:43:19,355 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.88 vs. limit=15.0
2023-11-22 14:43:28,589 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 8950, loss[loss=0.0675, simple_loss=0.08636, pruned_loss=0.01512, audio_tagging_loss=0.009207, over 15935.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09396, pruned_loss=0.01482, audio_tagging_loss=0.009284, over 3057721.08 frames. ], batch size: 61, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:43:28,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1983460.0, ans=0.125
2023-11-22 14:44:00,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=1983593.3333333333, ans=0.125
2023-11-22 14:44:04,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297550
2023-11-22 14:44:18,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1983726.6666666667, ans=0.125
2023-11-22 14:44:25,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1983726.6666666667, ans=0.2
2023-11-22 14:44:31,346 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9000, loss[loss=0.05564, simple_loss=0.07524, pruned_loss=0.009868, audio_tagging_loss=0.008155, over 15134.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09432, pruned_loss=0.01487, audio_tagging_loss=0.009156, over 3048802.38 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:44:31,350 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 14:45:11,818 INFO [train_asr.py:1253] (0/4) Epoch 25, validation: loss=0.06003, simple_loss=0.05148, pruned_loss=0.00513, audio_tagging_loss=0.02916, over 4681554.00 frames. 
2023-11-22 14:45:11,819 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 14:45:29,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=1983860.0, ans=0.125
2023-11-22 14:45:37,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=1983926.6666666667, ans=0.0
2023-11-22 14:45:49,647 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297600
2023-11-22 14:45:55,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1983993.3333333333, ans=0.0
2023-11-22 14:45:56,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.25 vs. limit=15.0
2023-11-22 14:45:59,590 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.485e+01 9.145e+01 9.654e+01 1.276e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-22 14:46:16,688 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9050, loss[loss=0.05686, simple_loss=0.07224, pruned_loss=0.01311, audio_tagging_loss=0.007627, over 14363.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09386, pruned_loss=0.01472, audio_tagging_loss=0.009154, over 3047436.27 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:46:21,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1984126.6666666667, ans=0.035
2023-11-22 14:46:37,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1984193.3333333333, ans=0.0
2023-11-22 14:46:40,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=1984193.3333333333, ans=0.2
2023-11-22 14:46:48,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=1984260.0, ans=0.2
2023-11-22 14:46:53,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297650
2023-11-22 14:46:57,929 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.27 vs. limit=15.0
2023-11-22 14:47:05,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1984326.6666666667, ans=0.125
2023-11-22 14:47:05,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1984326.6666666667, ans=22.5
2023-11-22 14:47:21,060 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9100, loss[loss=0.0868, simple_loss=0.1167, pruned_loss=0.02226, audio_tagging_loss=0.006211, over 15714.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09311, pruned_loss=0.01465, audio_tagging_loss=0.009158, over 3044633.93 frames. ], batch size: 61, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:47:55,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1984593.3333333333, ans=0.0
2023-11-22 14:47:57,875 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297700
2023-11-22 14:47:59,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=1984660.0, ans=0.125
2023-11-22 14:48:07,996 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.687e+01 8.161e+01 8.752e+01 9.286e+01 2.425e+02, threshold=1.750e+02, percent-clipped=1.0
2023-11-22 14:48:19,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1984726.6666666667, ans=0.0
2023-11-22 14:48:24,400 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9150, loss[loss=0.07977, simple_loss=0.1085, pruned_loss=0.01948, audio_tagging_loss=0.006053, over 15749.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.0934, pruned_loss=0.01472, audio_tagging_loss=0.009162, over 3044408.39 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 8.0
2023-11-22 14:48:44,970 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.60 vs. limit=22.5
2023-11-22 14:48:58,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1984926.6666666667, ans=0.125
2023-11-22 14:49:02,451 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297750
2023-11-22 14:49:28,688 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9200, loss[loss=0.06201, simple_loss=0.08313, pruned_loss=0.01275, audio_tagging_loss=0.007697, over 14852.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09289, pruned_loss=0.01469, audio_tagging_loss=0.009166, over 3038641.54 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:49:32,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1985126.6666666667, ans=0.1
2023-11-22 14:49:39,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1985126.6666666667, ans=0.1
2023-11-22 14:49:39,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=1985126.6666666667, ans=0.125
2023-11-22 14:49:42,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1985193.3333333333, ans=0.1
2023-11-22 14:50:05,724 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297800
2023-11-22 14:50:16,265 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.283e+01 9.032e+01 9.826e+01 1.218e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-22 14:50:19,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=1985393.3333333333, ans=0.125
2023-11-22 14:50:20,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=1985393.3333333333, ans=0.09899494936611666
2023-11-22 14:50:26,254 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.72 vs. limit=15.0
2023-11-22 14:50:30,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=1985393.3333333333, ans=0.2
2023-11-22 14:50:33,433 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9250, loss[loss=0.05069, simple_loss=0.06361, pruned_loss=0.007017, audio_tagging_loss=0.01187, over 14223.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09325, pruned_loss=0.01485, audio_tagging_loss=0.009155, over 3040064.52 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:50:40,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=1985460.0, ans=0.2
2023-11-22 14:50:50,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=1985526.6666666667, ans=0.0
2023-11-22 14:50:55,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1985526.6666666667, ans=0.125
2023-11-22 14:51:05,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1985593.3333333333, ans=0.125
2023-11-22 14:51:10,213 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297850
2023-11-22 14:51:26,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1985726.6666666667, ans=0.125
2023-11-22 14:51:34,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=1985726.6666666667, ans=0.125
2023-11-22 14:51:37,719 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9300, loss[loss=0.07026, simple_loss=0.09366, pruned_loss=0.01495, audio_tagging_loss=0.008481, over 15213.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09347, pruned_loss=0.01495, audio_tagging_loss=0.009182, over 3040294.88 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:51:40,394 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 14:51:44,467 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.27 vs. limit=6.0
2023-11-22 14:51:52,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=1985860.0, ans=0.0
2023-11-22 14:51:56,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=1985860.0, ans=0.2
2023-11-22 14:52:12,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1985926.6666666667, ans=0.125
2023-11-22 14:52:14,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297900
2023-11-22 14:52:19,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1985993.3333333333, ans=0.125
2023-11-22 14:52:24,531 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.630e+01 8.158e+01 8.859e+01 9.535e+01 1.280e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 14:52:41,556 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9350, loss[loss=0.05722, simple_loss=0.07528, pruned_loss=0.009829, audio_tagging_loss=0.00975, over 15396.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09342, pruned_loss=0.01495, audio_tagging_loss=0.009276, over 3039615.46 frames. ], batch size: 61, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:52:41,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1986126.6666666667, ans=0.125
2023-11-22 14:52:42,496 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.23 vs. limit=6.0
2023-11-22 14:52:47,409 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.29 vs. limit=15.0
2023-11-22 14:52:50,023 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.88 vs. limit=15.0
2023-11-22 14:52:52,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1986126.6666666667, ans=0.125
2023-11-22 14:53:13,322 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.21 vs. limit=15.0
2023-11-22 14:53:20,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 297950
2023-11-22 14:53:37,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=1986393.3333333333, ans=0.95
2023-11-22 14:53:38,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1986393.3333333333, ans=0.125
2023-11-22 14:53:48,026 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9400, loss[loss=0.07387, simple_loss=0.09464, pruned_loss=0.01694, audio_tagging_loss=0.009608, over 14721.00 frames. ], tot_loss[loss=0.071, simple_loss=0.0935, pruned_loss=0.01493, audio_tagging_loss=0.009325, over 3051670.22 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:53:54,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=1986460.0, ans=0.125
2023-11-22 14:54:04,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1986526.6666666667, ans=0.0
2023-11-22 14:54:20,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1986593.3333333333, ans=0.0
2023-11-22 14:54:24,938 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298000
2023-11-22 14:54:35,748 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.116e+01 8.691e+01 9.663e+01 1.322e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-22 14:54:52,078 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 14:54:53,302 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9450, loss[loss=0.08041, simple_loss=0.1103, pruned_loss=0.01769, audio_tagging_loss=0.007545, over 15920.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09315, pruned_loss=0.01495, audio_tagging_loss=0.009423, over 3049431.24 frames. ], batch size: 60, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:54:58,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=1986793.3333333333, ans=0.0
2023-11-22 14:55:02,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1986793.3333333333, ans=0.1
2023-11-22 14:55:24,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1986926.6666666667, ans=0.125
2023-11-22 14:55:29,064 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.30 vs. limit=15.0
2023-11-22 14:55:31,004 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298050
2023-11-22 14:55:57,825 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9500, loss[loss=0.07332, simple_loss=0.1074, pruned_loss=0.01458, audio_tagging_loss=0.005052, over 14850.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09305, pruned_loss=0.01493, audio_tagging_loss=0.009396, over 3053203.47 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:56:17,129 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.06 vs. limit=15.0
2023-11-22 14:56:18,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1987193.3333333333, ans=0.0
2023-11-22 14:56:24,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1987260.0, ans=0.125
2023-11-22 14:56:33,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=1987260.0, ans=0.125
2023-11-22 14:56:35,403 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298100
2023-11-22 14:56:39,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1987326.6666666667, ans=0.125
2023-11-22 14:56:45,226 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.353e+01 9.126e+01 1.009e+02 1.546e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-22 14:56:50,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=1987393.3333333333, ans=0.2
2023-11-22 14:57:02,719 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9550, loss[loss=0.08092, simple_loss=0.1066, pruned_loss=0.01843, audio_tagging_loss=0.009186, over 14449.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.094, pruned_loss=0.01516, audio_tagging_loss=0.009401, over 3049131.92 frames. ], batch size: 53, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 14:57:14,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1987460.0, ans=0.05
2023-11-22 14:57:19,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1987526.6666666667, ans=0.125
2023-11-22 14:57:21,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1987526.6666666667, ans=0.0
2023-11-22 14:57:34,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1987593.3333333333, ans=0.1
2023-11-22 14:57:41,054 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298150
2023-11-22 14:57:46,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=1987660.0, ans=0.0
2023-11-22 14:58:09,005 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9600, loss[loss=0.06221, simple_loss=0.08069, pruned_loss=0.008721, audio_tagging_loss=0.01314, over 16045.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.0932, pruned_loss=0.01491, audio_tagging_loss=0.009557, over 3042822.07 frames. ], batch size: 60, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 14:58:14,605 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.28 vs. limit=12.0
2023-11-22 14:58:18,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1987793.3333333333, ans=0.125
2023-11-22 14:58:23,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1987860.0, ans=0.1
2023-11-22 14:58:25,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1987860.0, ans=0.0
2023-11-22 14:58:32,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1987860.0, ans=0.125
2023-11-22 14:58:42,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1987926.6666666667, ans=0.125
2023-11-22 14:58:46,300 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298200
2023-11-22 14:58:46,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1987993.3333333333, ans=0.0
2023-11-22 14:58:57,434 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.336e+01 8.183e+01 8.755e+01 9.694e+01 2.318e+02, threshold=1.751e+02, percent-clipped=1.0
2023-11-22 14:59:13,503 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9650, loss[loss=0.07659, simple_loss=0.1008, pruned_loss=0.01548, audio_tagging_loss=0.01072, over 14354.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09329, pruned_loss=0.01472, audio_tagging_loss=0.00954, over 3044882.10 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 14:59:46,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1988260.0, ans=0.125
2023-11-22 14:59:49,575 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.10 vs. limit=12.0
2023-11-22 14:59:50,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1988260.0, ans=0.125
2023-11-22 14:59:51,335 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298250
2023-11-22 15:00:03,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1988393.3333333333, ans=0.125
2023-11-22 15:00:06,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1988393.3333333333, ans=0.125
2023-11-22 15:00:12,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1988393.3333333333, ans=0.125
2023-11-22 15:00:17,271 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9700, loss[loss=0.0516, simple_loss=0.07007, pruned_loss=0.00977, audio_tagging_loss=0.006794, over 14371.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09252, pruned_loss=0.0146, audio_tagging_loss=0.009365, over 3041637.02 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:00:33,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=1988526.6666666667, ans=0.0
2023-11-22 15:00:49,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.90 vs. limit=15.0
2023-11-22 15:00:52,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1988593.3333333333, ans=0.125
2023-11-22 15:00:55,303 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298300
2023-11-22 15:00:56,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=1988660.0, ans=0.2
2023-11-22 15:00:57,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1988660.0, ans=0.0
2023-11-22 15:01:03,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=1988660.0, ans=10.0
2023-11-22 15:01:04,939 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.401e+01 8.285e+01 8.809e+01 9.678e+01 1.163e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 15:01:17,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=1988726.6666666667, ans=0.125
2023-11-22 15:01:21,865 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9750, loss[loss=0.06597, simple_loss=0.0872, pruned_loss=0.01377, audio_tagging_loss=0.008605, over 15734.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09247, pruned_loss=0.0146, audio_tagging_loss=0.009268, over 3043007.58 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:01:28,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.65 vs. limit=15.0
2023-11-22 15:01:37,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=1988860.0, ans=0.125
2023-11-22 15:01:40,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1988860.0, ans=0.0
2023-11-22 15:01:46,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=1988926.6666666667, ans=0.125
2023-11-22 15:01:58,990 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298350
2023-11-22 15:01:59,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=1988993.3333333333, ans=0.0
2023-11-22 15:02:04,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1988993.3333333333, ans=0.07
2023-11-22 15:02:24,976 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.94 vs. limit=10.0
2023-11-22 15:02:25,385 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9800, loss[loss=0.05667, simple_loss=0.06886, pruned_loss=0.0115, audio_tagging_loss=0.01074, over 14333.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09354, pruned_loss=0.01483, audio_tagging_loss=0.009163, over 3053536.63 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:02:26,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1989126.6666666667, ans=0.0
2023-11-22 15:02:33,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1989126.6666666667, ans=0.0
2023-11-22 15:02:37,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=1989193.3333333333, ans=0.0
2023-11-22 15:02:43,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1989193.3333333333, ans=0.0
2023-11-22 15:02:43,728 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.53 vs. limit=12.0
2023-11-22 15:03:02,461 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298400
2023-11-22 15:03:14,262 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.717e+01 7.942e+01 8.704e+01 9.561e+01 1.255e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 15:03:14,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=1989326.6666666667, ans=0.0
2023-11-22 15:03:23,523 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:03:25,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1989393.3333333333, ans=0.125
2023-11-22 15:03:29,598 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9850, loss[loss=0.06869, simple_loss=0.07749, pruned_loss=0.01555, audio_tagging_loss=0.01439, over 14551.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09315, pruned_loss=0.01469, audio_tagging_loss=0.009152, over 3053225.33 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:03:47,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=1989526.6666666667, ans=0.1
2023-11-22 15:03:50,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1989526.6666666667, ans=0.125
2023-11-22 15:04:03,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=1989593.3333333333, ans=0.0
2023-11-22 15:04:03,294 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:04:06,707 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298450
2023-11-22 15:04:23,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1989726.6666666667, ans=0.125
2023-11-22 15:04:32,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1989793.3333333333, ans=0.125
2023-11-22 15:04:33,848 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9900, loss[loss=0.07452, simple_loss=0.09452, pruned_loss=0.018, audio_tagging_loss=0.009252, over 16625.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09408, pruned_loss=0.01487, audio_tagging_loss=0.00913, over 3054033.77 frames. ], batch size: 63, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:04:38,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1989793.3333333333, ans=0.0
2023-11-22 15:05:10,759 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298500
2023-11-22 15:05:12,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1989993.3333333333, ans=0.125
2023-11-22 15:05:23,017 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.052e+01 8.333e+01 9.011e+01 9.582e+01 1.423e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-22 15:05:30,953 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.97 vs. limit=10.0
2023-11-22 15:05:35,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=1990060.0, ans=0.05
2023-11-22 15:05:37,745 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 9950, loss[loss=0.08173, simple_loss=0.09787, pruned_loss=0.02225, audio_tagging_loss=0.01055, over 15310.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09404, pruned_loss=0.01475, audio_tagging_loss=0.00922, over 3062985.43 frames. ], batch size: 59, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:05:41,533 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.32 vs. limit=15.0
2023-11-22 15:05:53,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1990193.3333333333, ans=0.0
2023-11-22 15:06:15,361 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298550
2023-11-22 15:06:42,074 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10000, loss[loss=0.05617, simple_loss=0.08103, pruned_loss=0.00748, audio_tagging_loss=0.008174, over 16120.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09331, pruned_loss=0.01455, audio_tagging_loss=0.009195, over 3060221.80 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:06:47,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.10 vs. limit=15.0
2023-11-22 15:06:50,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1990460.0, ans=0.0
2023-11-22 15:07:03,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.84 vs. limit=12.0
2023-11-22 15:07:13,402 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.15 vs. limit=15.0
2023-11-22 15:07:19,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298600
2023-11-22 15:07:31,286 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.143e+01 8.830e+01 9.648e+01 3.146e+02, threshold=1.766e+02, percent-clipped=1.0
2023-11-22 15:07:35,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1990726.6666666667, ans=0.0
2023-11-22 15:07:45,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=1990793.3333333333, ans=0.125
2023-11-22 15:07:47,388 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10050, loss[loss=0.06919, simple_loss=0.08565, pruned_loss=0.01639, audio_tagging_loss=0.009971, over 15419.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09331, pruned_loss=0.01459, audio_tagging_loss=0.009178, over 3057806.48 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:07:58,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=1990860.0, ans=0.0
2023-11-22 15:08:09,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1990860.0, ans=0.0
2023-11-22 15:08:11,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=1990926.6666666667, ans=0.125
2023-11-22 15:08:14,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1990926.6666666667, ans=0.1
2023-11-22 15:08:15,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1990926.6666666667, ans=0.0
2023-11-22 15:08:24,278 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298650
2023-11-22 15:08:51,074 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10100, loss[loss=0.04911, simple_loss=0.06587, pruned_loss=0.007668, audio_tagging_loss=0.008505, over 15321.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09375, pruned_loss=0.01475, audio_tagging_loss=0.009222, over 3057026.05 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:08:57,851 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=15.0
2023-11-22 15:09:28,268 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298700
2023-11-22 15:09:39,081 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.326e+01 8.430e+01 8.908e+01 9.951e+01 1.277e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 15:09:39,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1991326.6666666667, ans=0.1
2023-11-22 15:09:41,547 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:09:42,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.07 vs. limit=15.0
2023-11-22 15:09:54,829 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10150, loss[loss=0.0895, simple_loss=0.1163, pruned_loss=0.01744, audio_tagging_loss=0.01389, over 16244.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09384, pruned_loss=0.01483, audio_tagging_loss=0.009307, over 3063282.49 frames. ], batch size: 61, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:10:04,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.01 vs. limit=22.5
2023-11-22 15:10:24,430 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:10:31,749 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298750
2023-11-22 15:10:36,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=1991660.0, ans=0.1
2023-11-22 15:10:58,822 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10200, loss[loss=0.05469, simple_loss=0.06803, pruned_loss=0.008793, audio_tagging_loss=0.01188, over 14274.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09362, pruned_loss=0.01481, audio_tagging_loss=0.009343, over 3064753.95 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:10:59,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=1991793.3333333333, ans=0.2
2023-11-22 15:11:02,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1991793.3333333333, ans=0.125
2023-11-22 15:11:21,512 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:11:36,001 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298800
2023-11-22 15:11:48,671 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.000e+01 8.315e+01 8.823e+01 9.660e+01 1.198e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 15:11:49,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=1992060.0, ans=0.95
2023-11-22 15:12:02,807 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10250, loss[loss=0.05402, simple_loss=0.06914, pruned_loss=0.008597, audio_tagging_loss=0.01085, over 15955.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.0936, pruned_loss=0.01481, audio_tagging_loss=0.00938, over 3063320.29 frames. ], batch size: 62, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:12:33,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=1992260.0, ans=0.0
2023-11-22 15:12:40,234 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298850
2023-11-22 15:12:41,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=1992326.6666666667, ans=0.0
2023-11-22 15:12:41,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1992326.6666666667, ans=0.125
2023-11-22 15:13:06,229 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10300, loss[loss=0.05559, simple_loss=0.07508, pruned_loss=0.006313, audio_tagging_loss=0.01174, over 16482.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.0929, pruned_loss=0.01451, audio_tagging_loss=0.009538, over 3062023.22 frames. ], batch size: 63, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:13:12,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1992460.0, ans=0.125
2023-11-22 15:13:18,558 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.40 vs. limit=15.0
2023-11-22 15:13:31,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1992593.3333333333, ans=0.1
2023-11-22 15:13:43,236 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298900
2023-11-22 15:13:46,146 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.74 vs. limit=22.5
2023-11-22 15:13:55,926 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.205e+01 8.813e+01 9.475e+01 1.270e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 15:13:56,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1992726.6666666667, ans=0.0
2023-11-22 15:14:10,573 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10350, loss[loss=0.08374, simple_loss=0.1121, pruned_loss=0.01893, audio_tagging_loss=0.008737, over 15896.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09379, pruned_loss=0.0147, audio_tagging_loss=0.009483, over 3061107.93 frames. ], batch size: 59, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:14:26,357 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.21 vs. limit=22.5
2023-11-22 15:14:41,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1992926.6666666667, ans=0.1
2023-11-22 15:14:44,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1992926.6666666667, ans=0.125
2023-11-22 15:14:47,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 298950
2023-11-22 15:14:47,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1992993.3333333333, ans=0.1
2023-11-22 15:15:14,454 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10400, loss[loss=0.07999, simple_loss=0.1085, pruned_loss=0.01658, audio_tagging_loss=0.009138, over 16148.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.09428, pruned_loss=0.01481, audio_tagging_loss=0.009453, over 3064276.11 frames. ], batch size: 58, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:15:22,234 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:15:32,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=1993193.3333333333, ans=0.125
2023-11-22 15:15:39,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1993260.0, ans=0.125
2023-11-22 15:15:41,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=1993260.0, ans=0.0
2023-11-22 15:15:46,929 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:15:51,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299000
2023-11-22 15:15:58,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=1993326.6666666667, ans=0.1
2023-11-22 15:16:02,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=1993326.6666666667, ans=0.125
2023-11-22 15:16:03,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1993326.6666666667, ans=0.125
2023-11-22 15:16:05,355 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.066e+01 8.819e+01 9.372e+01 1.193e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 15:16:06,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1993393.3333333333, ans=0.1
2023-11-22 15:16:17,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1993460.0, ans=0.125
2023-11-22 15:16:18,124 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10450, loss[loss=0.06679, simple_loss=0.08982, pruned_loss=0.01371, audio_tagging_loss=0.008169, over 14741.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09427, pruned_loss=0.01484, audio_tagging_loss=0.009473, over 3061813.91 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:16:19,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1993460.0, ans=0.07
2023-11-22 15:16:55,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299050
2023-11-22 15:17:03,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=1993660.0, ans=22.5
2023-11-22 15:17:04,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.75 vs. limit=15.0
2023-11-22 15:17:12,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1993726.6666666667, ans=0.1
2023-11-22 15:17:21,643 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10500, loss[loss=0.05858, simple_loss=0.07342, pruned_loss=0.01043, audio_tagging_loss=0.01144, over 14501.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09366, pruned_loss=0.0148, audio_tagging_loss=0.009338, over 3054319.53 frames. ], batch size: 56, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:17:26,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=1993793.3333333333, ans=0.2
2023-11-22 15:17:42,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1993860.0, ans=0.125
2023-11-22 15:17:47,923 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.15 vs. limit=15.0
2023-11-22 15:17:48,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1993926.6666666667, ans=0.1
2023-11-22 15:17:58,757 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299100
2023-11-22 15:18:13,217 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.803e+01 8.251e+01 8.969e+01 9.494e+01 1.165e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 15:18:25,934 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10550, loss[loss=0.07827, simple_loss=0.1033, pruned_loss=0.01588, audio_tagging_loss=0.01074, over 15647.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09345, pruned_loss=0.01492, audio_tagging_loss=0.009212, over 3047097.34 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:19:03,317 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299150
2023-11-22 15:19:07,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1994326.6666666667, ans=0.0
2023-11-22 15:19:07,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1994326.6666666667, ans=0.2
2023-11-22 15:19:08,125 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.16 vs. limit=10.0
2023-11-22 15:19:12,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1994326.6666666667, ans=0.0
2023-11-22 15:19:17,703 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.52 vs. limit=6.0
2023-11-22 15:19:29,001 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10600, loss[loss=0.0623, simple_loss=0.08418, pruned_loss=0.01205, audio_tagging_loss=0.008157, over 14566.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09327, pruned_loss=0.01494, audio_tagging_loss=0.009162, over 3039950.37 frames. ], batch size: 53, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:19:33,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=1994460.0, ans=0.025
2023-11-22 15:19:53,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=1994593.3333333333, ans=0.2
2023-11-22 15:19:58,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=1994593.3333333333, ans=0.0
2023-11-22 15:20:06,510 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299200
2023-11-22 15:20:08,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1994660.0, ans=0.125
2023-11-22 15:20:11,199 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.42 vs. limit=15.0
2023-11-22 15:20:14,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=1994660.0, ans=0.05
2023-11-22 15:20:20,908 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.207e+01 8.203e+01 8.764e+01 9.272e+01 1.300e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 15:20:25,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=1994726.6666666667, ans=0.07
2023-11-22 15:20:30,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=1994726.6666666667, ans=0.035
2023-11-22 15:20:32,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=1994793.3333333333, ans=0.125
2023-11-22 15:20:32,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1994793.3333333333, ans=0.1
2023-11-22 15:20:32,957 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10650, loss[loss=0.05963, simple_loss=0.0754, pruned_loss=0.011, audio_tagging_loss=0.01093, over 14350.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09333, pruned_loss=0.01492, audio_tagging_loss=0.009157, over 3038558.90 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:20:41,759 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:20:47,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=1994860.0, ans=0.125
2023-11-22 15:21:00,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=1994926.6666666667, ans=0.07
2023-11-22 15:21:10,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299250
2023-11-22 15:21:37,637 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10700, loss[loss=0.07473, simple_loss=0.09791, pruned_loss=0.0163, audio_tagging_loss=0.009471, over 15352.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09411, pruned_loss=0.01486, audio_tagging_loss=0.009101, over 3041428.50 frames. ], batch size: 57, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:21:42,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1995126.6666666667, ans=0.125
2023-11-22 15:22:11,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1995260.0, ans=0.1
2023-11-22 15:22:14,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299300
2023-11-22 15:22:28,550 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.250e+01 8.791e+01 9.452e+01 1.180e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 15:22:30,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=1995393.3333333333, ans=0.0
2023-11-22 15:22:32,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1995393.3333333333, ans=0.09899494936611666
2023-11-22 15:22:40,723 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10750, loss[loss=0.05598, simple_loss=0.0691, pruned_loss=0.01132, audio_tagging_loss=0.01011, over 14121.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09473, pruned_loss=0.01486, audio_tagging_loss=0.009039, over 3050567.91 frames. ], batch size: 55, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:22:49,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=1995460.0, ans=0.125
2023-11-22 15:22:49,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=1995460.0, ans=0.125
2023-11-22 15:23:17,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1995593.3333333333, ans=0.0
2023-11-22 15:23:18,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299350
2023-11-22 15:23:24,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1995660.0, ans=0.09899494936611666
2023-11-22 15:23:29,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=1995660.0, ans=0.0
2023-11-22 15:23:39,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1995726.6666666667, ans=0.125
2023-11-22 15:23:44,102 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10800, loss[loss=0.0899, simple_loss=0.1134, pruned_loss=0.02348, audio_tagging_loss=0.009731, over 14243.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09522, pruned_loss=0.01495, audio_tagging_loss=0.00905, over 3058679.03 frames. ], batch size: 53, lr: 2.71e-03, grad_scale: 32.0
2023-11-22 15:23:49,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=1995793.3333333333, ans=0.04949747468305833
2023-11-22 15:23:57,831 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.34 vs. limit=15.0
2023-11-22 15:24:03,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1995860.0, ans=0.1
2023-11-22 15:24:08,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=1995860.0, ans=10.0
2023-11-22 15:24:21,582 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299400
2023-11-22 15:24:31,669 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.97 vs. limit=10.0
2023-11-22 15:24:36,975 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.569e+01 8.241e+01 8.735e+01 9.368e+01 1.325e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 15:24:38,642 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:24:49,290 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10850, loss[loss=0.07625, simple_loss=0.1035, pruned_loss=0.01152, audio_tagging_loss=0.01299, over 15009.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09455, pruned_loss=0.01487, audio_tagging_loss=0.009185, over 3056908.78 frames. ], batch size: 54, lr: 2.71e-03, grad_scale: 16.0
2023-11-22 15:24:49,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1996126.6666666667, ans=0.125
2023-11-22 15:25:18,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1996260.0, ans=0.125
2023-11-22 15:25:23,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=1996260.0, ans=0.125
2023-11-22 15:25:25,950 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299450
2023-11-22 15:25:40,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=1996393.3333333333, ans=0.0
2023-11-22 15:25:49,807 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:25:53,382 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10900, loss[loss=0.0625, simple_loss=0.08201, pruned_loss=0.0114, audio_tagging_loss=0.01009, over 14082.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09404, pruned_loss=0.01476, audio_tagging_loss=0.00933, over 3056229.53 frames. ], batch size: 54, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:26:02,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1996460.0, ans=0.125
2023-11-22 15:26:05,679 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.28 vs. limit=15.0
2023-11-22 15:26:10,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.83 vs. limit=10.0
2023-11-22 15:26:23,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1996593.3333333333, ans=0.125
2023-11-22 15:26:30,800 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299500
2023-11-22 15:26:45,941 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.150e+01 8.424e+01 9.041e+01 9.724e+01 1.556e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-22 15:26:55,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=1996726.6666666667, ans=0.0
2023-11-22 15:26:57,242 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 10950, loss[loss=0.08653, simple_loss=0.1182, pruned_loss=0.02038, audio_tagging_loss=0.007038, over 15704.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09433, pruned_loss=0.01488, audio_tagging_loss=0.009248, over 3057599.13 frames. ], batch size: 57, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:27:04,756 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:27:22,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=1996926.6666666667, ans=0.0
2023-11-22 15:27:25,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=1996926.6666666667, ans=0.0
2023-11-22 15:27:28,423 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:27:34,181 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299550
2023-11-22 15:28:01,251 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11000, loss[loss=0.05742, simple_loss=0.07289, pruned_loss=0.01084, audio_tagging_loss=0.01013, over 15663.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09466, pruned_loss=0.01502, audio_tagging_loss=0.009223, over 3051431.54 frames. ], batch size: 60, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:28:02,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=1997126.6666666667, ans=0.125
2023-11-22 15:28:08,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1997126.6666666667, ans=0.1
2023-11-22 15:28:11,082 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:28:12,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=1997193.3333333333, ans=0.0
2023-11-22 15:28:25,041 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:28:30,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=1997260.0, ans=0.2
2023-11-22 15:28:37,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.87 vs. limit=15.0
2023-11-22 15:28:37,965 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299600
2023-11-22 15:28:43,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=1997326.6666666667, ans=0.125
2023-11-22 15:28:45,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=1997326.6666666667, ans=0.125
2023-11-22 15:28:53,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1997393.3333333333, ans=0.125
2023-11-22 15:28:53,990 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.275e+01 8.821e+01 9.536e+01 1.341e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 15:29:05,392 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11050, loss[loss=0.05741, simple_loss=0.07429, pruned_loss=0.01034, audio_tagging_loss=0.009931, over 15607.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09403, pruned_loss=0.01499, audio_tagging_loss=0.009289, over 3057084.63 frames. ], batch size: 59, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:29:23,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1997526.6666666667, ans=0.1
2023-11-22 15:29:39,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=1997593.3333333333, ans=0.125
2023-11-22 15:29:42,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299650
2023-11-22 15:29:43,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1997660.0, ans=0.125
2023-11-22 15:29:56,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1997726.6666666667, ans=0.0
2023-11-22 15:30:03,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.35 vs. limit=15.0
2023-11-22 15:30:08,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=1997793.3333333333, ans=0.0
2023-11-22 15:30:08,995 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11100, loss[loss=0.05824, simple_loss=0.07308, pruned_loss=0.009591, audio_tagging_loss=0.01211, over 14627.00 frames. ], tot_loss[loss=0.07119, simple_loss=0.09367, pruned_loss=0.01485, audio_tagging_loss=0.009512, over 3058237.72 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:30:09,606 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.04 vs. limit=15.0
2023-11-22 15:30:36,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=1997926.6666666667, ans=0.0
2023-11-22 15:30:40,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=1997926.6666666667, ans=0.125
2023-11-22 15:30:40,784 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:30:45,984 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299700
2023-11-22 15:30:51,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=1997993.3333333333, ans=0.09899494936611666
2023-11-22 15:31:01,209 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.427e+01 8.934e+01 9.666e+01 1.581e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 15:31:11,104 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.15 vs. limit=6.0
2023-11-22 15:31:12,947 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11150, loss[loss=0.07503, simple_loss=0.09569, pruned_loss=0.01637, audio_tagging_loss=0.01082, over 15535.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09458, pruned_loss=0.01498, audio_tagging_loss=0.009529, over 3058516.86 frames. ], batch size: 57, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:31:18,052 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.55 vs. limit=15.0
2023-11-22 15:31:37,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=1998260.0, ans=0.125
2023-11-22 15:31:44,385 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0
2023-11-22 15:31:50,005 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299750
2023-11-22 15:32:07,732 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.40 vs. limit=10.0
2023-11-22 15:32:15,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=1998460.0, ans=0.125
2023-11-22 15:32:16,622 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11200, loss[loss=0.07015, simple_loss=0.09729, pruned_loss=0.01185, audio_tagging_loss=0.009653, over 15698.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09456, pruned_loss=0.01504, audio_tagging_loss=0.009607, over 3059314.33 frames. ], batch size: 59, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:32:16,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1998460.0, ans=0.2
2023-11-22 15:32:16,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=1998460.0, ans=0.2
2023-11-22 15:32:41,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=1998593.3333333333, ans=0.125
2023-11-22 15:32:46,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1998593.3333333333, ans=0.1
2023-11-22 15:32:54,727 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299800
2023-11-22 15:32:54,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1998660.0, ans=0.125
2023-11-22 15:32:57,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=1998660.0, ans=0.125
2023-11-22 15:33:09,679 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.950e+01 8.174e+01 8.782e+01 9.719e+01 1.179e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 15:33:22,111 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11250, loss[loss=0.06127, simple_loss=0.08197, pruned_loss=0.011, audio_tagging_loss=0.009284, over 16252.00 frames. ], tot_loss[loss=0.07149, simple_loss=0.09363, pruned_loss=0.01505, audio_tagging_loss=0.009624, over 3056339.71 frames. ], batch size: 60, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:33:37,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=1998860.0, ans=0.0
2023-11-22 15:33:54,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=1998926.6666666667, ans=0.09899494936611666
2023-11-22 15:33:58,953 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299850
2023-11-22 15:34:02,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=1998993.3333333333, ans=0.125
2023-11-22 15:34:17,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=1999060.0, ans=0.125
2023-11-22 15:34:19,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1999060.0, ans=0.125
2023-11-22 15:34:26,198 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11300, loss[loss=0.08502, simple_loss=0.1091, pruned_loss=0.01972, audio_tagging_loss=0.01074, over 15423.00 frames. ], tot_loss[loss=0.07169, simple_loss=0.09409, pruned_loss=0.01508, audio_tagging_loss=0.00957, over 3051886.02 frames. ], batch size: 55, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:34:33,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1999126.6666666667, ans=0.1
2023-11-22 15:34:52,091 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.83 vs. limit=10.0
2023-11-22 15:35:03,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299900
2023-11-22 15:35:07,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1999326.6666666667, ans=0.1
2023-11-22 15:35:18,553 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.156e+01 8.491e+01 9.004e+01 9.715e+01 1.251e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 15:35:29,977 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11350, loss[loss=0.08659, simple_loss=0.1281, pruned_loss=0.0168, audio_tagging_loss=0.005741, over 16686.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09367, pruned_loss=0.0148, audio_tagging_loss=0.00943, over 3054479.13 frames. ], batch size: 62, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:35:30,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1999460.0, ans=0.125
2023-11-22 15:35:55,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1999593.3333333333, ans=0.125
2023-11-22 15:36:04,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=1999593.3333333333, ans=0.035
2023-11-22 15:36:07,158 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 299950
2023-11-22 15:36:16,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=1999660.0, ans=0.5
2023-11-22 15:36:33,736 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11400, loss[loss=0.06959, simple_loss=0.09442, pruned_loss=0.01193, audio_tagging_loss=0.01045, over 15419.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09331, pruned_loss=0.01469, audio_tagging_loss=0.00925, over 3056148.38 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:36:38,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=1999793.3333333333, ans=0.125
2023-11-22 15:36:40,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=1999793.3333333333, ans=0.0
2023-11-22 15:36:54,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=1999860.0, ans=0.0
2023-11-22 15:37:10,946 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300000
2023-11-22 15:37:12,401 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-300000.pt
2023-11-22 15:37:20,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=1999993.3333333333, ans=0.2
2023-11-22 15:37:24,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.00 vs. limit=15.0
2023-11-22 15:37:31,203 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.071e+01 8.858e+01 9.602e+01 1.299e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 15:37:37,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2000060.0, ans=0.0
2023-11-22 15:37:41,007 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11450, loss[loss=0.05986, simple_loss=0.08064, pruned_loss=0.0108, audio_tagging_loss=0.008732, over 14991.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09313, pruned_loss=0.01475, audio_tagging_loss=0.0091, over 3051809.83 frames. ], batch size: 56, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:37:41,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2000126.6666666667, ans=0.2
2023-11-22 15:37:57,210 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.34 vs. limit=22.5
2023-11-22 15:38:18,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300050
2023-11-22 15:38:19,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2000326.6666666667, ans=0.0
2023-11-22 15:38:32,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2000393.3333333333, ans=0.2
2023-11-22 15:38:36,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2000393.3333333333, ans=0.125
2023-11-22 15:38:44,819 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11500, loss[loss=0.04877, simple_loss=0.06274, pruned_loss=0.006565, audio_tagging_loss=0.01084, over 15064.00 frames. ], tot_loss[loss=0.07133, simple_loss=0.09437, pruned_loss=0.01506, audio_tagging_loss=0.009085, over 3055819.78 frames. ], batch size: 58, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:39:09,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2000593.3333333333, ans=0.2
2023-11-22 15:39:22,046 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300100
2023-11-22 15:39:33,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.68 vs. limit=15.0
2023-11-22 15:39:38,052 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.223e+01 8.870e+01 9.437e+01 1.249e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 15:39:43,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2000726.6666666667, ans=0.1
2023-11-22 15:39:47,760 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11550, loss[loss=0.08217, simple_loss=0.107, pruned_loss=0.01728, audio_tagging_loss=0.0114, over 14211.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09414, pruned_loss=0.01484, audio_tagging_loss=0.0091, over 3053157.17 frames. ], batch size: 52, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:39:51,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2000793.3333333333, ans=0.2
2023-11-22 15:40:12,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2000926.6666666667, ans=0.2
2023-11-22 15:40:13,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.40 vs. limit=15.0
2023-11-22 15:40:23,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2000926.6666666667, ans=0.125
2023-11-22 15:40:25,319 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300150
2023-11-22 15:40:26,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2000993.3333333333, ans=0.125
2023-11-22 15:40:27,700 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 15:40:30,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2000993.3333333333, ans=0.09899494936611666
2023-11-22 15:40:52,022 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11600, loss[loss=0.07346, simple_loss=0.09612, pruned_loss=0.0158, audio_tagging_loss=0.009594, over 15421.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09418, pruned_loss=0.01494, audio_tagging_loss=0.009179, over 3053117.80 frames. ], batch size: 58, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:41:19,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2001260.0, ans=0.0
2023-11-22 15:41:29,217 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300200
2023-11-22 15:41:30,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2001326.6666666667, ans=0.1
2023-11-22 15:41:33,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2001326.6666666667, ans=0.125
2023-11-22 15:41:41,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2001326.6666666667, ans=0.125
2023-11-22 15:41:46,179 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.408e+01 8.800e+01 9.381e+01 1.504e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 15:41:47,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2001393.3333333333, ans=0.125
2023-11-22 15:41:56,488 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11650, loss[loss=0.074, simple_loss=0.09754, pruned_loss=0.01572, audio_tagging_loss=0.009514, over 15498.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.0939, pruned_loss=0.01494, audio_tagging_loss=0.009261, over 3048515.89 frames. ], batch size: 57, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:42:02,035 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.59 vs. limit=6.0
2023-11-22 15:42:09,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2001526.6666666667, ans=0.1
2023-11-22 15:42:17,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2001526.6666666667, ans=0.2
2023-11-22 15:42:33,537 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300250
2023-11-22 15:42:38,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2001660.0, ans=0.125
2023-11-22 15:42:51,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2001726.6666666667, ans=0.0
2023-11-22 15:42:59,654 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11700, loss[loss=0.06314, simple_loss=0.07483, pruned_loss=0.01759, audio_tagging_loss=0.008139, over 14595.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09371, pruned_loss=0.01486, audio_tagging_loss=0.009246, over 3045984.43 frames. ], batch size: 57, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:43:32,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2001926.6666666667, ans=0.125
2023-11-22 15:43:37,218 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300300
2023-11-22 15:43:39,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2001993.3333333333, ans=0.1
2023-11-22 15:43:39,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2001993.3333333333, ans=0.0
2023-11-22 15:43:53,162 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.806e+01 8.322e+01 8.823e+01 9.489e+01 1.251e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 15:44:03,420 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11750, loss[loss=0.07044, simple_loss=0.09304, pruned_loss=0.01399, audio_tagging_loss=0.009932, over 15610.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.09293, pruned_loss=0.01493, audio_tagging_loss=0.009442, over 3048162.53 frames. ], batch size: 58, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:44:04,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2002126.6666666667, ans=0.2
2023-11-22 15:44:15,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2002193.3333333333, ans=0.125
2023-11-22 15:44:36,819 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.80 vs. limit=15.0
2023-11-22 15:44:39,735 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300350
2023-11-22 15:44:42,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2002326.6666666667, ans=0.125
2023-11-22 15:44:44,714 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.68 vs. limit=10.0
2023-11-22 15:44:45,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2002326.6666666667, ans=0.0
2023-11-22 15:45:07,050 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11800, loss[loss=0.07364, simple_loss=0.1044, pruned_loss=0.01492, audio_tagging_loss=0.006501, over 14571.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09284, pruned_loss=0.01507, audio_tagging_loss=0.009478, over 3044638.39 frames. ], batch size: 54, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:45:12,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2002460.0, ans=0.0
2023-11-22 15:45:17,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2002460.0, ans=0.2
2023-11-22 15:45:43,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300400
2023-11-22 15:46:01,592 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.204e+01 8.722e+01 9.366e+01 1.185e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-22 15:46:09,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2002793.3333333333, ans=0.125
2023-11-22 15:46:10,190 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11850, loss[loss=0.05619, simple_loss=0.07003, pruned_loss=0.00952, audio_tagging_loss=0.01165, over 15950.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.0936, pruned_loss=0.01526, audio_tagging_loss=0.009461, over 3041614.90 frames. ], batch size: 61, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:46:12,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2002793.3333333333, ans=0.125
2023-11-22 15:46:34,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2002926.6666666667, ans=0.0
2023-11-22 15:46:40,262 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.65 vs. limit=12.0
2023-11-22 15:46:47,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300450
2023-11-22 15:46:47,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2002993.3333333333, ans=0.0
2023-11-22 15:46:57,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2002993.3333333333, ans=0.05
2023-11-22 15:46:59,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2003060.0, ans=0.0
2023-11-22 15:47:12,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2003126.6666666667, ans=0.125
2023-11-22 15:47:13,753 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11900, loss[loss=0.08718, simple_loss=0.1231, pruned_loss=0.01731, audio_tagging_loss=0.008304, over 15767.00 frames. ], tot_loss[loss=0.07203, simple_loss=0.09456, pruned_loss=0.01536, audio_tagging_loss=0.009393, over 3040059.10 frames. ], batch size: 60, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:47:18,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2003126.6666666667, ans=0.1
2023-11-22 15:47:20,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2003126.6666666667, ans=0.0
2023-11-22 15:47:21,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2003126.6666666667, ans=0.0
2023-11-22 15:47:26,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2003193.3333333333, ans=0.125
2023-11-22 15:47:31,560 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.15 vs. limit=15.0
2023-11-22 15:47:43,652 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.41 vs. limit=22.5
2023-11-22 15:47:45,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2003260.0, ans=0.1
2023-11-22 15:47:50,473 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300500
2023-11-22 15:48:03,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2003393.3333333333, ans=0.0
2023-11-22 15:48:07,928 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.852e+01 8.243e+01 8.822e+01 9.597e+01 1.131e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 15:48:12,465 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:48:17,709 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 11950, loss[loss=0.08684, simple_loss=0.1255, pruned_loss=0.01665, audio_tagging_loss=0.007454, over 15565.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09309, pruned_loss=0.01497, audio_tagging_loss=0.009569, over 3037375.28 frames. ], batch size: 57, lr: 2.70e-03, grad_scale: 16.0
2023-11-22 15:48:18,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2003460.0, ans=0.0
2023-11-22 15:48:28,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2003460.0, ans=0.0
2023-11-22 15:48:28,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2003460.0, ans=0.2
2023-11-22 15:48:33,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2003526.6666666667, ans=0.0
2023-11-22 15:48:54,167 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300550
2023-11-22 15:49:18,858 INFO [train_asr.py:1221] (0/4) Epoch 25, batch 12000, loss[loss=0.07576, simple_loss=0.1006, pruned_loss=0.01628, audio_tagging_loss=0.0092, over 15870.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.0939, pruned_loss=0.01509, audio_tagging_loss=0.009579, over 3043722.03 frames. ], batch size: 58, lr: 2.70e-03, grad_scale: 32.0
2023-11-22 15:49:18,861 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 15:49:36,091 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.9468, 5.4499, 5.7971, 5.2071], device='cuda:0')
2023-11-22 15:49:37,222 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.6021, 2.9369, 3.1631, 2.7895], device='cuda:0')
2023-11-22 15:49:59,248 INFO [train_asr.py:1253] (0/4) Epoch 25, validation: loss=0.05961, simple_loss=0.05152, pruned_loss=0.005134, audio_tagging_loss=0.02872, over 4681554.00 frames. 
2023-11-22 15:49:59,249 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 15:49:59,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2003793.3333333333, ans=0.1
2023-11-22 15:50:29,904 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-25.pt
2023-11-22 15:51:02,515 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 0, loss[loss=0.09525, simple_loss=0.1034, pruned_loss=0.0211, audio_tagging_loss=0.02245, over 14779.00 frames. ], tot_loss[loss=0.09525, simple_loss=0.1034, pruned_loss=0.0211, audio_tagging_loss=0.02245, over 14779.00 frames. ], batch size: 56, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:51:02,518 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 15:51:37,686 INFO [train_asr.py:1253] (0/4) Epoch 26, validation: loss=0.05869, simple_loss=0.05153, pruned_loss=0.005094, audio_tagging_loss=0.02783, over 4681554.00 frames. 
2023-11-22 15:51:37,687 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 15:51:39,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2003960.0, ans=0.0
2023-11-22 15:51:43,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300600
2023-11-22 15:51:47,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2003960.0, ans=0.1
2023-11-22 15:52:01,846 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.089e+01 8.448e+01 9.337e+01 1.025e+02 1.392e+02, threshold=1.867e+02, percent-clipped=0.0
2023-11-22 15:52:16,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2004160.0, ans=0.125
2023-11-22 15:52:38,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2004226.6666666667, ans=0.1
2023-11-22 15:52:38,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2004226.6666666667, ans=0.125
2023-11-22 15:52:43,363 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 50, loss[loss=0.07441, simple_loss=0.0882, pruned_loss=0.013, audio_tagging_loss=0.01731, over 15003.00 frames. ], tot_loss[loss=0.08036, simple_loss=0.0949, pruned_loss=0.01515, audio_tagging_loss=0.01776, over 686620.58 frames. ], batch size: 59, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:52:48,472 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300650
2023-11-22 15:53:03,637 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.67 vs. limit=15.0
2023-11-22 15:53:30,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2004493.3333333333, ans=0.0
2023-11-22 15:53:44,071 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.99 vs. limit=8.0
2023-11-22 15:53:48,100 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 100, loss[loss=0.07914, simple_loss=0.09177, pruned_loss=0.01589, audio_tagging_loss=0.01736, over 14635.00 frames. ], tot_loss[loss=0.08021, simple_loss=0.09603, pruned_loss=0.01504, audio_tagging_loss=0.01716, over 1210406.82 frames. ], batch size: 56, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:53:53,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300700
2023-11-22 15:53:56,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2004626.6666666667, ans=0.125
2023-11-22 15:54:06,585 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:54:11,742 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.950e+01 8.718e+01 9.300e+01 1.020e+02 1.184e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-22 15:54:22,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2004760.0, ans=0.2
2023-11-22 15:54:39,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2004893.3333333333, ans=0.125
2023-11-22 15:54:45,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2004893.3333333333, ans=0.125
2023-11-22 15:54:47,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2004893.3333333333, ans=0.0
2023-11-22 15:54:53,103 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 150, loss[loss=0.06367, simple_loss=0.07282, pruned_loss=0.01564, audio_tagging_loss=0.01162, over 15253.00 frames. ], tot_loss[loss=0.07865, simple_loss=0.09624, pruned_loss=0.01537, audio_tagging_loss=0.01516, over 1616646.22 frames. ], batch size: 57, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:54:58,111 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300750
2023-11-22 15:55:04,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2005026.6666666667, ans=0.2
2023-11-22 15:55:11,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2005026.6666666667, ans=0.125
2023-11-22 15:55:16,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2005026.6666666667, ans=0.125
2023-11-22 15:55:26,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2005093.3333333333, ans=0.2
2023-11-22 15:55:57,038 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 200, loss[loss=0.05705, simple_loss=0.06812, pruned_loss=0.01273, audio_tagging_loss=0.01026, over 14029.00 frames. ], tot_loss[loss=0.07603, simple_loss=0.09512, pruned_loss=0.01498, audio_tagging_loss=0.0135, over 1930047.09 frames. ], batch size: 56, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:56:02,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300800
2023-11-22 15:56:08,770 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.14 vs. limit=15.0
2023-11-22 15:56:20,097 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.200e+01 8.352e+01 8.945e+01 1.004e+02 1.668e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-22 15:56:39,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2005493.3333333333, ans=0.1
2023-11-22 15:57:01,858 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 250, loss[loss=0.08192, simple_loss=0.1135, pruned_loss=0.01693, audio_tagging_loss=0.008228, over 15333.00 frames. ], tot_loss[loss=0.07485, simple_loss=0.09523, pruned_loss=0.015, audio_tagging_loss=0.01223, over 2179932.54 frames. ], batch size: 56, lr: 2.65e-03, grad_scale: 32.0
2023-11-22 15:57:06,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300850
2023-11-22 15:57:20,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2005693.3333333333, ans=15.0
2023-11-22 15:57:21,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2005693.3333333333, ans=0.1
2023-11-22 15:57:39,704 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 15:58:07,222 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 300, loss[loss=0.09144, simple_loss=0.122, pruned_loss=0.02183, audio_tagging_loss=0.008611, over 15165.00 frames. ], tot_loss[loss=0.0739, simple_loss=0.09498, pruned_loss=0.01503, audio_tagging_loss=0.01138, over 2374925.87 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 15:58:12,341 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300900
2023-11-22 15:58:24,671 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.22 vs. limit=15.0
2023-11-22 15:58:30,066 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.299e+01 8.395e+01 9.138e+01 9.781e+01 1.359e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-22 15:58:31,987 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.33 vs. limit=22.5
2023-11-22 15:58:35,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2006093.3333333333, ans=0.125
2023-11-22 15:58:42,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2006093.3333333333, ans=0.2
2023-11-22 15:58:58,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2006226.6666666667, ans=0.125
2023-11-22 15:59:12,888 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 350, loss[loss=0.06695, simple_loss=0.0901, pruned_loss=0.01215, audio_tagging_loss=0.009755, over 15856.00 frames. ], tot_loss[loss=0.07334, simple_loss=0.0948, pruned_loss=0.01507, audio_tagging_loss=0.01087, over 2528944.66 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 15:59:15,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2006293.3333333333, ans=0.0
2023-11-22 15:59:18,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 300950
2023-11-22 15:59:24,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2006360.0, ans=0.125
2023-11-22 15:59:51,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2006493.3333333333, ans=0.125
2023-11-22 15:59:58,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2006493.3333333333, ans=0.0
2023-11-22 16:00:01,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2006493.3333333333, ans=0.0
2023-11-22 16:00:17,751 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 400, loss[loss=0.05061, simple_loss=0.06454, pruned_loss=0.007912, audio_tagging_loss=0.01043, over 14762.00 frames. ], tot_loss[loss=0.0727, simple_loss=0.09422, pruned_loss=0.01512, audio_tagging_loss=0.01048, over 2651028.94 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:00:20,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2006626.6666666667, ans=0.1
2023-11-22 16:00:23,554 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301000
2023-11-22 16:00:37,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2006693.3333333333, ans=0.2
2023-11-22 16:00:42,288 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.912e+01 8.126e+01 8.693e+01 9.257e+01 1.166e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 16:01:00,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2006826.6666666667, ans=0.125
2023-11-22 16:01:05,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.55 vs. limit=12.0
2023-11-22 16:01:23,369 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 450, loss[loss=0.07098, simple_loss=0.08912, pruned_loss=0.01629, audio_tagging_loss=0.01012, over 15805.00 frames. ], tot_loss[loss=0.0721, simple_loss=0.09369, pruned_loss=0.01498, audio_tagging_loss=0.01027, over 2746697.94 frames. ], batch size: 61, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:01:29,090 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301050
2023-11-22 16:01:54,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2007093.3333333333, ans=0.125
2023-11-22 16:01:58,578 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.08 vs. limit=22.5
2023-11-22 16:02:15,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2007226.6666666667, ans=0.07
2023-11-22 16:02:28,743 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 500, loss[loss=0.06384, simple_loss=0.08547, pruned_loss=0.01212, audio_tagging_loss=0.008982, over 15311.00 frames. ], tot_loss[loss=0.07225, simple_loss=0.09442, pruned_loss=0.01501, audio_tagging_loss=0.01003, over 2815802.50 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:02:30,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2007293.3333333333, ans=0.125
2023-11-22 16:02:32,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2007293.3333333333, ans=0.0
2023-11-22 16:02:33,736 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301100
2023-11-22 16:02:34,003 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 16:02:42,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2007360.0, ans=0.0
2023-11-22 16:02:50,568 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.306e+01 8.908e+01 9.851e+01 1.431e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-22 16:02:58,744 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0
2023-11-22 16:03:32,323 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 550, loss[loss=0.072, simple_loss=0.1032, pruned_loss=0.01314, audio_tagging_loss=0.007257, over 15971.00 frames. ], tot_loss[loss=0.07179, simple_loss=0.0941, pruned_loss=0.01483, audio_tagging_loss=0.009913, over 2866757.34 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:03:37,309 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301150
2023-11-22 16:03:39,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2007626.6666666667, ans=0.0
2023-11-22 16:03:53,343 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.36 vs. limit=22.5
2023-11-22 16:04:01,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.78 vs. limit=15.0
2023-11-22 16:04:10,898 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.18 vs. limit=15.0
2023-11-22 16:04:36,992 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 600, loss[loss=0.07777, simple_loss=0.0991, pruned_loss=0.01729, audio_tagging_loss=0.01093, over 14414.00 frames. ], tot_loss[loss=0.07156, simple_loss=0.09385, pruned_loss=0.01483, audio_tagging_loss=0.009796, over 2907972.53 frames. ], batch size: 53, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:04:42,584 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301200
2023-11-22 16:04:45,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2007960.0, ans=0.125
2023-11-22 16:04:52,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2008026.6666666667, ans=0.125
2023-11-22 16:04:58,771 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.46 vs. limit=15.0
2023-11-22 16:05:01,751 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.667e+01 8.281e+01 8.745e+01 9.297e+01 1.116e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 16:05:21,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2008160.0, ans=0.125
2023-11-22 16:05:26,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.42 vs. limit=5.0
2023-11-22 16:05:35,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2008226.6666666667, ans=0.0
2023-11-22 16:05:37,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2008226.6666666667, ans=0.125
2023-11-22 16:05:42,008 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 650, loss[loss=0.07883, simple_loss=0.1072, pruned_loss=0.01654, audio_tagging_loss=0.008702, over 14942.00 frames. ], tot_loss[loss=0.07181, simple_loss=0.09451, pruned_loss=0.01483, audio_tagging_loss=0.009732, over 2934392.01 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:05:47,004 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301250
2023-11-22 16:05:48,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2008293.3333333333, ans=0.1
2023-11-22 16:06:13,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2008426.6666666667, ans=0.125
2023-11-22 16:06:22,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2008493.3333333333, ans=0.0
2023-11-22 16:06:23,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2008493.3333333333, ans=0.1
2023-11-22 16:06:27,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2008493.3333333333, ans=0.1
2023-11-22 16:06:28,983 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.13 vs. limit=22.5
2023-11-22 16:06:45,898 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 700, loss[loss=0.06231, simple_loss=0.07798, pruned_loss=0.01445, audio_tagging_loss=0.008866, over 14397.00 frames. ], tot_loss[loss=0.072, simple_loss=0.09468, pruned_loss=0.01501, audio_tagging_loss=0.009651, over 2964817.08 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:06:50,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301300
2023-11-22 16:07:11,110 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.534e+01 8.183e+01 8.760e+01 9.542e+01 1.269e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 16:07:13,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2008760.0, ans=0.09899494936611666
2023-11-22 16:07:23,393 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.03 vs. limit=22.5
2023-11-22 16:07:26,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2008826.6666666667, ans=0.125
2023-11-22 16:07:27,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.12 vs. limit=22.5
2023-11-22 16:07:37,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2008893.3333333333, ans=0.125
2023-11-22 16:07:47,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2008893.3333333333, ans=0.125
2023-11-22 16:07:49,633 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 750, loss[loss=0.09159, simple_loss=0.1258, pruned_loss=0.02059, audio_tagging_loss=0.008089, over 15140.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09487, pruned_loss=0.01501, audio_tagging_loss=0.009598, over 2992388.57 frames. ], batch size: 53, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:07:53,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2008960.0, ans=0.125
2023-11-22 16:07:55,401 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301350
2023-11-22 16:08:35,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2009160.0, ans=0.1
2023-11-22 16:08:46,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2009226.6666666667, ans=0.0
2023-11-22 16:08:49,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2009226.6666666667, ans=0.0
2023-11-22 16:08:50,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2009226.6666666667, ans=0.1
2023-11-22 16:08:52,415 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.72 vs. limit=6.0
2023-11-22 16:08:55,392 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 800, loss[loss=0.06114, simple_loss=0.08213, pruned_loss=0.01136, audio_tagging_loss=0.008718, over 14865.00 frames. ], tot_loss[loss=0.07189, simple_loss=0.09475, pruned_loss=0.01501, audio_tagging_loss=0.009507, over 3000526.65 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:09:00,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301400
2023-11-22 16:09:19,559 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.729e+01 8.335e+01 8.906e+01 9.640e+01 1.225e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-22 16:09:19,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2009426.6666666667, ans=0.125
2023-11-22 16:09:48,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.21 vs. limit=15.0
2023-11-22 16:09:56,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.33 vs. limit=15.0
2023-11-22 16:10:00,005 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 850, loss[loss=0.07672, simple_loss=0.1085, pruned_loss=0.01546, audio_tagging_loss=0.007026, over 15032.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09505, pruned_loss=0.01502, audio_tagging_loss=0.009576, over 3014170.00 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:10:00,276 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 16:10:04,944 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301450
2023-11-22 16:10:25,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2009760.0, ans=0.1
2023-11-22 16:10:38,226 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 16:10:39,620 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.64 vs. limit=22.5
2023-11-22 16:10:45,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2009826.6666666667, ans=0.2
2023-11-22 16:10:57,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2009893.3333333333, ans=0.0
2023-11-22 16:11:03,344 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 900, loss[loss=0.07264, simple_loss=0.09216, pruned_loss=0.01539, audio_tagging_loss=0.01116, over 15221.00 frames. ], tot_loss[loss=0.07247, simple_loss=0.09512, pruned_loss=0.01519, audio_tagging_loss=0.009722, over 3021534.25 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:11:08,965 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301500
2023-11-22 16:11:09,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.45 vs. limit=15.0
2023-11-22 16:11:14,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2009960.0, ans=0.125
2023-11-22 16:11:27,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2010026.6666666667, ans=0.0
2023-11-22 16:11:28,585 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.162e+01 8.648e+01 9.209e+01 1.007e+02 1.462e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-22 16:11:49,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2010160.0, ans=10.0
2023-11-22 16:12:07,523 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 950, loss[loss=0.09246, simple_loss=0.1265, pruned_loss=0.02033, audio_tagging_loss=0.008869, over 14989.00 frames. ], tot_loss[loss=0.0729, simple_loss=0.09576, pruned_loss=0.01542, audio_tagging_loss=0.009601, over 3023688.55 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:12:13,792 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301550
2023-11-22 16:12:30,065 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.80 vs. limit=6.0
2023-11-22 16:12:41,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2010426.6666666667, ans=0.1
2023-11-22 16:12:47,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2010493.3333333333, ans=0.1
2023-11-22 16:12:48,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2010493.3333333333, ans=0.1
2023-11-22 16:13:11,477 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1000, loss[loss=0.06559, simple_loss=0.08313, pruned_loss=0.01173, audio_tagging_loss=0.0123, over 15647.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09511, pruned_loss=0.01523, audio_tagging_loss=0.009433, over 3024042.93 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:13:15,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2010626.6666666667, ans=0.125
2023-11-22 16:13:16,282 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301600
2023-11-22 16:13:19,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2010626.6666666667, ans=0.0
2023-11-22 16:13:25,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2010693.3333333333, ans=0.025
2023-11-22 16:13:29,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2010693.3333333333, ans=0.125
2023-11-22 16:13:35,431 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.080e+01 8.207e+01 8.822e+01 9.755e+01 1.375e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 16:13:39,722 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:13:50,253 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.69 vs. limit=10.0
2023-11-22 16:13:51,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2010826.6666666667, ans=0.125
2023-11-22 16:13:54,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2010826.6666666667, ans=0.1
2023-11-22 16:14:15,384 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1050, loss[loss=0.07735, simple_loss=0.1029, pruned_loss=0.01688, audio_tagging_loss=0.009038, over 14776.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09475, pruned_loss=0.01519, audio_tagging_loss=0.0093, over 3022123.49 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:14:15,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2010960.0, ans=0.0
2023-11-22 16:14:20,368 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301650
2023-11-22 16:14:48,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2011093.3333333333, ans=0.125
2023-11-22 16:15:20,093 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1100, loss[loss=0.08589, simple_loss=0.1189, pruned_loss=0.0184, audio_tagging_loss=0.008043, over 15214.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09422, pruned_loss=0.01503, audio_tagging_loss=0.00924, over 3025801.34 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:15:20,527 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 16:15:23,764 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:15:25,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301700
2023-11-22 16:15:28,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2011293.3333333333, ans=0.2
2023-11-22 16:15:32,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2011360.0, ans=0.0
2023-11-22 16:15:32,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2011360.0, ans=0.125
2023-11-22 16:15:35,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2011360.0, ans=0.125
2023-11-22 16:15:37,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2011360.0, ans=0.0
2023-11-22 16:15:44,422 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.766e+01 8.159e+01 8.793e+01 9.276e+01 1.252e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 16:15:49,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2011426.6666666667, ans=0.125
2023-11-22 16:15:55,093 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.86 vs. limit=15.0
2023-11-22 16:16:21,915 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.39 vs. limit=15.0
2023-11-22 16:16:24,797 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1150, loss[loss=0.06765, simple_loss=0.08829, pruned_loss=0.01391, audio_tagging_loss=0.009594, over 16062.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09305, pruned_loss=0.01492, audio_tagging_loss=0.009232, over 3029618.93 frames. ], batch size: 60, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:16:28,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2011626.6666666667, ans=0.2
2023-11-22 16:16:29,748 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301750
2023-11-22 16:16:31,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2011626.6666666667, ans=0.125
2023-11-22 16:16:33,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2011626.6666666667, ans=0.125
2023-11-22 16:16:45,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2011693.3333333333, ans=0.0
2023-11-22 16:17:01,954 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.06 vs. limit=15.0
2023-11-22 16:17:04,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2011826.6666666667, ans=0.1
2023-11-22 16:17:14,011 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.28 vs. limit=22.5
2023-11-22 16:17:17,901 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.05 vs. limit=15.0
2023-11-22 16:17:23,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2011893.3333333333, ans=0.1
2023-11-22 16:17:28,012 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1200, loss[loss=0.07435, simple_loss=0.09451, pruned_loss=0.01483, audio_tagging_loss=0.01226, over 14561.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09214, pruned_loss=0.01484, audio_tagging_loss=0.009303, over 3033505.84 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:17:29,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2011960.0, ans=0.125
2023-11-22 16:17:33,048 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301800
2023-11-22 16:17:51,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2012026.6666666667, ans=0.1
2023-11-22 16:17:54,133 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.678e+01 8.163e+01 8.768e+01 9.430e+01 1.301e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 16:17:56,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2012093.3333333333, ans=0.125
2023-11-22 16:17:58,473 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.76 vs. limit=22.5
2023-11-22 16:18:14,440 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 16:18:14,991 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.10 vs. limit=15.0
2023-11-22 16:18:32,639 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1250, loss[loss=0.07053, simple_loss=0.1003, pruned_loss=0.01164, audio_tagging_loss=0.008758, over 16177.00 frames. ], tot_loss[loss=0.07084, simple_loss=0.09307, pruned_loss=0.01499, audio_tagging_loss=0.009323, over 3032010.10 frames. ], batch size: 61, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:18:35,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2012293.3333333333, ans=0.125
2023-11-22 16:18:37,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301850
2023-11-22 16:18:45,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2012360.0, ans=0.07
2023-11-22 16:18:53,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2012360.0, ans=0.0
2023-11-22 16:19:00,459 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.94 vs. limit=10.0
2023-11-22 16:19:36,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2012626.6666666667, ans=0.0
2023-11-22 16:19:37,024 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1300, loss[loss=0.06971, simple_loss=0.09792, pruned_loss=0.01311, audio_tagging_loss=0.007631, over 15132.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09274, pruned_loss=0.01484, audio_tagging_loss=0.009321, over 3030533.37 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:19:37,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2012626.6666666667, ans=0.0
2023-11-22 16:19:42,725 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301900
2023-11-22 16:20:02,345 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.148e+01 8.961e+01 9.729e+01 1.373e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 16:20:23,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2012826.6666666667, ans=0.0
2023-11-22 16:20:37,602 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=13.00 vs. limit=15.0
2023-11-22 16:20:41,722 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1350, loss[loss=0.06257, simple_loss=0.0845, pruned_loss=0.01309, audio_tagging_loss=0.007226, over 15166.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09239, pruned_loss=0.0147, audio_tagging_loss=0.0094, over 3030877.95 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:20:42,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2012960.0, ans=0.125
2023-11-22 16:20:46,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 301950
2023-11-22 16:21:27,991 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.64 vs. limit=12.0
2023-11-22 16:21:29,506 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:21:30,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2013160.0, ans=0.125
2023-11-22 16:21:43,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2013226.6666666667, ans=0.125
2023-11-22 16:21:46,589 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1400, loss[loss=0.0834, simple_loss=0.1173, pruned_loss=0.01777, audio_tagging_loss=0.006961, over 14996.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09217, pruned_loss=0.01463, audio_tagging_loss=0.009424, over 3039644.70 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:21:51,686 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302000
2023-11-22 16:22:03,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2013360.0, ans=0.2
2023-11-22 16:22:07,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2013360.0, ans=0.2
2023-11-22 16:22:13,013 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.088e+01 8.833e+01 9.465e+01 1.089e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 16:22:50,769 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1450, loss[loss=0.06235, simple_loss=0.08438, pruned_loss=0.01143, audio_tagging_loss=0.00874, over 16254.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09208, pruned_loss=0.01446, audio_tagging_loss=0.009425, over 3035429.75 frames. ], batch size: 61, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:22:55,909 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302050
2023-11-22 16:22:57,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2013626.6666666667, ans=0.125
2023-11-22 16:23:09,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2013693.3333333333, ans=0.0
2023-11-22 16:23:22,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2013760.0, ans=0.125
2023-11-22 16:23:31,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2013826.6666666667, ans=0.0
2023-11-22 16:23:39,652 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.76 vs. limit=10.0
2023-11-22 16:23:53,702 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1500, loss[loss=0.06822, simple_loss=0.09002, pruned_loss=0.01415, audio_tagging_loss=0.00906, over 14240.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09374, pruned_loss=0.01492, audio_tagging_loss=0.009415, over 3041636.93 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:23:59,437 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302100
2023-11-22 16:24:05,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2014026.6666666667, ans=0.125
2023-11-22 16:24:17,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2014026.6666666667, ans=0.125
2023-11-22 16:24:20,918 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.201e+01 8.781e+01 9.408e+01 1.231e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 16:24:38,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2014160.0, ans=0.125
2023-11-22 16:24:44,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2014226.6666666667, ans=0.1
2023-11-22 16:24:58,111 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1550, loss[loss=0.08633, simple_loss=0.1192, pruned_loss=0.01615, audio_tagging_loss=0.01057, over 15377.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09288, pruned_loss=0.01482, audio_tagging_loss=0.009494, over 3032620.76 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:25:04,312 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302150
2023-11-22 16:25:17,768 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.32 vs. limit=22.5
2023-11-22 16:25:25,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2014426.6666666667, ans=0.125
2023-11-22 16:25:26,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2014426.6666666667, ans=0.0
2023-11-22 16:25:32,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2014426.6666666667, ans=0.0
2023-11-22 16:25:43,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2014493.3333333333, ans=0.125
2023-11-22 16:25:53,418 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.45 vs. limit=15.0
2023-11-22 16:25:54,165 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 16:26:03,155 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1600, loss[loss=0.0706, simple_loss=0.08766, pruned_loss=0.01696, audio_tagging_loss=0.009806, over 15621.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09312, pruned_loss=0.01501, audio_tagging_loss=0.00955, over 3036416.07 frames. ], batch size: 60, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:26:07,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302200
2023-11-22 16:26:12,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2014626.6666666667, ans=0.2
2023-11-22 16:26:29,466 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.323e+01 8.320e+01 8.949e+01 9.605e+01 1.369e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 16:26:45,528 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.593e-03
2023-11-22 16:27:06,578 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1650, loss[loss=0.08742, simple_loss=0.1213, pruned_loss=0.01815, audio_tagging_loss=0.008606, over 15793.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09248, pruned_loss=0.01481, audio_tagging_loss=0.009607, over 3038496.30 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:27:11,669 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302250
2023-11-22 16:27:17,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2014960.0, ans=0.0
2023-11-22 16:27:35,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2015093.3333333333, ans=0.125
2023-11-22 16:28:00,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2015226.6666666667, ans=0.5
2023-11-22 16:28:00,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2015226.6666666667, ans=0.125
2023-11-22 16:28:10,473 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1700, loss[loss=0.07294, simple_loss=0.09737, pruned_loss=0.01845, audio_tagging_loss=0.005809, over 14589.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09222, pruned_loss=0.01479, audio_tagging_loss=0.009626, over 3037375.36 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:28:15,394 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302300
2023-11-22 16:28:34,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2015426.6666666667, ans=0.125
2023-11-22 16:28:36,855 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.605e+01 8.196e+01 8.694e+01 9.442e+01 1.208e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 16:28:39,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2015426.6666666667, ans=0.125
2023-11-22 16:28:52,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2015493.3333333333, ans=0.95
2023-11-22 16:29:03,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2015560.0, ans=0.025
2023-11-22 16:29:13,282 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1750, loss[loss=0.05089, simple_loss=0.06528, pruned_loss=0.006132, audio_tagging_loss=0.01212, over 14113.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09209, pruned_loss=0.01487, audio_tagging_loss=0.009548, over 3032846.89 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:29:14,148 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.70 vs. limit=22.5
2023-11-22 16:29:18,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302350
2023-11-22 16:29:27,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2015693.3333333333, ans=0.2
2023-11-22 16:29:31,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2015693.3333333333, ans=0.125
2023-11-22 16:29:33,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2015693.3333333333, ans=0.2
2023-11-22 16:29:50,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2015826.6666666667, ans=0.125
2023-11-22 16:30:06,636 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.55 vs. limit=22.5
2023-11-22 16:30:17,045 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1800, loss[loss=0.07289, simple_loss=0.09736, pruned_loss=0.01566, audio_tagging_loss=0.008544, over 14373.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09171, pruned_loss=0.01465, audio_tagging_loss=0.009473, over 3037623.46 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:30:22,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302400
2023-11-22 16:30:43,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2016093.3333333333, ans=0.125
2023-11-22 16:30:43,635 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.44 vs. limit=15.0
2023-11-22 16:30:44,313 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.100e+01 8.696e+01 9.181e+01 1.134e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 16:30:51,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2016093.3333333333, ans=0.125
2023-11-22 16:31:07,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2016226.6666666667, ans=0.125
2023-11-22 16:31:08,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2016226.6666666667, ans=0.0
2023-11-22 16:31:16,319 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.17 vs. limit=22.5
2023-11-22 16:31:20,406 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1850, loss[loss=0.08834, simple_loss=0.1154, pruned_loss=0.02118, audio_tagging_loss=0.009475, over 14752.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09199, pruned_loss=0.01467, audio_tagging_loss=0.009385, over 3039826.03 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:31:25,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302450
2023-11-22 16:31:41,060 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.18 vs. limit=15.0
2023-11-22 16:31:43,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2016360.0, ans=0.0
2023-11-22 16:32:01,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2016493.3333333333, ans=0.1
2023-11-22 16:32:01,521 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.87 vs. limit=22.5
2023-11-22 16:32:07,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2016493.3333333333, ans=0.1
2023-11-22 16:32:09,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2016493.3333333333, ans=0.125
2023-11-22 16:32:26,257 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1900, loss[loss=0.05642, simple_loss=0.07619, pruned_loss=0.0116, audio_tagging_loss=0.006725, over 13952.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09142, pruned_loss=0.01447, audio_tagging_loss=0.009337, over 3038476.31 frames. ], batch size: 54, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:32:32,000 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302500
2023-11-22 16:32:35,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2016626.6666666667, ans=0.1
2023-11-22 16:32:52,801 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.955e+01 8.093e+01 8.828e+01 9.639e+01 1.190e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 16:32:54,622 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=12.0
2023-11-22 16:33:16,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2016893.3333333333, ans=0.09899494936611666
2023-11-22 16:33:29,827 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 1950, loss[loss=0.07006, simple_loss=0.09321, pruned_loss=0.01684, audio_tagging_loss=0.006607, over 14345.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09157, pruned_loss=0.01446, audio_tagging_loss=0.009291, over 3037963.48 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:33:34,842 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302550
2023-11-22 16:33:46,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2017026.6666666667, ans=0.1
2023-11-22 16:34:14,187 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.88 vs. limit=15.0
2023-11-22 16:34:17,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2017160.0, ans=0.125
2023-11-22 16:34:19,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2017226.6666666667, ans=0.0
2023-11-22 16:34:23,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2017226.6666666667, ans=0.1
2023-11-22 16:34:29,443 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.24 vs. limit=15.0
2023-11-22 16:34:32,530 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2000, loss[loss=0.08736, simple_loss=0.1154, pruned_loss=0.0213, audio_tagging_loss=0.008341, over 15086.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09232, pruned_loss=0.01467, audio_tagging_loss=0.009325, over 3038462.94 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 32.0
2023-11-22 16:34:37,549 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302600
2023-11-22 16:34:44,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2017360.0, ans=0.2
2023-11-22 16:34:47,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2017360.0, ans=0.125
2023-11-22 16:35:00,828 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.236e+01 8.987e+01 9.619e+01 1.204e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 16:35:28,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2017560.0, ans=0.0
2023-11-22 16:35:28,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2017560.0, ans=0.125
2023-11-22 16:35:37,106 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2050, loss[loss=0.06348, simple_loss=0.07471, pruned_loss=0.01326, audio_tagging_loss=0.01287, over 13312.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09308, pruned_loss=0.01484, audio_tagging_loss=0.009244, over 3033690.42 frames. ], batch size: 52, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:35:42,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302650
2023-11-22 16:35:47,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.91 vs. limit=22.5
2023-11-22 16:35:49,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2017693.3333333333, ans=0.125
2023-11-22 16:35:52,361 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.32 vs. limit=22.5
2023-11-22 16:36:26,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2017893.3333333333, ans=0.125
2023-11-22 16:36:40,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2017960.0, ans=0.125
2023-11-22 16:36:41,122 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2100, loss[loss=0.07198, simple_loss=0.08698, pruned_loss=0.01882, audio_tagging_loss=0.009666, over 15232.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09298, pruned_loss=0.01475, audio_tagging_loss=0.009226, over 3035614.82 frames. ], batch size: 59, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:36:42,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2017960.0, ans=0.0
2023-11-22 16:36:46,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302700
2023-11-22 16:36:59,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.36 vs. limit=15.0
2023-11-22 16:37:03,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2018026.6666666667, ans=0.125
2023-11-22 16:37:07,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2018093.3333333333, ans=0.09899494936611666
2023-11-22 16:37:09,223 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.575e+01 8.367e+01 8.996e+01 9.804e+01 1.259e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 16:37:19,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2018160.0, ans=0.0
2023-11-22 16:37:31,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2018226.6666666667, ans=0.125
2023-11-22 16:37:42,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2018226.6666666667, ans=0.125
2023-11-22 16:37:44,332 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2150, loss[loss=0.06077, simple_loss=0.07315, pruned_loss=0.01252, audio_tagging_loss=0.01168, over 15877.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09393, pruned_loss=0.01493, audio_tagging_loss=0.009216, over 3038457.65 frames. ], batch size: 62, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:37:44,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2018293.3333333333, ans=0.0
2023-11-22 16:37:49,350 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302750
2023-11-22 16:37:50,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2018293.3333333333, ans=0.125
2023-11-22 16:37:51,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2018293.3333333333, ans=0.125
2023-11-22 16:38:02,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2018360.0, ans=0.5
2023-11-22 16:38:15,595 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.16 vs. limit=8.0
2023-11-22 16:38:18,993 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.74 vs. limit=15.0
2023-11-22 16:38:23,076 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:38:45,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2018560.0, ans=0.125
2023-11-22 16:38:47,926 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2200, loss[loss=0.07207, simple_loss=0.09821, pruned_loss=0.0136, audio_tagging_loss=0.00937, over 14769.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09417, pruned_loss=0.01491, audio_tagging_loss=0.009302, over 3043793.80 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:38:52,929 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302800
2023-11-22 16:38:54,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2018626.6666666667, ans=0.0
2023-11-22 16:39:02,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2018693.3333333333, ans=0.0
2023-11-22 16:39:16,787 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.468e+01 8.391e+01 8.919e+01 9.600e+01 1.144e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 16:39:24,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2018826.6666666667, ans=0.0
2023-11-22 16:39:25,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2018826.6666666667, ans=0.125
2023-11-22 16:39:32,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2018826.6666666667, ans=10.0
2023-11-22 16:39:34,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2018826.6666666667, ans=0.0
2023-11-22 16:39:51,945 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2250, loss[loss=0.08583, simple_loss=0.1114, pruned_loss=0.02308, audio_tagging_loss=0.007051, over 14863.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09394, pruned_loss=0.0149, audio_tagging_loss=0.009279, over 3044919.37 frames. ], batch size: 55, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:39:56,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302850
2023-11-22 16:40:06,877 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 16:40:25,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2019093.3333333333, ans=0.1
2023-11-22 16:40:29,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2019160.0, ans=0.125
2023-11-22 16:40:38,085 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.34 vs. limit=15.0
2023-11-22 16:40:39,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten.whitening_limit, batch_count=2019160.0, ans=15.0
2023-11-22 16:40:49,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2019226.6666666667, ans=0.125
2023-11-22 16:40:54,895 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2300, loss[loss=0.0591, simple_loss=0.07125, pruned_loss=0.01266, audio_tagging_loss=0.01082, over 15920.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09372, pruned_loss=0.01489, audio_tagging_loss=0.009274, over 3044698.97 frames. ], batch size: 60, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:40:57,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2019293.3333333333, ans=0.125
2023-11-22 16:40:59,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302900
2023-11-22 16:41:03,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2019293.3333333333, ans=0.125
2023-11-22 16:41:17,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2019360.0, ans=0.0
2023-11-22 16:41:20,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2019426.6666666667, ans=0.0
2023-11-22 16:41:24,526 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.904e+01 8.135e+01 8.659e+01 9.382e+01 1.572e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-22 16:41:36,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.99 vs. limit=15.0
2023-11-22 16:41:39,230 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.46 vs. limit=15.0
2023-11-22 16:41:42,986 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.79 vs. limit=12.0
2023-11-22 16:41:50,925 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:41:56,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2019560.0, ans=0.07
2023-11-22 16:41:58,392 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2350, loss[loss=0.06954, simple_loss=0.09648, pruned_loss=0.01194, audio_tagging_loss=0.009352, over 16420.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09346, pruned_loss=0.01473, audio_tagging_loss=0.009395, over 3045742.05 frames. ], batch size: 61, lr: 2.64e-03, grad_scale: 8.0
2023-11-22 16:42:00,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2019626.6666666667, ans=0.2
2023-11-22 16:42:03,998 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 302950
2023-11-22 16:42:29,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2019760.0, ans=0.125
2023-11-22 16:42:33,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.88 vs. limit=15.0
2023-11-22 16:42:38,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2019826.6666666667, ans=0.2
2023-11-22 16:43:00,442 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.12 vs. limit=10.0
2023-11-22 16:43:02,288 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2400, loss[loss=0.0724, simple_loss=0.09938, pruned_loss=0.01498, audio_tagging_loss=0.007724, over 15407.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09379, pruned_loss=0.01476, audio_tagging_loss=0.009438, over 3051815.31 frames. ], batch size: 58, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:43:07,879 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303000
2023-11-22 16:43:27,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2020093.3333333333, ans=0.2
2023-11-22 16:43:31,206 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.331e+01 8.278e+01 8.806e+01 9.570e+01 1.233e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 16:44:03,780 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.18 vs. limit=22.5
2023-11-22 16:44:05,440 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2450, loss[loss=0.08541, simple_loss=0.1078, pruned_loss=0.02399, audio_tagging_loss=0.007527, over 14830.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09371, pruned_loss=0.01474, audio_tagging_loss=0.009495, over 3054522.67 frames. ], batch size: 56, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:44:07,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2020293.3333333333, ans=0.125
2023-11-22 16:44:10,421 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303050
2023-11-22 16:44:20,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2020360.0, ans=0.125
2023-11-22 16:44:20,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2020360.0, ans=0.125
2023-11-22 16:44:33,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2020426.6666666667, ans=0.0
2023-11-22 16:44:41,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2020426.6666666667, ans=0.0
2023-11-22 16:44:46,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2020493.3333333333, ans=0.1
2023-11-22 16:44:47,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2020493.3333333333, ans=0.0
2023-11-22 16:45:08,153 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2500, loss[loss=0.07674, simple_loss=0.09718, pruned_loss=0.01927, audio_tagging_loss=0.008881, over 15133.00 frames. ], tot_loss[loss=0.07151, simple_loss=0.09388, pruned_loss=0.01499, audio_tagging_loss=0.009586, over 3051186.56 frames. ], batch size: 57, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:45:11,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2020626.6666666667, ans=0.025
2023-11-22 16:45:13,806 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303100
2023-11-22 16:45:27,043 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-22 16:45:30,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2020693.3333333333, ans=0.0
2023-11-22 16:45:37,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2020760.0, ans=0.5
2023-11-22 16:45:37,912 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.127e+01 8.699e+01 9.305e+01 1.342e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 16:45:47,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2020826.6666666667, ans=0.0
2023-11-22 16:45:51,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2020826.6666666667, ans=0.2
2023-11-22 16:46:12,049 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2550, loss[loss=0.08439, simple_loss=0.115, pruned_loss=0.01952, audio_tagging_loss=0.007343, over 14294.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09384, pruned_loss=0.01503, audio_tagging_loss=0.009515, over 3048429.07 frames. ], batch size: 53, lr: 2.64e-03, grad_scale: 16.0
2023-11-22 16:46:16,927 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303150
2023-11-22 16:46:26,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2021026.6666666667, ans=0.07
2023-11-22 16:46:27,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2021026.6666666667, ans=0.0
2023-11-22 16:46:29,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2021026.6666666667, ans=0.1
2023-11-22 16:46:32,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2021026.6666666667, ans=0.1
2023-11-22 16:46:35,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2021026.6666666667, ans=0.125
2023-11-22 16:46:37,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2021093.3333333333, ans=0.125
2023-11-22 16:46:43,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2021093.3333333333, ans=0.1
2023-11-22 16:46:55,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2021160.0, ans=0.0
2023-11-22 16:46:55,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2021160.0, ans=0.0
2023-11-22 16:47:00,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2021160.0, ans=0.0
2023-11-22 16:47:15,971 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2600, loss[loss=0.07251, simple_loss=0.09189, pruned_loss=0.0157, audio_tagging_loss=0.01087, over 14337.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09244, pruned_loss=0.01471, audio_tagging_loss=0.009459, over 3046361.58 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:47:20,955 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303200
2023-11-22 16:47:26,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2021293.3333333333, ans=0.0
2023-11-22 16:47:26,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2021293.3333333333, ans=0.0
2023-11-22 16:47:37,865 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.92 vs. limit=6.0
2023-11-22 16:47:44,774 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.809e+01 8.118e+01 8.704e+01 9.634e+01 1.548e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 16:47:54,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2021493.3333333333, ans=0.2
2023-11-22 16:47:57,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2021493.3333333333, ans=0.125
2023-11-22 16:48:17,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2021560.0, ans=0.1
2023-11-22 16:48:19,363 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2650, loss[loss=0.08131, simple_loss=0.1114, pruned_loss=0.02014, audio_tagging_loss=0.005496, over 15407.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09307, pruned_loss=0.0149, audio_tagging_loss=0.009311, over 3043164.88 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:48:22,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2021626.6666666667, ans=0.125
2023-11-22 16:48:24,367 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303250
2023-11-22 16:48:35,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2021693.3333333333, ans=0.95
2023-11-22 16:48:43,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2021693.3333333333, ans=0.125
2023-11-22 16:48:45,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.whiten.whitening_limit, batch_count=2021760.0, ans=12.0
2023-11-22 16:49:23,150 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2700, loss[loss=0.07151, simple_loss=0.08817, pruned_loss=0.01638, audio_tagging_loss=0.01105, over 15237.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09288, pruned_loss=0.01479, audio_tagging_loss=0.009254, over 3046977.43 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:49:28,827 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303300
2023-11-22 16:49:31,830 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.14 vs. limit=15.0
2023-11-22 16:49:51,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2022093.3333333333, ans=0.0
2023-11-22 16:49:53,758 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.014e+01 8.226e+01 8.902e+01 9.755e+01 1.486e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-22 16:50:26,598 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2750, loss[loss=0.07548, simple_loss=0.09767, pruned_loss=0.01827, audio_tagging_loss=0.008374, over 14780.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09278, pruned_loss=0.01459, audio_tagging_loss=0.009251, over 3050617.62 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 8.0
2023-11-22 16:50:29,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2022293.3333333333, ans=0.125
2023-11-22 16:50:31,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303350
2023-11-22 16:50:58,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2022426.6666666667, ans=0.125
2023-11-22 16:51:01,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2022426.6666666667, ans=0.0
2023-11-22 16:51:02,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2022426.6666666667, ans=0.09899494936611666
2023-11-22 16:51:15,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2022493.3333333333, ans=0.125
2023-11-22 16:51:21,029 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:51:23,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2022560.0, ans=0.05
2023-11-22 16:51:30,236 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2800, loss[loss=0.07004, simple_loss=0.101, pruned_loss=0.01356, audio_tagging_loss=0.005954, over 13912.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09352, pruned_loss=0.0146, audio_tagging_loss=0.009197, over 3041625.40 frames. ], batch size: 53, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:51:35,411 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303400
2023-11-22 16:51:46,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2022693.3333333333, ans=0.0
2023-11-22 16:51:47,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2022693.3333333333, ans=0.2
2023-11-22 16:51:48,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2022693.3333333333, ans=0.0
2023-11-22 16:51:50,673 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.61 vs. limit=10.0
2023-11-22 16:51:51,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2022693.3333333333, ans=0.125
2023-11-22 16:51:53,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2022693.3333333333, ans=0.125
2023-11-22 16:51:54,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2022693.3333333333, ans=0.1
2023-11-22 16:51:58,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2022760.0, ans=0.125
2023-11-22 16:51:59,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2022760.0, ans=0.5
2023-11-22 16:52:01,427 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.550e+01 8.038e+01 8.757e+01 9.417e+01 1.241e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 16:52:08,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.30 vs. limit=15.0
2023-11-22 16:52:25,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2022893.3333333333, ans=0.5
2023-11-22 16:52:34,459 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2850, loss[loss=0.06145, simple_loss=0.0719, pruned_loss=0.01388, audio_tagging_loss=0.01162, over 14742.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09319, pruned_loss=0.01486, audio_tagging_loss=0.009238, over 3038521.04 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:52:40,055 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303450
2023-11-22 16:52:47,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2023026.6666666667, ans=0.0
2023-11-22 16:52:48,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2023026.6666666667, ans=0.2
2023-11-22 16:53:11,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2023160.0, ans=0.125
2023-11-22 16:53:14,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2023160.0, ans=0.1
2023-11-22 16:53:29,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2023226.6666666667, ans=0.0
2023-11-22 16:53:37,520 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2900, loss[loss=0.09561, simple_loss=0.1325, pruned_loss=0.02241, audio_tagging_loss=0.006934, over 16011.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09257, pruned_loss=0.01466, audio_tagging_loss=0.009171, over 3035087.81 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:53:42,460 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303500
2023-11-22 16:54:07,706 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.961e+01 8.261e+01 8.993e+01 9.849e+01 1.244e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 16:54:09,689 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.02 vs. limit=15.0
2023-11-22 16:54:24,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2023493.3333333333, ans=0.0
2023-11-22 16:54:31,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2023560.0, ans=0.0
2023-11-22 16:54:39,787 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 2950, loss[loss=0.05993, simple_loss=0.08423, pruned_loss=0.007287, audio_tagging_loss=0.01053, over 15089.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09315, pruned_loss=0.01467, audio_tagging_loss=0.009178, over 3034983.09 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:54:45,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303550
2023-11-22 16:55:31,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2023893.3333333333, ans=0.125
2023-11-22 16:55:43,877 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3000, loss[loss=0.101, simple_loss=0.1438, pruned_loss=0.02184, audio_tagging_loss=0.007278, over 16514.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09274, pruned_loss=0.01471, audio_tagging_loss=0.009316, over 3037465.91 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:55:43,880 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 16:56:12,729 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.5996, 3.5784, 3.8832, 3.4191], device='cuda:0')
2023-11-22 16:56:24,248 INFO [train_asr.py:1253] (0/4) Epoch 26, validation: loss=0.05863, simple_loss=0.05148, pruned_loss=0.005087, audio_tagging_loss=0.0278, over 4681554.00 frames. 
2023-11-22 16:56:24,249 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 16:56:28,711 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.38 vs. limit=22.5
2023-11-22 16:56:29,230 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303600
2023-11-22 16:56:38,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2024026.6666666667, ans=0.125
2023-11-22 16:56:41,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2024026.6666666667, ans=0.0
2023-11-22 16:56:42,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2024026.6666666667, ans=0.125
2023-11-22 16:56:55,188 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.309e+01 8.958e+01 9.598e+01 2.915e+02, threshold=1.792e+02, percent-clipped=1.0
2023-11-22 16:57:09,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2024160.0, ans=0.2
2023-11-22 16:57:11,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2024160.0, ans=0.0
2023-11-22 16:57:12,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2024160.0, ans=0.125
2023-11-22 16:57:27,863 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3050, loss[loss=0.07845, simple_loss=0.09598, pruned_loss=0.01761, audio_tagging_loss=0.01286, over 15137.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09213, pruned_loss=0.01478, audio_tagging_loss=0.009393, over 3036185.19 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:57:32,993 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303650
2023-11-22 16:57:42,489 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.30 vs. limit=15.0
2023-11-22 16:57:54,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2024426.6666666667, ans=0.125
2023-11-22 16:58:06,781 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 16:58:21,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2024560.0, ans=0.0
2023-11-22 16:58:23,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.74 vs. limit=22.5
2023-11-22 16:58:30,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2024560.0, ans=0.0
2023-11-22 16:58:33,083 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3100, loss[loss=0.05925, simple_loss=0.07601, pruned_loss=0.009026, audio_tagging_loss=0.01223, over 15618.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09273, pruned_loss=0.01482, audio_tagging_loss=0.009333, over 3043924.45 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:58:38,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303700
2023-11-22 16:58:53,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2024693.3333333333, ans=0.125
2023-11-22 16:58:57,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2024760.0, ans=0.1
2023-11-22 16:59:03,002 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.458e+01 8.373e+01 8.955e+01 9.319e+01 1.385e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 16:59:19,611 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.20 vs. limit=10.0
2023-11-22 16:59:33,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2024893.3333333333, ans=0.125
2023-11-22 16:59:36,722 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3150, loss[loss=0.08606, simple_loss=0.1187, pruned_loss=0.0197, audio_tagging_loss=0.007001, over 16074.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09328, pruned_loss=0.01484, audio_tagging_loss=0.009383, over 3045200.27 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 16:59:41,714 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303750
2023-11-22 17:00:39,346 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3200, loss[loss=0.06602, simple_loss=0.07648, pruned_loss=0.01326, audio_tagging_loss=0.01452, over 14673.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.09472, pruned_loss=0.01524, audio_tagging_loss=0.009362, over 3053494.60 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:00:44,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303800
2023-11-22 17:01:05,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2025426.6666666667, ans=0.1
2023-11-22 17:01:10,783 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.434e+01 8.209e+01 8.808e+01 9.580e+01 1.231e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 17:01:18,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2025493.3333333333, ans=0.125
2023-11-22 17:01:21,105 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.98 vs. limit=22.5
2023-11-22 17:01:36,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2025560.0, ans=0.95
2023-11-22 17:01:39,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2025560.0, ans=0.125
2023-11-22 17:01:42,708 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3250, loss[loss=0.07066, simple_loss=0.09641, pruned_loss=0.01457, audio_tagging_loss=0.007886, over 15004.00 frames. ], tot_loss[loss=0.0716, simple_loss=0.09414, pruned_loss=0.01499, audio_tagging_loss=0.009538, over 3052292.84 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:01:48,413 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303850
2023-11-22 17:01:51,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2025626.6666666667, ans=0.125
2023-11-22 17:01:55,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2025693.3333333333, ans=0.0
2023-11-22 17:02:46,318 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3300, loss[loss=0.04958, simple_loss=0.06086, pruned_loss=0.009847, audio_tagging_loss=0.009304, over 15096.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09331, pruned_loss=0.01496, audio_tagging_loss=0.009586, over 3048448.27 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:02:46,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2025960.0, ans=0.125
2023-11-22 17:02:51,240 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303900
2023-11-22 17:03:16,791 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.797e+01 8.259e+01 8.793e+01 9.669e+01 1.578e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 17:03:19,567 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.82 vs. limit=22.5
2023-11-22 17:03:21,312 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:03:31,391 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.07 vs. limit=15.0
2023-11-22 17:03:31,687 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.34 vs. limit=22.5
2023-11-22 17:03:35,284 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.75 vs. limit=15.0
2023-11-22 17:03:49,243 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3350, loss[loss=0.09001, simple_loss=0.1209, pruned_loss=0.02129, audio_tagging_loss=0.008261, over 15429.00 frames. ], tot_loss[loss=0.07196, simple_loss=0.0945, pruned_loss=0.01524, audio_tagging_loss=0.009466, over 3059231.24 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:03:54,280 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 303950
2023-11-22 17:04:02,282 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.80 vs. limit=22.5
2023-11-22 17:04:04,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.92 vs. limit=22.5
2023-11-22 17:04:33,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2026493.3333333333, ans=0.125
2023-11-22 17:04:40,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2026560.0, ans=0.125
2023-11-22 17:04:41,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2026560.0, ans=0.0
2023-11-22 17:04:43,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2026560.0, ans=0.125
2023-11-22 17:04:45,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2026560.0, ans=0.0
2023-11-22 17:04:45,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2026560.0, ans=0.125
2023-11-22 17:04:51,538 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3400, loss[loss=0.06137, simple_loss=0.08666, pruned_loss=0.01001, audio_tagging_loss=0.008024, over 15870.00 frames. ], tot_loss[loss=0.0715, simple_loss=0.09436, pruned_loss=0.01501, audio_tagging_loss=0.009309, over 3056065.71 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:04:56,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304000
2023-11-22 17:04:58,416 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-304000.pt
2023-11-22 17:05:09,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2026693.3333333333, ans=0.125
2023-11-22 17:05:26,256 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.152e+01 8.823e+01 9.410e+01 1.182e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-22 17:05:37,835 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.55 vs. limit=15.0
2023-11-22 17:05:41,254 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:05:42,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2026826.6666666667, ans=0.125
2023-11-22 17:05:51,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2026893.3333333333, ans=0.125
2023-11-22 17:05:58,784 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3450, loss[loss=0.0832, simple_loss=0.1161, pruned_loss=0.01684, audio_tagging_loss=0.008323, over 15348.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09386, pruned_loss=0.01466, audio_tagging_loss=0.009292, over 3054755.90 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:06:03,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304050
2023-11-22 17:06:04,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2026960.0, ans=0.2
2023-11-22 17:06:04,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2026960.0, ans=0.1
2023-11-22 17:06:16,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2027026.6666666667, ans=0.125
2023-11-22 17:06:20,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2027026.6666666667, ans=0.0
2023-11-22 17:06:20,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2027026.6666666667, ans=0.125
2023-11-22 17:06:23,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2027093.3333333333, ans=0.0
2023-11-22 17:06:33,797 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.32 vs. limit=15.0
2023-11-22 17:06:36,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2027160.0, ans=0.0
2023-11-22 17:06:56,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2027226.6666666667, ans=0.125
2023-11-22 17:07:01,442 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3500, loss[loss=0.06865, simple_loss=0.09329, pruned_loss=0.01398, audio_tagging_loss=0.008018, over 14806.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.0942, pruned_loss=0.01474, audio_tagging_loss=0.009224, over 3052391.91 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:07:06,410 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304100
2023-11-22 17:07:07,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2027293.3333333333, ans=0.125
2023-11-22 17:07:34,002 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.704e+01 8.467e+01 8.969e+01 9.686e+01 1.239e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 17:07:36,575 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:07:42,263 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.83 vs. limit=15.0
2023-11-22 17:08:04,506 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3550, loss[loss=0.08663, simple_loss=0.1214, pruned_loss=0.02024, audio_tagging_loss=0.005686, over 16438.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09457, pruned_loss=0.01487, audio_tagging_loss=0.009315, over 3057519.36 frames. ], batch size: 62, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:08:10,134 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304150
2023-11-22 17:08:12,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2027626.6666666667, ans=0.1
2023-11-22 17:08:16,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2027693.3333333333, ans=0.125
2023-11-22 17:08:24,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.86 vs. limit=22.5
2023-11-22 17:08:28,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2027693.3333333333, ans=0.125
2023-11-22 17:08:36,533 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.41 vs. limit=6.0
2023-11-22 17:08:54,705 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.54 vs. limit=6.0
2023-11-22 17:09:00,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2027893.3333333333, ans=0.2
2023-11-22 17:09:08,693 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3600, loss[loss=0.06002, simple_loss=0.07843, pruned_loss=0.01231, audio_tagging_loss=0.008498, over 14882.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09362, pruned_loss=0.01463, audio_tagging_loss=0.009279, over 3056696.38 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:09:14,348 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304200
2023-11-22 17:09:19,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2027960.0, ans=0.035
2023-11-22 17:09:41,226 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.567e+01 8.177e+01 8.783e+01 9.582e+01 1.117e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 17:09:50,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2028160.0, ans=0.0
2023-11-22 17:10:10,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2028226.6666666667, ans=0.125
2023-11-22 17:10:11,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.23 vs. limit=22.5
2023-11-22 17:10:13,387 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3650, loss[loss=0.06204, simple_loss=0.07151, pruned_loss=0.0164, audio_tagging_loss=0.009881, over 15588.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09409, pruned_loss=0.01464, audio_tagging_loss=0.009232, over 3054653.05 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:10:18,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304250
2023-11-22 17:10:36,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2028360.0, ans=0.1
2023-11-22 17:10:46,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2028426.6666666667, ans=0.2
2023-11-22 17:10:47,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2028426.6666666667, ans=0.2
2023-11-22 17:10:48,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2028426.6666666667, ans=0.125
2023-11-22 17:10:52,726 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.81 vs. limit=15.0
2023-11-22 17:10:53,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2028493.3333333333, ans=0.0
2023-11-22 17:11:16,845 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3700, loss[loss=0.062, simple_loss=0.07552, pruned_loss=0.01419, audio_tagging_loss=0.01005, over 14226.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09392, pruned_loss=0.0146, audio_tagging_loss=0.009229, over 3058921.06 frames. ], batch size: 53, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:11:17,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2028626.6666666667, ans=0.125
2023-11-22 17:11:21,777 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304300
2023-11-22 17:11:44,700 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:11:50,444 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 8.264e+01 8.888e+01 9.594e+01 1.594e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 17:12:06,217 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.63 vs. limit=15.0
2023-11-22 17:12:21,545 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3750, loss[loss=0.05792, simple_loss=0.07567, pruned_loss=0.009461, audio_tagging_loss=0.01063, over 14916.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09453, pruned_loss=0.01484, audio_tagging_loss=0.009188, over 3060083.43 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:12:26,508 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304350
2023-11-22 17:13:03,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2029160.0, ans=0.125
2023-11-22 17:13:05,524 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:13:25,468 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3800, loss[loss=0.06635, simple_loss=0.09229, pruned_loss=0.01181, audio_tagging_loss=0.008392, over 15805.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09449, pruned_loss=0.01476, audio_tagging_loss=0.009246, over 3056124.18 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:13:25,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2029293.3333333333, ans=0.2
2023-11-22 17:13:30,593 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304400
2023-11-22 17:13:30,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2029293.3333333333, ans=0.125
2023-11-22 17:13:56,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2029426.6666666667, ans=0.125
2023-11-22 17:14:00,218 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.081e+01 8.257e+01 8.903e+01 9.664e+01 1.355e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-22 17:14:14,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2029493.3333333333, ans=0.1
2023-11-22 17:14:16,139 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:14:23,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=2029560.0, ans=10.0
2023-11-22 17:14:30,804 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3850, loss[loss=0.07343, simple_loss=0.1024, pruned_loss=0.01553, audio_tagging_loss=0.006704, over 15022.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09407, pruned_loss=0.01487, audio_tagging_loss=0.009259, over 3049810.82 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:14:35,753 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304450
2023-11-22 17:14:38,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2029626.6666666667, ans=0.0
2023-11-22 17:14:40,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.71 vs. limit=12.0
2023-11-22 17:15:08,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2029760.0, ans=0.5
2023-11-22 17:15:09,532 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0
2023-11-22 17:15:16,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2029826.6666666667, ans=0.125
2023-11-22 17:15:20,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2029826.6666666667, ans=0.0
2023-11-22 17:15:20,644 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.10 vs. limit=15.0
2023-11-22 17:15:35,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.83 vs. limit=15.0
2023-11-22 17:15:35,617 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3900, loss[loss=0.07544, simple_loss=0.1036, pruned_loss=0.01618, audio_tagging_loss=0.007458, over 14939.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.09422, pruned_loss=0.01498, audio_tagging_loss=0.009344, over 3045680.34 frames. ], batch size: 54, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:15:39,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2029960.0, ans=0.125
2023-11-22 17:15:41,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304500
2023-11-22 17:16:04,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2030093.3333333333, ans=0.0
2023-11-22 17:16:08,782 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.197e+01 8.799e+01 9.518e+01 1.700e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 17:16:12,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2030160.0, ans=0.1
2023-11-22 17:16:22,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2030160.0, ans=0.0
2023-11-22 17:16:30,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2030226.6666666667, ans=0.125
2023-11-22 17:16:36,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2030226.6666666667, ans=0.125
2023-11-22 17:16:39,739 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 3950, loss[loss=0.07699, simple_loss=0.1017, pruned_loss=0.01672, audio_tagging_loss=0.009438, over 15235.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09479, pruned_loss=0.01504, audio_tagging_loss=0.009333, over 3043252.83 frames. ], batch size: 56, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:16:39,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2030293.3333333333, ans=0.1
2023-11-22 17:16:44,845 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304550
2023-11-22 17:16:46,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2030293.3333333333, ans=0.07
2023-11-22 17:16:47,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2030293.3333333333, ans=0.125
2023-11-22 17:16:54,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2030360.0, ans=0.125
2023-11-22 17:17:10,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2030426.6666666667, ans=0.125
2023-11-22 17:17:10,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2030426.6666666667, ans=0.125
2023-11-22 17:17:22,574 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:17:23,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2030493.3333333333, ans=0.0
2023-11-22 17:17:25,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2030493.3333333333, ans=0.95
2023-11-22 17:17:36,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2030560.0, ans=0.2
2023-11-22 17:17:43,614 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4000, loss[loss=0.08462, simple_loss=0.1056, pruned_loss=0.02173, audio_tagging_loss=0.01008, over 16639.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.095, pruned_loss=0.01515, audio_tagging_loss=0.009363, over 3047985.79 frames. ], batch size: 63, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:17:48,581 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304600
2023-11-22 17:17:52,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=2030626.6666666667, ans=0.02
2023-11-22 17:18:17,463 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.393e+01 9.096e+01 9.757e+01 1.242e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-22 17:18:48,578 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4050, loss[loss=0.04747, simple_loss=0.05904, pruned_loss=0.00715, audio_tagging_loss=0.01081, over 14091.00 frames. ], tot_loss[loss=0.07226, simple_loss=0.09517, pruned_loss=0.01527, audio_tagging_loss=0.00941, over 3047956.69 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:18:52,389 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:18:54,255 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304650
2023-11-22 17:18:54,389 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:19:08,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2031026.6666666667, ans=15.0
2023-11-22 17:19:17,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2031093.3333333333, ans=0.125
2023-11-22 17:19:24,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2031093.3333333333, ans=0.0
2023-11-22 17:19:28,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2031160.0, ans=0.125
2023-11-22 17:19:33,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2031160.0, ans=0.2
2023-11-22 17:19:52,422 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4100, loss[loss=0.05159, simple_loss=0.05664, pruned_loss=0.007742, audio_tagging_loss=0.01553, over 12944.00 frames. ], tot_loss[loss=0.07224, simple_loss=0.09519, pruned_loss=0.01521, audio_tagging_loss=0.009433, over 3043845.94 frames. ], batch size: 52, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:19:53,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2031293.3333333333, ans=0.125
2023-11-22 17:19:57,315 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304700
2023-11-22 17:20:02,875 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.62 vs. limit=15.0
2023-11-22 17:20:16,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2031426.6666666667, ans=0.0
2023-11-22 17:20:19,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2031426.6666666667, ans=0.0
2023-11-22 17:20:22,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2031426.6666666667, ans=0.125
2023-11-22 17:20:22,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2031426.6666666667, ans=0.125
2023-11-22 17:20:25,412 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.231e+01 8.289e+01 8.885e+01 9.485e+01 1.195e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 17:20:40,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2031493.3333333333, ans=0.035
2023-11-22 17:20:56,262 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4150, loss[loss=0.06854, simple_loss=0.09303, pruned_loss=0.01297, audio_tagging_loss=0.009065, over 15795.00 frames. ], tot_loss[loss=0.07216, simple_loss=0.09523, pruned_loss=0.01526, audio_tagging_loss=0.00929, over 3039267.21 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:21:00,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2031626.6666666667, ans=0.1
2023-11-22 17:21:01,813 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304750
2023-11-22 17:21:10,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2031693.3333333333, ans=0.0
2023-11-22 17:21:15,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2031693.3333333333, ans=0.0
2023-11-22 17:21:26,086 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.17 vs. limit=5.0
2023-11-22 17:21:42,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2031826.6666666667, ans=0.1
2023-11-22 17:21:43,417 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:21:56,658 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.16 vs. limit=12.0
2023-11-22 17:22:00,327 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4200, loss[loss=0.07921, simple_loss=0.105, pruned_loss=0.01813, audio_tagging_loss=0.008576, over 15751.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09505, pruned_loss=0.01528, audio_tagging_loss=0.009245, over 3041152.57 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:22:05,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304800
2023-11-22 17:22:09,729 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.01 vs. limit=22.5
2023-11-22 17:22:12,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2032026.6666666667, ans=0.2
2023-11-22 17:22:21,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2032026.6666666667, ans=0.2
2023-11-22 17:22:34,567 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.472e+01 8.973e+01 9.606e+01 1.148e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-22 17:22:35,329 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.84 vs. limit=6.0
2023-11-22 17:22:36,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2032093.3333333333, ans=0.125
2023-11-22 17:22:44,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2032160.0, ans=0.2
2023-11-22 17:22:56,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2032226.6666666667, ans=0.2
2023-11-22 17:22:56,757 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.75 vs. limit=22.5
2023-11-22 17:23:04,821 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4250, loss[loss=0.07556, simple_loss=0.1001, pruned_loss=0.01747, audio_tagging_loss=0.00803, over 15212.00 frames. ], tot_loss[loss=0.07212, simple_loss=0.09569, pruned_loss=0.01515, audio_tagging_loss=0.009127, over 3048293.02 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:23:09,756 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304850
2023-11-22 17:23:19,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2032360.0, ans=0.125
2023-11-22 17:23:19,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2032360.0, ans=10.0
2023-11-22 17:23:29,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2032426.6666666667, ans=0.125
2023-11-22 17:23:54,107 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.38 vs. limit=22.5
2023-11-22 17:23:59,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2032560.0, ans=0.05
2023-11-22 17:24:08,363 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4300, loss[loss=0.08329, simple_loss=0.1135, pruned_loss=0.01868, audio_tagging_loss=0.007848, over 15611.00 frames. ], tot_loss[loss=0.07222, simple_loss=0.09581, pruned_loss=0.01526, audio_tagging_loss=0.009052, over 3048651.88 frames. ], batch size: 62, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:24:08,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2032626.6666666667, ans=0.0
2023-11-22 17:24:13,381 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304900
2023-11-22 17:24:13,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2032626.6666666667, ans=0.125
2023-11-22 17:24:43,399 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.31 vs. limit=15.0
2023-11-22 17:24:44,058 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.878e+01 8.402e+01 8.884e+01 9.738e+01 1.155e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 17:24:45,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2032760.0, ans=0.1
2023-11-22 17:24:49,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.43 vs. limit=15.0
2023-11-22 17:24:58,515 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.74 vs. limit=6.0
2023-11-22 17:25:13,304 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4350, loss[loss=0.05245, simple_loss=0.06364, pruned_loss=0.008804, audio_tagging_loss=0.01182, over 16240.00 frames. ], tot_loss[loss=0.07248, simple_loss=0.09642, pruned_loss=0.0153, audio_tagging_loss=0.008975, over 3048226.94 frames. ], batch size: 63, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:25:13,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2032960.0, ans=0.0
2023-11-22 17:25:16,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2032960.0, ans=0.0
2023-11-22 17:25:16,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2032960.0, ans=0.125
2023-11-22 17:25:19,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 304950
2023-11-22 17:25:24,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2032960.0, ans=0.04949747468305833
2023-11-22 17:25:36,521 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.89 vs. limit=12.0
2023-11-22 17:25:48,376 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=8.61 vs. limit=15.0
2023-11-22 17:25:55,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2033160.0, ans=0.0
2023-11-22 17:26:09,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2033226.6666666667, ans=0.07
2023-11-22 17:26:18,672 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4400, loss[loss=0.08201, simple_loss=0.1092, pruned_loss=0.02039, audio_tagging_loss=0.007015, over 14951.00 frames. ], tot_loss[loss=0.07188, simple_loss=0.09555, pruned_loss=0.01509, audio_tagging_loss=0.009013, over 3043715.94 frames. ], batch size: 53, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:26:23,561 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305000
2023-11-22 17:26:31,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2033360.0, ans=0.0
2023-11-22 17:26:52,031 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.342e+01 8.115e+01 8.926e+01 9.630e+01 1.276e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-22 17:27:04,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.82 vs. limit=10.0
2023-11-22 17:27:07,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2033493.3333333333, ans=0.1
2023-11-22 17:27:10,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2033560.0, ans=0.125
2023-11-22 17:27:22,544 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4450, loss[loss=0.04753, simple_loss=0.06138, pruned_loss=0.005917, audio_tagging_loss=0.01092, over 14215.00 frames. ], tot_loss[loss=0.07234, simple_loss=0.09595, pruned_loss=0.01534, audio_tagging_loss=0.009032, over 3048594.84 frames. ], batch size: 53, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:27:27,583 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305050
2023-11-22 17:27:31,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.31 vs. limit=15.0
2023-11-22 17:27:59,461 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.95 vs. limit=22.5
2023-11-22 17:28:25,540 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4500, loss[loss=0.06854, simple_loss=0.09583, pruned_loss=0.01609, audio_tagging_loss=0.004529, over 14787.00 frames. ], tot_loss[loss=0.07236, simple_loss=0.09585, pruned_loss=0.01542, audio_tagging_loss=0.009015, over 3048493.70 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:28:27,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2033960.0, ans=0.125
2023-11-22 17:28:31,157 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305100
2023-11-22 17:28:37,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2033960.0, ans=0.125
2023-11-22 17:28:44,882 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.24 vs. limit=15.0
2023-11-22 17:29:01,043 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.904e+01 8.181e+01 8.834e+01 9.479e+01 1.227e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 17:29:05,104 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:29:24,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2034226.6666666667, ans=0.125
2023-11-22 17:29:31,658 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4550, loss[loss=0.0777, simple_loss=0.1031, pruned_loss=0.01866, audio_tagging_loss=0.007471, over 15670.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09495, pruned_loss=0.0152, audio_tagging_loss=0.009084, over 3042651.78 frames. ], batch size: 59, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:29:36,707 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305150
2023-11-22 17:29:48,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2034360.0, ans=0.125
2023-11-22 17:30:21,208 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:30:34,625 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4600, loss[loss=0.07513, simple_loss=0.09766, pruned_loss=0.01719, audio_tagging_loss=0.00911, over 15318.00 frames. ], tot_loss[loss=0.07186, simple_loss=0.09492, pruned_loss=0.01528, audio_tagging_loss=0.009121, over 3040161.78 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:30:39,758 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305200
2023-11-22 17:30:51,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2034693.3333333333, ans=0.125
2023-11-22 17:31:11,489 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.052e+01 8.212e+01 8.747e+01 9.317e+01 1.226e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 17:31:33,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2034893.3333333333, ans=0.0
2023-11-22 17:31:38,366 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4650, loss[loss=0.06362, simple_loss=0.08179, pruned_loss=0.01119, audio_tagging_loss=0.01154, over 14906.00 frames. ], tot_loss[loss=0.07157, simple_loss=0.09435, pruned_loss=0.01502, audio_tagging_loss=0.009373, over 3044327.26 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:31:43,282 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305250
2023-11-22 17:31:49,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2034960.0, ans=0.125
2023-11-22 17:32:01,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2035026.6666666667, ans=0.1
2023-11-22 17:32:17,656 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:32:27,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2035160.0, ans=0.0
2023-11-22 17:32:43,678 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4700, loss[loss=0.07886, simple_loss=0.1074, pruned_loss=0.01826, audio_tagging_loss=0.006894, over 14657.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09462, pruned_loss=0.01497, audio_tagging_loss=0.009344, over 3045751.82 frames. ], batch size: 55, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:32:45,623 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.20 vs. limit=15.0
2023-11-22 17:32:49,346 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305300
2023-11-22 17:33:18,406 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.537e+01 8.112e+01 8.766e+01 9.442e+01 1.103e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-22 17:33:47,871 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4750, loss[loss=0.04525, simple_loss=0.05259, pruned_loss=0.008374, audio_tagging_loss=0.01058, over 14918.00 frames. ], tot_loss[loss=0.07106, simple_loss=0.09402, pruned_loss=0.0147, audio_tagging_loss=0.009347, over 3042905.28 frames. ], batch size: 57, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:33:52,952 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305350
2023-11-22 17:34:22,340 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.64 vs. limit=22.5
2023-11-22 17:34:26,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2035826.6666666667, ans=0.125
2023-11-22 17:34:27,148 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.72 vs. limit=15.0
2023-11-22 17:34:27,239 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.70 vs. limit=15.0
2023-11-22 17:34:46,314 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.90 vs. limit=22.5
2023-11-22 17:34:51,751 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4800, loss[loss=0.07311, simple_loss=0.09233, pruned_loss=0.01452, audio_tagging_loss=0.01242, over 16925.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09258, pruned_loss=0.01457, audio_tagging_loss=0.00955, over 3053213.26 frames. ], batch size: 64, lr: 2.63e-03, grad_scale: 32.0
2023-11-22 17:34:53,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2035960.0, ans=0.04949747468305833
2023-11-22 17:34:56,894 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305400
2023-11-22 17:35:08,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2036026.6666666667, ans=0.0
2023-11-22 17:35:28,860 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.644e+01 8.281e+01 8.887e+01 9.672e+01 1.150e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 17:35:31,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2036160.0, ans=0.1
2023-11-22 17:35:33,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2036160.0, ans=0.125
2023-11-22 17:35:57,040 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4850, loss[loss=0.06038, simple_loss=0.07077, pruned_loss=0.01389, audio_tagging_loss=0.0111, over 15609.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09261, pruned_loss=0.01469, audio_tagging_loss=0.009689, over 3053685.93 frames. ], batch size: 60, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:36:02,694 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305450
2023-11-22 17:36:04,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2036293.3333333333, ans=0.125
2023-11-22 17:36:13,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2036360.0, ans=0.125
2023-11-22 17:36:24,480 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.34 vs. limit=15.0
2023-11-22 17:36:53,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2036560.0, ans=0.125
2023-11-22 17:36:58,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2036560.0, ans=0.125
2023-11-22 17:37:01,325 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4900, loss[loss=0.06236, simple_loss=0.08459, pruned_loss=0.01054, audio_tagging_loss=0.009529, over 15262.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09342, pruned_loss=0.01464, audio_tagging_loss=0.009543, over 3056915.23 frames. ], batch size: 58, lr: 2.63e-03, grad_scale: 16.0
2023-11-22 17:37:06,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305500
2023-11-22 17:37:11,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2036626.6666666667, ans=0.125
2023-11-22 17:37:20,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.78 vs. limit=6.0
2023-11-22 17:37:27,158 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:37:34,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2036760.0, ans=0.1
2023-11-22 17:37:38,240 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.464e+01 8.092e+01 8.921e+01 9.687e+01 1.263e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 17:37:42,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2036826.6666666667, ans=0.2
2023-11-22 17:38:04,902 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 4950, loss[loss=0.08012, simple_loss=0.1042, pruned_loss=0.0195, audio_tagging_loss=0.008518, over 16159.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09166, pruned_loss=0.01435, audio_tagging_loss=0.009557, over 3045652.66 frames. ], batch size: 63, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:38:09,861 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305550
2023-11-22 17:38:12,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2036960.0, ans=0.125
2023-11-22 17:38:16,456 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:38:19,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2037026.6666666667, ans=0.0
2023-11-22 17:38:24,266 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.83 vs. limit=22.5
2023-11-22 17:38:54,005 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.72 vs. limit=12.0
2023-11-22 17:39:01,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2037226.6666666667, ans=0.125
2023-11-22 17:39:05,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2037226.6666666667, ans=0.125
2023-11-22 17:39:10,256 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5000, loss[loss=0.04434, simple_loss=0.05885, pruned_loss=0.006187, audio_tagging_loss=0.008729, over 14706.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.0928, pruned_loss=0.01444, audio_tagging_loss=0.009314, over 3047803.44 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:39:15,801 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305600
2023-11-22 17:39:30,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2037360.0, ans=0.125
2023-11-22 17:39:42,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2037426.6666666667, ans=0.125
2023-11-22 17:39:47,348 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 8.198e+01 8.797e+01 9.538e+01 1.250e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 17:39:48,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2037493.3333333333, ans=0.025
2023-11-22 17:39:58,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2037493.3333333333, ans=0.0
2023-11-22 17:40:06,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2037560.0, ans=0.5
2023-11-22 17:40:07,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2037560.0, ans=0.125
2023-11-22 17:40:15,611 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5050, loss[loss=0.06448, simple_loss=0.08874, pruned_loss=0.01138, audio_tagging_loss=0.008739, over 15201.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.0933, pruned_loss=0.0144, audio_tagging_loss=0.009241, over 3048754.48 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:40:21,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305650
2023-11-22 17:41:27,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2037893.3333333333, ans=0.125
2023-11-22 17:41:46,539 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5100, loss[loss=0.0857, simple_loss=0.1102, pruned_loss=0.02231, audio_tagging_loss=0.008287, over 15378.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09338, pruned_loss=0.01438, audio_tagging_loss=0.009148, over 3049172.26 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 8.0
2023-11-22 17:41:50,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2037960.0, ans=0.125
2023-11-22 17:41:54,053 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305700
2023-11-22 17:42:11,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0
2023-11-22 17:42:12,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2038026.6666666667, ans=0.125
2023-11-22 17:42:18,931 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.58 vs. limit=15.0
2023-11-22 17:42:25,997 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.63 vs. limit=12.0
2023-11-22 17:42:41,749 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.755e+01 8.235e+01 8.784e+01 9.418e+01 1.512e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 17:42:54,137 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.04 vs. limit=12.0
2023-11-22 17:42:57,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff3.min_abs, batch_count=2038160.0, ans=0.2
2023-11-22 17:43:19,591 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5150, loss[loss=0.07846, simple_loss=0.1089, pruned_loss=0.01635, audio_tagging_loss=0.007642, over 15050.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09318, pruned_loss=0.01439, audio_tagging_loss=0.009171, over 3045610.11 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 8.0
2023-11-22 17:43:23,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2038293.3333333333, ans=0.1
2023-11-22 17:43:27,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305750
2023-11-22 17:43:36,735 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:44:13,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2038493.3333333333, ans=0.035
2023-11-22 17:44:15,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2038493.3333333333, ans=0.2
2023-11-22 17:44:52,039 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5200, loss[loss=0.0842, simple_loss=0.1136, pruned_loss=0.02036, audio_tagging_loss=0.007053, over 15409.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.0937, pruned_loss=0.01454, audio_tagging_loss=0.009054, over 3052242.50 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:44:53,021 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.06 vs. limit=22.5
2023-11-22 17:44:59,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305800
2023-11-22 17:45:22,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2038693.3333333333, ans=0.1
2023-11-22 17:45:48,141 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.015e+01 8.193e+01 8.836e+01 9.389e+01 1.241e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 17:46:03,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.47 vs. limit=15.0
2023-11-22 17:46:24,771 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5250, loss[loss=0.08876, simple_loss=0.1212, pruned_loss=0.02028, audio_tagging_loss=0.007862, over 15554.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09283, pruned_loss=0.01444, audio_tagging_loss=0.009136, over 3047777.35 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:46:32,890 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305850
2023-11-22 17:46:35,796 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.54 vs. limit=15.0
2023-11-22 17:46:40,934 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.61 vs. limit=15.0
2023-11-22 17:47:43,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2039226.6666666667, ans=0.125
2023-11-22 17:47:57,873 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5300, loss[loss=0.07485, simple_loss=0.1065, pruned_loss=0.01303, audio_tagging_loss=0.008565, over 15588.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.093, pruned_loss=0.01456, audio_tagging_loss=0.009184, over 3042950.91 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:48:05,342 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305900
2023-11-22 17:48:16,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2039360.0, ans=0.035
2023-11-22 17:48:24,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2039360.0, ans=0.0
2023-11-22 17:48:24,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2039360.0, ans=0.125
2023-11-22 17:48:38,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2039426.6666666667, ans=0.125
2023-11-22 17:48:53,041 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.119e+01 8.483e+01 8.855e+01 9.483e+01 1.164e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 17:49:06,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2039493.3333333333, ans=0.05
2023-11-22 17:49:18,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2039560.0, ans=0.2
2023-11-22 17:49:27,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2039560.0, ans=0.125
2023-11-22 17:49:30,737 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5350, loss[loss=0.06763, simple_loss=0.08597, pruned_loss=0.01497, audio_tagging_loss=0.009673, over 14831.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09322, pruned_loss=0.01448, audio_tagging_loss=0.009239, over 3037739.26 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:49:38,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 305950
2023-11-22 17:49:56,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2039693.3333333333, ans=0.0
2023-11-22 17:50:27,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2039826.6666666667, ans=0.2
2023-11-22 17:50:44,942 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.66 vs. limit=10.0
2023-11-22 17:50:49,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2039893.3333333333, ans=0.2
2023-11-22 17:51:02,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2039960.0, ans=0.05
2023-11-22 17:51:03,658 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5400, loss[loss=0.07423, simple_loss=0.09905, pruned_loss=0.01332, audio_tagging_loss=0.01138, over 15571.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09377, pruned_loss=0.01444, audio_tagging_loss=0.009229, over 3038324.49 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:51:11,130 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306000
2023-11-22 17:51:35,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2040026.6666666667, ans=0.05
2023-11-22 17:51:59,438 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.025e+01 8.288e+01 8.843e+01 9.633e+01 1.187e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 17:52:04,124 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.16 vs. limit=22.5
2023-11-22 17:52:13,740 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.94 vs. limit=15.0
2023-11-22 17:52:27,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2040226.6666666667, ans=0.0
2023-11-22 17:52:36,547 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5450, loss[loss=0.07406, simple_loss=0.09899, pruned_loss=0.01694, audio_tagging_loss=0.007621, over 14871.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09352, pruned_loss=0.01443, audio_tagging_loss=0.009275, over 3041728.94 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:52:44,773 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306050
2023-11-22 17:53:40,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2040493.3333333333, ans=0.125
2023-11-22 17:54:08,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2040626.6666666667, ans=0.2
2023-11-22 17:54:08,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2040626.6666666667, ans=0.125
2023-11-22 17:54:10,071 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5500, loss[loss=0.07174, simple_loss=0.07983, pruned_loss=0.01825, audio_tagging_loss=0.01358, over 16983.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09336, pruned_loss=0.0144, audio_tagging_loss=0.0093, over 3044920.46 frames. ], batch size: 65, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:54:17,298 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306100
2023-11-22 17:54:18,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.50 vs. limit=22.5
2023-11-22 17:55:01,994 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:55:05,360 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.963e+01 8.451e+01 8.930e+01 9.689e+01 1.282e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 17:55:19,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2040826.6666666667, ans=0.0
2023-11-22 17:55:25,374 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.18 vs. limit=12.0
2023-11-22 17:55:32,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2040893.3333333333, ans=0.125
2023-11-22 17:55:42,909 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5550, loss[loss=0.06282, simple_loss=0.0748, pruned_loss=0.01574, audio_tagging_loss=0.009676, over 14609.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09285, pruned_loss=0.01434, audio_tagging_loss=0.009414, over 3042493.08 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:55:50,395 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306150
2023-11-22 17:55:52,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2040960.0, ans=0.0
2023-11-22 17:57:13,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2041293.3333333333, ans=0.125
2023-11-22 17:57:14,865 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5600, loss[loss=0.06725, simple_loss=0.08901, pruned_loss=0.01129, audio_tagging_loss=0.01145, over 16143.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09354, pruned_loss=0.01467, audio_tagging_loss=0.009565, over 3051160.10 frames. ], batch size: 62, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 17:57:17,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2041293.3333333333, ans=0.1
2023-11-22 17:57:22,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306200
2023-11-22 17:57:26,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2041293.3333333333, ans=0.5
2023-11-22 17:58:04,327 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.05 vs. limit=6.0
2023-11-22 17:58:11,319 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.138e+01 8.696e+01 9.555e+01 1.151e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 17:58:13,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2041493.3333333333, ans=0.125
2023-11-22 17:58:20,037 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 17:58:38,251 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5650, loss[loss=0.06723, simple_loss=0.08372, pruned_loss=0.01413, audio_tagging_loss=0.01124, over 15552.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09362, pruned_loss=0.01461, audio_tagging_loss=0.00959, over 3057269.06 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 17:58:43,788 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306250
2023-11-22 17:58:45,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.23 vs. limit=15.0
2023-11-22 17:59:01,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2041693.3333333333, ans=0.125
2023-11-22 17:59:08,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2041760.0, ans=0.1
2023-11-22 17:59:13,447 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 17:59:15,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff3.min_abs, batch_count=2041826.6666666667, ans=0.2
2023-11-22 17:59:17,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2041826.6666666667, ans=0.2
2023-11-22 17:59:21,701 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.43 vs. limit=15.0
2023-11-22 17:59:25,439 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.87 vs. limit=15.0
2023-11-22 17:59:42,506 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5700, loss[loss=0.06764, simple_loss=0.09712, pruned_loss=0.01009, audio_tagging_loss=0.008989, over 16724.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09306, pruned_loss=0.01448, audio_tagging_loss=0.009519, over 3058357.77 frames. ], batch size: 62, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 17:59:47,621 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306300
2023-11-22 17:59:47,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2041960.0, ans=0.125
2023-11-22 18:00:05,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2042093.3333333333, ans=0.1
2023-11-22 18:00:22,595 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.203e+01 8.769e+01 9.418e+01 1.173e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 18:00:46,281 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5750, loss[loss=0.09759, simple_loss=0.1295, pruned_loss=0.02326, audio_tagging_loss=0.00959, over 16960.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09344, pruned_loss=0.01454, audio_tagging_loss=0.009414, over 3060915.31 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:00:51,441 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306350
2023-11-22 18:00:55,728 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.52 vs. limit=10.0
2023-11-22 18:01:04,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2042360.0, ans=0.125
2023-11-22 18:01:09,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2042360.0, ans=0.0
2023-11-22 18:01:09,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.65 vs. limit=22.5
2023-11-22 18:01:19,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2042426.6666666667, ans=0.125
2023-11-22 18:01:33,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2042493.3333333333, ans=0.125
2023-11-22 18:01:44,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2042560.0, ans=0.2
2023-11-22 18:01:45,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2042560.0, ans=0.0
2023-11-22 18:01:51,692 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5800, loss[loss=0.07015, simple_loss=0.0939, pruned_loss=0.01368, audio_tagging_loss=0.009523, over 15678.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09287, pruned_loss=0.0145, audio_tagging_loss=0.009326, over 3059285.15 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:01:51,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2042626.6666666667, ans=0.125
2023-11-22 18:01:57,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306400
2023-11-22 18:02:30,602 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.85 vs. limit=15.0
2023-11-22 18:02:31,135 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.101e+01 8.686e+01 9.623e+01 1.169e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-22 18:02:32,992 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.05 vs. limit=6.0
2023-11-22 18:02:56,976 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5850, loss[loss=0.07208, simple_loss=0.09891, pruned_loss=0.01438, audio_tagging_loss=0.008242, over 15443.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09195, pruned_loss=0.01437, audio_tagging_loss=0.009219, over 3053600.13 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:03:02,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306450
2023-11-22 18:03:24,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2043093.3333333333, ans=0.2
2023-11-22 18:03:24,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2043093.3333333333, ans=0.125
2023-11-22 18:03:54,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2043226.6666666667, ans=0.1
2023-11-22 18:04:06,859 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5900, loss[loss=0.0866, simple_loss=0.126, pruned_loss=0.01604, audio_tagging_loss=0.007543, over 17289.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.0918, pruned_loss=0.01444, audio_tagging_loss=0.009263, over 3054428.13 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:04:07,772 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.27 vs. limit=15.0
2023-11-22 18:04:13,918 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306500
2023-11-22 18:04:24,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2043360.0, ans=0.1
2023-11-22 18:04:31,072 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=12.0
2023-11-22 18:04:58,469 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.49 vs. limit=6.0
2023-11-22 18:05:02,417 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.655e+01 8.183e+01 8.699e+01 9.524e+01 1.128e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 18:05:15,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2043493.3333333333, ans=0.1
2023-11-22 18:05:31,700 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.15 vs. limit=15.0
2023-11-22 18:05:34,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2043626.6666666667, ans=0.0
2023-11-22 18:05:35,773 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 5950, loss[loss=0.1019, simple_loss=0.1422, pruned_loss=0.02495, audio_tagging_loss=0.005812, over 15224.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.0926, pruned_loss=0.01451, audio_tagging_loss=0.009172, over 3058633.97 frames. ], batch size: 54, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:05:43,019 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306550
2023-11-22 18:05:47,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=2043626.6666666667, ans=22.5
2023-11-22 18:06:10,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2043760.0, ans=0.0
2023-11-22 18:06:15,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2043760.0, ans=0.125
2023-11-22 18:07:05,376 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6000, loss[loss=0.07762, simple_loss=0.09506, pruned_loss=0.01866, audio_tagging_loss=0.01144, over 15864.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09292, pruned_loss=0.01448, audio_tagging_loss=0.00923, over 3058308.78 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:07:05,386 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 18:07:32,834 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4442, 3.8521, 2.6304, 3.6093], device='cuda:0')
2023-11-22 18:07:34,031 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([6.0102, 5.8852, 5.6414, 5.6000], device='cuda:0')
2023-11-22 18:07:56,211 INFO [train_asr.py:1253] (0/4) Epoch 26, validation: loss=0.05819, simple_loss=0.05149, pruned_loss=0.005105, audio_tagging_loss=0.02734, over 4681554.00 frames. 
2023-11-22 18:07:56,212 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 18:08:01,701 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306600
2023-11-22 18:08:10,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2044026.6666666667, ans=0.0
2023-11-22 18:08:28,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2044093.3333333333, ans=0.125
2023-11-22 18:08:36,922 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.220e+01 8.809e+01 9.457e+01 1.359e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 18:08:37,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2044160.0, ans=0.09899494936611666
2023-11-22 18:08:43,050 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 18:08:49,913 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.54 vs. limit=15.0
2023-11-22 18:09:01,891 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6050, loss[loss=0.0563, simple_loss=0.07363, pruned_loss=0.01013, audio_tagging_loss=0.009358, over 14858.00 frames. ], tot_loss[loss=0.07034, simple_loss=0.09334, pruned_loss=0.01451, audio_tagging_loss=0.009155, over 3058271.65 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:09:06,909 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306650
2023-11-22 18:09:08,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2044293.3333333333, ans=0.1
2023-11-22 18:09:17,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2044360.0, ans=0.04949747468305833
2023-11-22 18:09:35,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2044426.6666666667, ans=0.125
2023-11-22 18:10:02,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2044560.0, ans=0.5
2023-11-22 18:10:04,708 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6100, loss[loss=0.07785, simple_loss=0.09992, pruned_loss=0.01874, audio_tagging_loss=0.009151, over 15313.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09424, pruned_loss=0.01474, audio_tagging_loss=0.009032, over 3060824.84 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:10:10,235 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306700
2023-11-22 18:10:19,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2044693.3333333333, ans=0.2
2023-11-22 18:10:46,189 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.417e+01 8.381e+01 8.850e+01 9.348e+01 1.200e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 18:11:08,987 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6150, loss[loss=0.07137, simple_loss=0.1012, pruned_loss=0.01283, audio_tagging_loss=0.007943, over 16356.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09375, pruned_loss=0.01458, audio_tagging_loss=0.009032, over 3055874.16 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:11:11,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2044960.0, ans=0.125
2023-11-22 18:11:11,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2044960.0, ans=0.1
2023-11-22 18:11:13,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306750
2023-11-22 18:11:49,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.51 vs. limit=15.0
2023-11-22 18:11:54,284 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:11:57,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2045160.0, ans=0.125
2023-11-22 18:12:03,107 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.30 vs. limit=22.5
2023-11-22 18:12:13,104 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6200, loss[loss=0.06856, simple_loss=0.09347, pruned_loss=0.01262, audio_tagging_loss=0.009207, over 15198.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09349, pruned_loss=0.01459, audio_tagging_loss=0.009109, over 3056228.43 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:12:18,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306800
2023-11-22 18:12:22,159 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.85 vs. limit=22.5
2023-11-22 18:12:23,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2045293.3333333333, ans=0.125
2023-11-22 18:12:41,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2045426.6666666667, ans=10.0
2023-11-22 18:12:48,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.94 vs. limit=15.0
2023-11-22 18:12:53,572 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.167e+01 8.049e+01 8.588e+01 9.515e+01 1.238e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-22 18:13:07,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2045560.0, ans=0.0
2023-11-22 18:13:07,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2045560.0, ans=0.1
2023-11-22 18:13:17,049 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6250, loss[loss=0.07226, simple_loss=0.08972, pruned_loss=0.01497, audio_tagging_loss=0.01244, over 15188.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09395, pruned_loss=0.01487, audio_tagging_loss=0.009203, over 3054748.92 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:13:22,089 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306850
2023-11-22 18:13:32,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2045693.3333333333, ans=0.1
2023-11-22 18:13:41,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2045760.0, ans=0.1
2023-11-22 18:14:12,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2045893.3333333333, ans=0.125
2023-11-22 18:14:17,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2045893.3333333333, ans=0.0
2023-11-22 18:14:21,631 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6300, loss[loss=0.06505, simple_loss=0.07968, pruned_loss=0.01163, audio_tagging_loss=0.01358, over 15710.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09272, pruned_loss=0.01463, audio_tagging_loss=0.009424, over 3057194.41 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:14:25,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2045960.0, ans=0.2
2023-11-22 18:14:26,534 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306900
2023-11-22 18:14:28,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2045960.0, ans=0.125
2023-11-22 18:14:42,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2046026.6666666667, ans=0.1
2023-11-22 18:14:50,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2046093.3333333333, ans=0.125
2023-11-22 18:15:02,392 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.359e+01 8.414e+01 9.060e+01 9.921e+01 1.444e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-22 18:15:09,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2046160.0, ans=0.125
2023-11-22 18:15:14,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2046226.6666666667, ans=0.04949747468305833
2023-11-22 18:15:19,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2046226.6666666667, ans=0.0
2023-11-22 18:15:25,746 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6350, loss[loss=0.05147, simple_loss=0.06525, pruned_loss=0.00744, audio_tagging_loss=0.0114, over 15981.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09284, pruned_loss=0.01452, audio_tagging_loss=0.009501, over 3056090.49 frames. ], batch size: 62, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:15:31,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 306950
2023-11-22 18:15:48,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2046360.0, ans=0.1
2023-11-22 18:15:59,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2046426.6666666667, ans=0.1
2023-11-22 18:16:03,313 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.02 vs. limit=22.5
2023-11-22 18:16:18,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2046560.0, ans=0.0
2023-11-22 18:16:29,687 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6400, loss[loss=0.06818, simple_loss=0.09398, pruned_loss=0.01207, audio_tagging_loss=0.009116, over 15373.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09305, pruned_loss=0.01457, audio_tagging_loss=0.009495, over 3051081.30 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:16:34,586 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307000
2023-11-22 18:16:41,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2046693.3333333333, ans=0.125
2023-11-22 18:16:48,354 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.32 vs. limit=15.0
2023-11-22 18:16:55,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2046760.0, ans=0.0
2023-11-22 18:17:06,907 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.78 vs. limit=15.0
2023-11-22 18:17:10,999 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.820e+01 8.418e+01 9.159e+01 1.026e+02 1.241e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-22 18:17:24,268 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.54 vs. limit=15.0
2023-11-22 18:17:33,068 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6450, loss[loss=0.0755, simple_loss=0.09578, pruned_loss=0.01835, audio_tagging_loss=0.00925, over 15399.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09249, pruned_loss=0.01472, audio_tagging_loss=0.009535, over 3044234.84 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:17:34,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2046960.0, ans=0.0
2023-11-22 18:17:38,680 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307050
2023-11-22 18:18:01,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2047093.3333333333, ans=0.0
2023-11-22 18:18:09,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2047093.3333333333, ans=0.125
2023-11-22 18:18:14,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2047160.0, ans=0.0
2023-11-22 18:18:24,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-22 18:18:31,382 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.02 vs. limit=10.0
2023-11-22 18:18:36,816 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6500, loss[loss=0.0739, simple_loss=0.09507, pruned_loss=0.01921, audio_tagging_loss=0.007148, over 15717.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09276, pruned_loss=0.0147, audio_tagging_loss=0.009553, over 3042799.93 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:18:42,972 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307100
2023-11-22 18:19:19,260 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.186e+01 8.231e+01 8.746e+01 9.661e+01 1.657e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 18:19:30,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2047560.0, ans=0.0
2023-11-22 18:19:41,394 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6550, loss[loss=0.06005, simple_loss=0.07057, pruned_loss=0.01353, audio_tagging_loss=0.01124, over 15535.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.0922, pruned_loss=0.01461, audio_tagging_loss=0.009413, over 3053040.25 frames. ], batch size: 59, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:19:44,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2047626.6666666667, ans=0.125
2023-11-22 18:19:46,413 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307150
2023-11-22 18:19:47,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2047626.6666666667, ans=0.0
2023-11-22 18:19:49,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2047626.6666666667, ans=0.125
2023-11-22 18:20:39,578 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.14 vs. limit=10.0
2023-11-22 18:20:44,975 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6600, loss[loss=0.05578, simple_loss=0.07073, pruned_loss=0.01281, audio_tagging_loss=0.007606, over 16024.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09269, pruned_loss=0.01458, audio_tagging_loss=0.00924, over 3050636.94 frames. ], batch size: 62, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:20:50,122 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307200
2023-11-22 18:21:00,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_na.min_abs, batch_count=2048026.6666666667, ans=0.02
2023-11-22 18:21:12,848 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.30 vs. limit=15.0
2023-11-22 18:21:14,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2048093.3333333333, ans=0.0
2023-11-22 18:21:27,583 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.541e+01 8.316e+01 8.739e+01 9.394e+01 1.270e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 18:21:29,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2048160.0, ans=0.0
2023-11-22 18:21:31,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2048160.0, ans=0.0
2023-11-22 18:21:36,571 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:21:49,023 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6650, loss[loss=0.06874, simple_loss=0.0949, pruned_loss=0.01169, audio_tagging_loss=0.009602, over 15791.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09346, pruned_loss=0.01484, audio_tagging_loss=0.009139, over 3047505.79 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:21:54,589 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307250
2023-11-22 18:22:22,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2048426.6666666667, ans=0.125
2023-11-22 18:22:33,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2048493.3333333333, ans=0.1
2023-11-22 18:22:36,237 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.83 vs. limit=22.5
2023-11-22 18:22:53,630 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6700, loss[loss=0.06986, simple_loss=0.07947, pruned_loss=0.01715, audio_tagging_loss=0.01297, over 14428.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09392, pruned_loss=0.01492, audio_tagging_loss=0.009047, over 3052681.34 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:22:56,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2048626.6666666667, ans=0.1
2023-11-22 18:22:58,522 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307300
2023-11-22 18:23:27,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2048760.0, ans=0.125
2023-11-22 18:23:35,469 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.221e+01 9.103e+01 9.902e+01 1.404e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-22 18:23:37,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2048826.6666666667, ans=0.1
2023-11-22 18:23:56,361 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6750, loss[loss=0.07287, simple_loss=0.08729, pruned_loss=0.0192, audio_tagging_loss=0.01003, over 15907.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09399, pruned_loss=0.0149, audio_tagging_loss=0.009116, over 3053253.33 frames. ], batch size: 60, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:24:00,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2048960.0, ans=0.0
2023-11-22 18:24:01,400 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307350
2023-11-22 18:24:17,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2049026.6666666667, ans=0.05
2023-11-22 18:24:17,429 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:24:22,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2049093.3333333333, ans=0.125
2023-11-22 18:24:24,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2049093.3333333333, ans=0.0
2023-11-22 18:24:36,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2049160.0, ans=0.0
2023-11-22 18:24:59,207 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6800, loss[loss=0.07156, simple_loss=0.09484, pruned_loss=0.01589, audio_tagging_loss=0.00825, over 16599.00 frames. ], tot_loss[loss=0.07122, simple_loss=0.09439, pruned_loss=0.01498, audio_tagging_loss=0.009048, over 3052937.42 frames. ], batch size: 63, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:25:04,180 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307400
2023-11-22 18:25:40,520 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.331e+01 8.948e+01 9.524e+01 1.247e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 18:25:46,316 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.06 vs. limit=15.0
2023-11-22 18:25:50,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2049560.0, ans=0.0
2023-11-22 18:25:53,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2049560.0, ans=0.125
2023-11-22 18:26:03,647 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6850, loss[loss=0.06855, simple_loss=0.09163, pruned_loss=0.01374, audio_tagging_loss=0.008997, over 15266.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09405, pruned_loss=0.01472, audio_tagging_loss=0.009001, over 3051717.19 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:26:03,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2049626.6666666667, ans=0.1
2023-11-22 18:26:08,449 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307450
2023-11-22 18:26:30,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2049760.0, ans=0.0
2023-11-22 18:27:03,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2049893.3333333333, ans=0.0
2023-11-22 18:27:06,596 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6900, loss[loss=0.07125, simple_loss=0.09655, pruned_loss=0.01594, audio_tagging_loss=0.007034, over 15135.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09368, pruned_loss=0.01464, audio_tagging_loss=0.009055, over 3050312.27 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:27:11,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307500
2023-11-22 18:27:25,930 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.03 vs. limit=22.5
2023-11-22 18:27:45,373 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.02 vs. limit=6.0
2023-11-22 18:27:49,689 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.175e+01 8.211e+01 8.819e+01 9.704e+01 1.350e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 18:27:51,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2050160.0, ans=0.125
2023-11-22 18:27:56,105 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 18:27:56,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2050160.0, ans=0.125
2023-11-22 18:28:07,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2050226.6666666667, ans=0.125
2023-11-22 18:28:11,159 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 6950, loss[loss=0.07459, simple_loss=0.1006, pruned_loss=0.01458, audio_tagging_loss=0.009691, over 14435.00 frames. ], tot_loss[loss=0.07104, simple_loss=0.0944, pruned_loss=0.01482, audio_tagging_loss=0.009022, over 3044585.04 frames. ], batch size: 56, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:28:13,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2050293.3333333333, ans=0.125
2023-11-22 18:28:16,399 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307550
2023-11-22 18:28:27,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.29 vs. limit=15.0
2023-11-22 18:28:40,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2050426.6666666667, ans=0.0
2023-11-22 18:28:51,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2050493.3333333333, ans=0.05
2023-11-22 18:28:51,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2050493.3333333333, ans=0.1
2023-11-22 18:28:54,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2050493.3333333333, ans=0.07
2023-11-22 18:29:10,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2050560.0, ans=0.0
2023-11-22 18:29:12,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2050560.0, ans=0.125
2023-11-22 18:29:17,603 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7000, loss[loss=0.06268, simple_loss=0.08034, pruned_loss=0.0103, audio_tagging_loss=0.01221, over 16716.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09353, pruned_loss=0.01464, audio_tagging_loss=0.009099, over 3043506.97 frames. ], batch size: 65, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:29:17,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2050626.6666666667, ans=0.125
2023-11-22 18:29:23,233 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307600
2023-11-22 18:29:42,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2050760.0, ans=0.125
2023-11-22 18:29:59,744 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.823e+01 8.302e+01 8.887e+01 9.651e+01 1.249e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 18:30:14,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2050893.3333333333, ans=0.0
2023-11-22 18:30:22,295 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7050, loss[loss=0.06823, simple_loss=0.08608, pruned_loss=0.01258, audio_tagging_loss=0.01261, over 14339.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09277, pruned_loss=0.01436, audio_tagging_loss=0.009203, over 3048769.33 frames. ], batch size: 55, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:30:27,201 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307650
2023-11-22 18:30:51,305 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:30:52,035 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.98 vs. limit=15.0
2023-11-22 18:31:16,186 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:31:17,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2051226.6666666667, ans=0.125
2023-11-22 18:31:27,268 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7100, loss[loss=0.06572, simple_loss=0.08842, pruned_loss=0.0119, audio_tagging_loss=0.009603, over 15760.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09282, pruned_loss=0.01435, audio_tagging_loss=0.009314, over 3050860.53 frames. ], batch size: 58, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:31:32,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307700
2023-11-22 18:31:46,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2051360.0, ans=0.2
2023-11-22 18:32:11,639 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.336e+01 9.020e+01 9.845e+01 2.040e+02, threshold=1.804e+02, percent-clipped=1.0
2023-11-22 18:32:30,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2051560.0, ans=0.125
2023-11-22 18:32:32,736 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7150, loss[loss=0.06787, simple_loss=0.09749, pruned_loss=0.01101, audio_tagging_loss=0.008116, over 15946.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09264, pruned_loss=0.01439, audio_tagging_loss=0.009341, over 3049246.95 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 16.0
2023-11-22 18:32:36,116 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:32:38,306 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307750
2023-11-22 18:32:40,130 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.08 vs. limit=15.0
2023-11-22 18:32:40,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2051626.6666666667, ans=0.0
2023-11-22 18:32:50,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2051693.3333333333, ans=0.0
2023-11-22 18:33:11,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2051826.6666666667, ans=0.125
2023-11-22 18:33:14,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2051826.6666666667, ans=0.1
2023-11-22 18:33:32,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2051893.3333333333, ans=0.125
2023-11-22 18:33:36,863 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7200, loss[loss=0.07842, simple_loss=0.09566, pruned_loss=0.02055, audio_tagging_loss=0.01004, over 15062.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09349, pruned_loss=0.01462, audio_tagging_loss=0.009379, over 3047553.10 frames. ], batch size: 57, lr: 2.62e-03, grad_scale: 32.0
2023-11-22 18:33:37,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2051960.0, ans=0.125
2023-11-22 18:33:39,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2051960.0, ans=0.07
2023-11-22 18:33:42,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307800
2023-11-22 18:34:01,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2052093.3333333333, ans=0.1
2023-11-22 18:34:04,009 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.72 vs. limit=15.0
2023-11-22 18:34:17,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2052160.0, ans=0.1
2023-11-22 18:34:20,788 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.282e+01 8.868e+01 9.678e+01 1.341e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 18:34:35,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2052226.6666666667, ans=0.0
2023-11-22 18:34:40,981 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7250, loss[loss=0.07429, simple_loss=0.09318, pruned_loss=0.01518, audio_tagging_loss=0.01252, over 14860.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09324, pruned_loss=0.01441, audio_tagging_loss=0.009443, over 3039085.97 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:34:46,168 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307850
2023-11-22 18:35:05,193 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:35:12,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2052426.6666666667, ans=0.0
2023-11-22 18:35:12,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2052426.6666666667, ans=0.07
2023-11-22 18:35:16,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2052426.6666666667, ans=0.1
2023-11-22 18:35:19,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2052493.3333333333, ans=0.1
2023-11-22 18:35:23,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2052493.3333333333, ans=0.0
2023-11-22 18:35:31,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2052560.0, ans=0.0
2023-11-22 18:35:36,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2052560.0, ans=0.125
2023-11-22 18:35:45,806 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7300, loss[loss=0.05939, simple_loss=0.08373, pruned_loss=0.008978, audio_tagging_loss=0.008548, over 15511.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.09294, pruned_loss=0.0143, audio_tagging_loss=0.009353, over 3043823.48 frames. ], batch size: 60, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:35:50,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2052626.6666666667, ans=0.125
2023-11-22 18:35:51,261 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307900
2023-11-22 18:35:59,952 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:36:19,980 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:36:28,168 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.001e+01 8.613e+01 9.353e+01 1.295e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-22 18:36:49,247 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7350, loss[loss=0.05951, simple_loss=0.07599, pruned_loss=0.01175, audio_tagging_loss=0.009761, over 14999.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09282, pruned_loss=0.01446, audio_tagging_loss=0.009137, over 3046640.31 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:36:54,158 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 307950
2023-11-22 18:37:17,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2053093.3333333333, ans=0.1
2023-11-22 18:37:29,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2053160.0, ans=0.125
2023-11-22 18:37:37,386 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.99 vs. limit=15.0
2023-11-22 18:37:53,619 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7400, loss[loss=0.06768, simple_loss=0.09465, pruned_loss=0.01275, audio_tagging_loss=0.007597, over 15547.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09333, pruned_loss=0.0145, audio_tagging_loss=0.009117, over 3051226.84 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:37:58,539 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308000
2023-11-22 18:38:00,099 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-308000.pt
2023-11-22 18:38:12,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2053360.0, ans=0.0
2023-11-22 18:38:21,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2053360.0, ans=0.125
2023-11-22 18:38:41,005 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.692e+01 8.189e+01 8.818e+01 9.680e+01 1.265e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 18:38:43,688 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.07 vs. limit=12.0
2023-11-22 18:38:45,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2053493.3333333333, ans=0.0
2023-11-22 18:39:02,139 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7450, loss[loss=0.07954, simple_loss=0.1101, pruned_loss=0.01691, audio_tagging_loss=0.00757, over 15955.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09346, pruned_loss=0.01449, audio_tagging_loss=0.009106, over 3048292.82 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:39:08,389 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308050
2023-11-22 18:39:08,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2053626.6666666667, ans=0.0
2023-11-22 18:40:07,086 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7500, loss[loss=0.05724, simple_loss=0.07198, pruned_loss=0.01104, audio_tagging_loss=0.01021, over 13398.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09273, pruned_loss=0.01438, audio_tagging_loss=0.009142, over 3041719.29 frames. ], batch size: 54, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:40:12,161 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308100
2023-11-22 18:40:19,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2054026.6666666667, ans=0.0
2023-11-22 18:40:52,215 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.656e+01 8.267e+01 8.775e+01 9.244e+01 1.175e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-22 18:41:10,631 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7550, loss[loss=0.03485, simple_loss=0.03378, pruned_loss=0.004579, audio_tagging_loss=0.01338, over 16329.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09255, pruned_loss=0.01445, audio_tagging_loss=0.009039, over 3042536.61 frames. ], batch size: 65, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:41:16,134 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308150
2023-11-22 18:41:26,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_na.min_abs, batch_count=2054360.0, ans=0.02
2023-11-22 18:41:50,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2054493.3333333333, ans=0.04949747468305833
2023-11-22 18:41:51,362 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.88 vs. limit=22.5
2023-11-22 18:41:57,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2054493.3333333333, ans=0.125
2023-11-22 18:42:12,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2054560.0, ans=0.0
2023-11-22 18:42:12,676 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.48 vs. limit=15.0
2023-11-22 18:42:13,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2054626.6666666667, ans=0.0
2023-11-22 18:42:14,489 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7600, loss[loss=0.0716, simple_loss=0.09338, pruned_loss=0.01431, audio_tagging_loss=0.0106, over 15301.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09224, pruned_loss=0.01446, audio_tagging_loss=0.009081, over 3041489.97 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:42:20,773 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308200
2023-11-22 18:42:30,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2054693.3333333333, ans=0.0
2023-11-22 18:42:38,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2054693.3333333333, ans=10.0
2023-11-22 18:42:59,730 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 7.973e+01 8.478e+01 9.439e+01 1.214e+02, threshold=1.696e+02, percent-clipped=0.0
2023-11-22 18:43:04,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2054893.3333333333, ans=0.0
2023-11-22 18:43:13,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.85 vs. limit=6.0
2023-11-22 18:43:14,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2054893.3333333333, ans=0.125
2023-11-22 18:43:20,424 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7650, loss[loss=0.07882, simple_loss=0.09806, pruned_loss=0.02036, audio_tagging_loss=0.009426, over 14714.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09213, pruned_loss=0.01448, audio_tagging_loss=0.009111, over 3045772.30 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:43:25,698 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308250
2023-11-22 18:44:08,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2055160.0, ans=0.1
2023-11-22 18:44:12,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2055226.6666666667, ans=10.0
2023-11-22 18:44:13,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2055226.6666666667, ans=0.0
2023-11-22 18:44:17,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2055226.6666666667, ans=0.125
2023-11-22 18:44:23,957 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.96 vs. limit=15.0
2023-11-22 18:44:24,543 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7700, loss[loss=0.06847, simple_loss=0.09265, pruned_loss=0.01286, audio_tagging_loss=0.009286, over 15445.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09251, pruned_loss=0.01447, audio_tagging_loss=0.009147, over 3046125.19 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:44:29,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308300
2023-11-22 18:44:30,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.47 vs. limit=10.0
2023-11-22 18:44:49,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2055360.0, ans=0.125
2023-11-22 18:44:49,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.51 vs. limit=15.0
2023-11-22 18:45:10,224 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.353e+01 9.009e+01 9.552e+01 1.417e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-22 18:45:14,752 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.33 vs. limit=15.0
2023-11-22 18:45:21,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2055560.0, ans=0.125
2023-11-22 18:45:29,300 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7750, loss[loss=0.06071, simple_loss=0.06989, pruned_loss=0.0145, audio_tagging_loss=0.01126, over 15098.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09292, pruned_loss=0.01466, audio_tagging_loss=0.009245, over 3044885.16 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:45:34,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308350
2023-11-22 18:45:39,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2055626.6666666667, ans=0.125
2023-11-22 18:45:46,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2055693.3333333333, ans=0.125
2023-11-22 18:46:15,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2055826.6666666667, ans=0.125
2023-11-22 18:46:15,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2055826.6666666667, ans=0.0
2023-11-22 18:46:15,338 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.00 vs. limit=22.5
2023-11-22 18:46:31,556 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.51 vs. limit=15.0
2023-11-22 18:46:34,443 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7800, loss[loss=0.0676, simple_loss=0.09724, pruned_loss=0.01136, audio_tagging_loss=0.007619, over 15860.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09365, pruned_loss=0.01487, audio_tagging_loss=0.009158, over 3045879.86 frames. ], batch size: 60, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:46:39,336 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308400
2023-11-22 18:46:44,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2055960.0, ans=0.2
2023-11-22 18:46:49,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2056026.6666666667, ans=0.125
2023-11-22 18:47:02,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2056093.3333333333, ans=0.0
2023-11-22 18:47:04,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2056093.3333333333, ans=0.5
2023-11-22 18:47:21,063 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.003e+01 8.337e+01 8.829e+01 9.514e+01 1.389e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 18:47:30,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.75 vs. limit=15.0
2023-11-22 18:47:33,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2056226.6666666667, ans=0.125
2023-11-22 18:47:38,096 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7850, loss[loss=0.07874, simple_loss=0.1088, pruned_loss=0.01617, audio_tagging_loss=0.008153, over 15733.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09395, pruned_loss=0.01508, audio_tagging_loss=0.009258, over 3050644.76 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:47:43,173 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308450
2023-11-22 18:47:57,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2056360.0, ans=0.125
2023-11-22 18:48:12,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2056426.6666666667, ans=0.2
2023-11-22 18:48:24,001 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.63 vs. limit=22.5
2023-11-22 18:48:33,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2056560.0, ans=0.0
2023-11-22 18:48:41,460 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7900, loss[loss=0.07935, simple_loss=0.1057, pruned_loss=0.01878, audio_tagging_loss=0.007742, over 15699.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09397, pruned_loss=0.01501, audio_tagging_loss=0.009298, over 3045321.69 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:48:46,520 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308500
2023-11-22 18:49:20,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2056826.6666666667, ans=0.125
2023-11-22 18:49:27,229 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.747e+01 8.185e+01 8.852e+01 9.450e+01 1.416e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-22 18:49:46,637 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 7950, loss[loss=0.05703, simple_loss=0.07707, pruned_loss=0.008891, audio_tagging_loss=0.009604, over 15398.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09327, pruned_loss=0.01474, audio_tagging_loss=0.009612, over 3049927.70 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:49:51,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308550
2023-11-22 18:50:00,023 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 18:50:15,206 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.88 vs. limit=15.0
2023-11-22 18:50:18,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2057093.3333333333, ans=0.125
2023-11-22 18:50:27,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2057160.0, ans=0.0
2023-11-22 18:50:37,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2057226.6666666667, ans=0.2
2023-11-22 18:50:38,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2057226.6666666667, ans=0.0
2023-11-22 18:50:45,655 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.20 vs. limit=10.0
2023-11-22 18:50:49,794 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8000, loss[loss=0.07044, simple_loss=0.08963, pruned_loss=0.01502, audio_tagging_loss=0.01061, over 15347.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09254, pruned_loss=0.01477, audio_tagging_loss=0.009644, over 3047489.78 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:50:54,929 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308600
2023-11-22 18:51:10,452 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.64 vs. limit=15.0
2023-11-22 18:51:36,566 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.984e+01 8.209e+01 8.705e+01 9.523e+01 1.226e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 18:51:53,700 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8050, loss[loss=0.05006, simple_loss=0.06327, pruned_loss=0.008903, audio_tagging_loss=0.009519, over 14401.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09312, pruned_loss=0.01482, audio_tagging_loss=0.009612, over 3047613.65 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:51:58,736 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308650
2023-11-22 18:52:23,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2057760.0, ans=0.1
2023-11-22 18:52:25,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2057760.0, ans=0.125
2023-11-22 18:52:30,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2057760.0, ans=0.0
2023-11-22 18:52:32,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.65 vs. limit=10.0
2023-11-22 18:52:32,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2057826.6666666667, ans=0.0
2023-11-22 18:52:34,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2057826.6666666667, ans=0.0
2023-11-22 18:52:57,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2057960.0, ans=0.125
2023-11-22 18:52:59,061 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8100, loss[loss=0.05431, simple_loss=0.0702, pruned_loss=0.01071, audio_tagging_loss=0.008498, over 14209.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09288, pruned_loss=0.0148, audio_tagging_loss=0.009504, over 3042958.16 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:53:04,555 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308700
2023-11-22 18:53:05,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2057960.0, ans=0.1
2023-11-22 18:53:35,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2058160.0, ans=0.125
2023-11-22 18:53:45,846 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.339e+01 8.206e+01 8.841e+01 9.625e+01 1.237e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-22 18:53:58,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2058226.6666666667, ans=0.09899494936611666
2023-11-22 18:54:03,384 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8150, loss[loss=0.07968, simple_loss=0.1099, pruned_loss=0.01823, audio_tagging_loss=0.006512, over 15230.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09355, pruned_loss=0.01493, audio_tagging_loss=0.009225, over 3047360.84 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:54:08,437 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308750
2023-11-22 18:54:24,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2058360.0, ans=0.125
2023-11-22 18:54:27,369 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.43 vs. limit=15.0
2023-11-22 18:55:07,331 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8200, loss[loss=0.07724, simple_loss=0.1054, pruned_loss=0.01442, audio_tagging_loss=0.01012, over 14701.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09369, pruned_loss=0.01491, audio_tagging_loss=0.00913, over 3053010.99 frames. ], batch size: 54, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:55:07,350 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 18:55:12,437 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308800
2023-11-22 18:55:19,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2058693.3333333333, ans=0.125
2023-11-22 18:55:24,720 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.07 vs. limit=15.0
2023-11-22 18:55:25,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2058693.3333333333, ans=0.125
2023-11-22 18:55:28,766 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.72 vs. limit=15.0
2023-11-22 18:55:34,808 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.66 vs. limit=22.5
2023-11-22 18:55:45,389 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.15 vs. limit=22.5
2023-11-22 18:55:55,511 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.302e+01 8.890e+01 9.670e+01 1.288e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 18:56:11,869 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8250, loss[loss=0.06532, simple_loss=0.08711, pruned_loss=0.01323, audio_tagging_loss=0.008526, over 15042.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09293, pruned_loss=0.0149, audio_tagging_loss=0.009147, over 3048266.69 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:56:12,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2058960.0, ans=0.125
2023-11-22 18:56:17,382 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308850
2023-11-22 18:56:17,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2058960.0, ans=0.0
2023-11-22 18:56:44,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2059093.3333333333, ans=0.125
2023-11-22 18:56:54,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2059160.0, ans=0.125
2023-11-22 18:57:00,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2059160.0, ans=0.2
2023-11-22 18:57:01,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2059226.6666666667, ans=0.125
2023-11-22 18:57:06,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2059226.6666666667, ans=0.0
2023-11-22 18:57:07,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2059226.6666666667, ans=0.125
2023-11-22 18:57:16,298 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8300, loss[loss=0.07031, simple_loss=0.08868, pruned_loss=0.01728, audio_tagging_loss=0.008695, over 15356.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09261, pruned_loss=0.01475, audio_tagging_loss=0.009151, over 3051457.97 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:57:21,467 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308900
2023-11-22 18:57:30,300 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 18:57:35,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2059360.0, ans=0.2
2023-11-22 18:57:36,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2059360.0, ans=0.1
2023-11-22 18:57:40,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2059426.6666666667, ans=0.2
2023-11-22 18:58:03,825 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.260e+01 8.357e+01 8.920e+01 9.669e+01 1.328e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 18:58:14,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2059560.0, ans=0.0
2023-11-22 18:58:20,141 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8350, loss[loss=0.06309, simple_loss=0.09139, pruned_loss=0.007563, audio_tagging_loss=0.00983, over 15687.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09306, pruned_loss=0.01491, audio_tagging_loss=0.009136, over 3051115.24 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 18:58:25,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 308950
2023-11-22 18:58:50,901 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.43 vs. limit=15.0
2023-11-22 18:59:02,437 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.17 vs. limit=15.0
2023-11-22 18:59:17,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=15.0
2023-11-22 18:59:22,821 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8400, loss[loss=0.05428, simple_loss=0.06396, pruned_loss=0.01078, audio_tagging_loss=0.01152, over 15258.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09269, pruned_loss=0.01479, audio_tagging_loss=0.009095, over 3038765.44 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 18:59:29,001 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309000
2023-11-22 18:59:49,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2060093.3333333333, ans=0.125
2023-11-22 19:00:01,492 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.30 vs. limit=15.0
2023-11-22 19:00:05,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2060160.0, ans=0.0
2023-11-22 19:00:10,225 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.438e+01 8.273e+01 8.789e+01 9.898e+01 1.441e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-22 19:00:10,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2060160.0, ans=0.025
2023-11-22 19:00:15,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.27 vs. limit=15.0
2023-11-22 19:00:28,046 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8450, loss[loss=0.06651, simple_loss=0.08659, pruned_loss=0.0127, audio_tagging_loss=0.01051, over 15178.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09321, pruned_loss=0.01489, audio_tagging_loss=0.009083, over 3044788.04 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:00:32,497 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.90 vs. limit=22.5
2023-11-22 19:00:33,061 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309050
2023-11-22 19:00:34,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2060293.3333333333, ans=0.0
2023-11-22 19:00:37,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.91 vs. limit=10.0
2023-11-22 19:00:38,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2060293.3333333333, ans=0.125
2023-11-22 19:00:43,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2060360.0, ans=0.125
2023-11-22 19:00:57,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2060426.6666666667, ans=0.2
2023-11-22 19:01:01,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2060426.6666666667, ans=0.125
2023-11-22 19:01:03,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2060426.6666666667, ans=0.04949747468305833
2023-11-22 19:01:06,489 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.10 vs. limit=6.0
2023-11-22 19:01:10,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2060493.3333333333, ans=0.125
2023-11-22 19:01:31,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2060626.6666666667, ans=0.07
2023-11-22 19:01:32,001 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8500, loss[loss=0.05656, simple_loss=0.07244, pruned_loss=0.01126, audio_tagging_loss=0.00908, over 14375.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09296, pruned_loss=0.0147, audio_tagging_loss=0.009177, over 3040374.06 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:01:33,726 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.35 vs. limit=15.0
2023-11-22 19:01:36,990 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309100
2023-11-22 19:01:38,850 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.66 vs. limit=6.0
2023-11-22 19:01:38,902 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.52 vs. limit=15.0
2023-11-22 19:02:11,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2060826.6666666667, ans=0.2
2023-11-22 19:02:16,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2060826.6666666667, ans=0.09899494936611666
2023-11-22 19:02:19,382 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.798e+01 8.300e+01 8.864e+01 9.381e+01 1.141e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 19:02:30,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2060893.3333333333, ans=0.07
2023-11-22 19:02:35,143 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8550, loss[loss=0.06902, simple_loss=0.09535, pruned_loss=0.01096, audio_tagging_loss=0.01038, over 15071.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09256, pruned_loss=0.01443, audio_tagging_loss=0.009177, over 3042259.34 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:02:39,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2060960.0, ans=0.95
2023-11-22 19:02:41,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309150
2023-11-22 19:02:50,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2061026.6666666667, ans=0.0
2023-11-22 19:02:55,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2061026.6666666667, ans=0.125
2023-11-22 19:03:09,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2061093.3333333333, ans=0.09899494936611666
2023-11-22 19:03:12,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2061093.3333333333, ans=0.125
2023-11-22 19:03:22,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2061160.0, ans=0.1
2023-11-22 19:03:40,638 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8600, loss[loss=0.08967, simple_loss=0.1216, pruned_loss=0.02253, audio_tagging_loss=0.006345, over 15938.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09315, pruned_loss=0.01451, audio_tagging_loss=0.009209, over 3051824.80 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:03:45,530 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309200
2023-11-22 19:03:57,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2061360.0, ans=0.0
2023-11-22 19:03:57,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2061360.0, ans=0.1
2023-11-22 19:04:10,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.98 vs. limit=15.0
2023-11-22 19:04:13,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2061426.6666666667, ans=0.0
2023-11-22 19:04:27,934 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.159e+01 8.278e+01 9.087e+01 9.783e+01 1.157e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-22 19:04:28,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2061493.3333333333, ans=0.125
2023-11-22 19:04:34,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2061560.0, ans=0.1
2023-11-22 19:04:42,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2061626.6666666667, ans=0.0
2023-11-22 19:04:43,830 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8650, loss[loss=0.06882, simple_loss=0.09174, pruned_loss=0.01431, audio_tagging_loss=0.008636, over 15439.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09282, pruned_loss=0.01444, audio_tagging_loss=0.009285, over 3053468.05 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:04:49,426 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309250
2023-11-22 19:05:05,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2061693.3333333333, ans=10.0
2023-11-22 19:05:30,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.23 vs. limit=15.0
2023-11-22 19:05:36,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2061893.3333333333, ans=0.1
2023-11-22 19:05:37,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2061893.3333333333, ans=0.0
2023-11-22 19:05:47,816 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8700, loss[loss=0.08058, simple_loss=0.1145, pruned_loss=0.01704, audio_tagging_loss=0.006291, over 15178.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09344, pruned_loss=0.01454, audio_tagging_loss=0.00931, over 3054729.04 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:05:53,022 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309300
2023-11-22 19:05:53,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2061960.0, ans=0.125
2023-11-22 19:05:57,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2061960.0, ans=0.125
2023-11-22 19:06:34,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2062160.0, ans=0.0
2023-11-22 19:06:36,750 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.774e+01 8.375e+01 8.886e+01 9.746e+01 1.312e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 19:06:49,177 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=2.531e-03
2023-11-22 19:06:50,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2062226.6666666667, ans=0.1
2023-11-22 19:06:52,461 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8750, loss[loss=0.05449, simple_loss=0.06454, pruned_loss=0.01055, audio_tagging_loss=0.01167, over 14300.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09368, pruned_loss=0.01458, audio_tagging_loss=0.009357, over 3048289.51 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:06:56,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2062293.3333333333, ans=0.025
2023-11-22 19:06:57,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309350
2023-11-22 19:07:55,115 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8800, loss[loss=0.05379, simple_loss=0.06425, pruned_loss=0.01093, audio_tagging_loss=0.01074, over 16522.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09431, pruned_loss=0.01482, audio_tagging_loss=0.009401, over 3052315.51 frames. ], batch size: 61, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:07:57,248 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.88 vs. limit=22.5
2023-11-22 19:08:00,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309400
2023-11-22 19:08:09,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2062693.3333333333, ans=0.125
2023-11-22 19:08:14,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2062693.3333333333, ans=0.0
2023-11-22 19:08:43,647 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.328e+01 8.323e+01 8.895e+01 9.421e+01 1.402e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 19:08:58,983 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8850, loss[loss=0.0597, simple_loss=0.07092, pruned_loss=0.009842, audio_tagging_loss=0.01439, over 15395.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.09453, pruned_loss=0.01493, audio_tagging_loss=0.009418, over 3053946.76 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:09:03,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309450
2023-11-22 19:09:03,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2062960.0, ans=0.0
2023-11-22 19:09:10,422 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:09:20,870 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:09:27,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2063093.3333333333, ans=0.1
2023-11-22 19:09:30,105 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.13 vs. limit=10.0
2023-11-22 19:09:47,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2063160.0, ans=0.125
2023-11-22 19:09:51,589 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.37 vs. limit=22.5
2023-11-22 19:09:53,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2063226.6666666667, ans=0.125
2023-11-22 19:10:02,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2063293.3333333333, ans=0.125
2023-11-22 19:10:02,972 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8900, loss[loss=0.073, simple_loss=0.09206, pruned_loss=0.01785, audio_tagging_loss=0.009111, over 15596.00 frames. ], tot_loss[loss=0.07175, simple_loss=0.09496, pruned_loss=0.01506, audio_tagging_loss=0.009216, over 3056998.52 frames. ], batch size: 58, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:10:03,826 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.37 vs. limit=15.0
2023-11-22 19:10:04,790 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.43 vs. limit=22.5
2023-11-22 19:10:07,933 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309500
2023-11-22 19:10:12,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2063293.3333333333, ans=0.125
2023-11-22 19:10:14,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2063360.0, ans=0.125
2023-11-22 19:10:23,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2063360.0, ans=0.125
2023-11-22 19:10:50,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2063493.3333333333, ans=0.0
2023-11-22 19:10:50,973 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 8.152e+01 8.729e+01 9.572e+01 1.153e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 19:11:05,639 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 8950, loss[loss=0.07966, simple_loss=0.1126, pruned_loss=0.01524, audio_tagging_loss=0.008116, over 15973.00 frames. ], tot_loss[loss=0.07168, simple_loss=0.09508, pruned_loss=0.01508, audio_tagging_loss=0.009064, over 3059516.33 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:11:07,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2063626.6666666667, ans=0.125
2023-11-22 19:11:10,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309550
2023-11-22 19:11:30,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2063760.0, ans=0.125
2023-11-22 19:12:07,983 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9000, loss[loss=0.07383, simple_loss=0.1017, pruned_loss=0.01465, audio_tagging_loss=0.008311, over 15435.00 frames. ], tot_loss[loss=0.07199, simple_loss=0.0957, pruned_loss=0.01517, audio_tagging_loss=0.008975, over 3061312.00 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:12:07,986 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 19:12:30,226 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9101, 3.7076, 4.9273, 4.3155], device='cuda:0')
2023-11-22 19:12:36,378 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.8052, 1.4405, 3.5329, 3.0612, 2.9995, 3.2560, 2.9891, 3.1862],
       device='cuda:0')
2023-11-22 19:12:42,609 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.0792, 4.0184, 3.8694, 3.1840], device='cuda:0')
2023-11-22 19:12:45,269 INFO [train_asr.py:1253] (0/4) Epoch 26, validation: loss=0.0595, simple_loss=0.05137, pruned_loss=0.00505, audio_tagging_loss=0.02877, over 4681554.00 frames. 
2023-11-22 19:12:45,270 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 19:12:47,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2063960.0, ans=0.125
2023-11-22 19:12:50,192 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309600
2023-11-22 19:12:59,625 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.74 vs. limit=15.0
2023-11-22 19:13:12,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2064093.3333333333, ans=0.0
2023-11-22 19:13:16,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2064093.3333333333, ans=0.125
2023-11-22 19:13:21,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2064160.0, ans=0.1
2023-11-22 19:13:34,773 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.292e+01 8.198e+01 8.954e+01 9.788e+01 1.376e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 19:13:39,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2064226.6666666667, ans=0.125
2023-11-22 19:13:48,332 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9050, loss[loss=0.07592, simple_loss=0.08797, pruned_loss=0.02211, audio_tagging_loss=0.009832, over 15097.00 frames. ], tot_loss[loss=0.07155, simple_loss=0.09503, pruned_loss=0.01505, audio_tagging_loss=0.008988, over 3057301.47 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:13:50,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2064293.3333333333, ans=0.0
2023-11-22 19:13:53,488 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309650
2023-11-22 19:13:59,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2064360.0, ans=0.125
2023-11-22 19:14:04,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2064360.0, ans=0.125
2023-11-22 19:14:08,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2064360.0, ans=0.04949747468305833
2023-11-22 19:14:12,382 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.24 vs. limit=15.0
2023-11-22 19:14:21,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2064426.6666666667, ans=0.125
2023-11-22 19:14:40,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2064560.0, ans=0.2
2023-11-22 19:14:42,981 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.98 vs. limit=15.0
2023-11-22 19:14:50,833 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9100, loss[loss=0.0522, simple_loss=0.06419, pruned_loss=0.009833, audio_tagging_loss=0.01027, over 14538.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09453, pruned_loss=0.01498, audio_tagging_loss=0.009047, over 3055782.52 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:14:55,766 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309700
2023-11-22 19:14:57,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2064626.6666666667, ans=0.07
2023-11-22 19:14:58,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2064626.6666666667, ans=0.125
2023-11-22 19:15:08,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2064693.3333333333, ans=0.1
2023-11-22 19:15:22,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2064760.0, ans=0.2
2023-11-22 19:15:27,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2064826.6666666667, ans=0.125
2023-11-22 19:15:31,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2064826.6666666667, ans=0.125
2023-11-22 19:15:39,616 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.178e+01 9.169e+01 9.914e+01 1.442e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-22 19:15:55,005 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9150, loss[loss=0.07756, simple_loss=0.09919, pruned_loss=0.01919, audio_tagging_loss=0.008778, over 15100.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09435, pruned_loss=0.01497, audio_tagging_loss=0.009102, over 3054935.67 frames. ], batch size: 56, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:15:59,952 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309750
2023-11-22 19:16:00,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2064960.0, ans=0.1
2023-11-22 19:16:23,213 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:16:41,050 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.07 vs. limit=6.0
2023-11-22 19:16:45,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2065226.6666666667, ans=0.125
2023-11-22 19:16:45,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2065226.6666666667, ans=0.125
2023-11-22 19:16:57,896 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9200, loss[loss=0.07035, simple_loss=0.08324, pruned_loss=0.01745, audio_tagging_loss=0.01128, over 14745.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.094, pruned_loss=0.01474, audio_tagging_loss=0.009087, over 3054950.98 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 32.0
2023-11-22 19:17:02,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309800
2023-11-22 19:17:10,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2065360.0, ans=0.125
2023-11-22 19:17:25,820 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.09 vs. limit=22.5
2023-11-22 19:17:27,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2065426.6666666667, ans=0.1
2023-11-22 19:17:48,104 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.424e+01 8.996e+01 9.911e+01 1.165e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 19:17:53,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2065560.0, ans=0.2
2023-11-22 19:18:00,311 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9250, loss[loss=0.06618, simple_loss=0.09107, pruned_loss=0.01073, audio_tagging_loss=0.009915, over 15297.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09461, pruned_loss=0.01475, audio_tagging_loss=0.009009, over 3057830.47 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:18:00,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2065626.6666666667, ans=0.125
2023-11-22 19:18:00,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2065626.6666666667, ans=10.0
2023-11-22 19:18:05,402 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309850
2023-11-22 19:18:10,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2065626.6666666667, ans=0.035
2023-11-22 19:18:13,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2065693.3333333333, ans=0.125
2023-11-22 19:18:17,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2065693.3333333333, ans=0.125
2023-11-22 19:18:23,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2065693.3333333333, ans=0.0
2023-11-22 19:18:36,057 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.08 vs. limit=15.0
2023-11-22 19:18:37,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.16 vs. limit=15.0
2023-11-22 19:18:40,097 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.77 vs. limit=15.0
2023-11-22 19:18:43,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2065826.6666666667, ans=0.07
2023-11-22 19:18:57,096 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.11 vs. limit=15.0
2023-11-22 19:19:04,414 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9300, loss[loss=0.0733, simple_loss=0.09716, pruned_loss=0.01567, audio_tagging_loss=0.009046, over 15956.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09407, pruned_loss=0.01464, audio_tagging_loss=0.008966, over 3051567.65 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:19:10,653 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309900
2023-11-22 19:19:20,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2066026.6666666667, ans=0.125
2023-11-22 19:19:32,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2066093.3333333333, ans=0.09899494936611666
2023-11-22 19:19:36,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2066093.3333333333, ans=15.0
2023-11-22 19:19:48,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2066160.0, ans=0.0
2023-11-22 19:19:52,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2066160.0, ans=0.1
2023-11-22 19:19:52,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2066160.0, ans=0.125
2023-11-22 19:19:56,392 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 7.933e+01 8.549e+01 9.311e+01 1.177e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 19:20:04,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2066226.6666666667, ans=0.05
2023-11-22 19:20:09,197 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9350, loss[loss=0.08945, simple_loss=0.1182, pruned_loss=0.02001, audio_tagging_loss=0.01034, over 15296.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.09476, pruned_loss=0.0148, audio_tagging_loss=0.009142, over 3050781.13 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:20:14,047 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 309950
2023-11-22 19:20:52,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2066493.3333333333, ans=0.07
2023-11-22 19:20:56,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2066493.3333333333, ans=0.0
2023-11-22 19:20:58,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2066560.0, ans=0.125
2023-11-22 19:21:11,930 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9400, loss[loss=0.08729, simple_loss=0.1081, pruned_loss=0.0247, audio_tagging_loss=0.00853, over 15985.00 frames. ], tot_loss[loss=0.07141, simple_loss=0.09449, pruned_loss=0.0149, audio_tagging_loss=0.009265, over 3049784.50 frames. ], batch size: 60, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:21:13,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2066626.6666666667, ans=0.0
2023-11-22 19:21:16,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310000
2023-11-22 19:21:24,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2066693.3333333333, ans=0.125
2023-11-22 19:21:30,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2066693.3333333333, ans=0.125
2023-11-22 19:21:33,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2066693.3333333333, ans=0.125
2023-11-22 19:21:36,997 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.10 vs. limit=22.5
2023-11-22 19:21:37,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2066760.0, ans=0.04949747468305833
2023-11-22 19:22:01,084 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.26 vs. limit=15.0
2023-11-22 19:22:03,898 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.065e+01 8.287e+01 9.048e+01 9.651e+01 1.161e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 19:22:06,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2066893.3333333333, ans=0.05
2023-11-22 19:22:09,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2066893.3333333333, ans=0.0
2023-11-22 19:22:13,674 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:22:15,725 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.50 vs. limit=15.0
2023-11-22 19:22:16,563 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9450, loss[loss=0.07555, simple_loss=0.09209, pruned_loss=0.01646, audio_tagging_loss=0.01305, over 15321.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09332, pruned_loss=0.01475, audio_tagging_loss=0.009407, over 3047583.07 frames. ], batch size: 59, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:22:22,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310050
2023-11-22 19:22:27,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2066960.0, ans=0.125
2023-11-22 19:22:36,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2067026.6666666667, ans=0.0
2023-11-22 19:22:40,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff2.min_abs, batch_count=2067026.6666666667, ans=0.1
2023-11-22 19:22:40,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2067026.6666666667, ans=0.125
2023-11-22 19:22:41,115 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.83 vs. limit=10.0
2023-11-22 19:22:43,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2067093.3333333333, ans=0.0
2023-11-22 19:22:45,771 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.48 vs. limit=6.0
2023-11-22 19:22:50,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2067093.3333333333, ans=0.0
2023-11-22 19:22:51,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2067093.3333333333, ans=0.0
2023-11-22 19:22:58,140 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.65 vs. limit=15.0
2023-11-22 19:23:20,770 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9500, loss[loss=0.0809, simple_loss=0.1057, pruned_loss=0.01706, audio_tagging_loss=0.01098, over 16072.00 frames. ], tot_loss[loss=0.07098, simple_loss=0.09351, pruned_loss=0.01476, audio_tagging_loss=0.009457, over 3050346.14 frames. ], batch size: 57, lr: 2.61e-03, grad_scale: 8.0
2023-11-22 19:23:24,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2067293.3333333333, ans=0.04949747468305833
2023-11-22 19:23:26,384 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310100
2023-11-22 19:23:31,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2067293.3333333333, ans=0.0
2023-11-22 19:23:37,966 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.83 vs. limit=15.0
2023-11-22 19:23:42,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2067360.0, ans=0.0
2023-11-22 19:23:54,221 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.22 vs. limit=12.0
2023-11-22 19:24:12,952 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.144e+01 8.401e+01 8.999e+01 9.659e+01 1.129e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-22 19:24:15,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2067560.0, ans=0.0
2023-11-22 19:24:24,416 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9550, loss[loss=0.07001, simple_loss=0.09436, pruned_loss=0.01244, audio_tagging_loss=0.01039, over 14421.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09279, pruned_loss=0.01464, audio_tagging_loss=0.009577, over 3045797.22 frames. ], batch size: 53, lr: 2.61e-03, grad_scale: 8.0
2023-11-22 19:24:28,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2067626.6666666667, ans=0.1
2023-11-22 19:24:29,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310150
2023-11-22 19:24:29,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2067626.6666666667, ans=0.125
2023-11-22 19:24:49,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2067760.0, ans=0.2
2023-11-22 19:25:01,901 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.23 vs. limit=10.0
2023-11-22 19:25:06,065 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.29 vs. limit=15.0
2023-11-22 19:25:10,991 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.79 vs. limit=22.5
2023-11-22 19:25:25,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2067893.3333333333, ans=0.1
2023-11-22 19:25:27,157 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9600, loss[loss=0.0715, simple_loss=0.0885, pruned_loss=0.01562, audio_tagging_loss=0.01163, over 15371.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09294, pruned_loss=0.01458, audio_tagging_loss=0.009657, over 3052057.71 frames. ], batch size: 60, lr: 2.61e-03, grad_scale: 16.0
2023-11-22 19:25:32,824 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310200
2023-11-22 19:25:36,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2067960.0, ans=0.125
2023-11-22 19:25:43,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.52 vs. limit=6.0
2023-11-22 19:25:46,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2068026.6666666667, ans=0.125
2023-11-22 19:26:02,432 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.52 vs. limit=15.0
2023-11-22 19:26:19,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2068226.6666666667, ans=0.125
2023-11-22 19:26:20,653 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.228e+01 8.239e+01 8.821e+01 9.524e+01 1.134e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 19:26:22,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2068226.6666666667, ans=0.125
2023-11-22 19:26:25,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2068226.6666666667, ans=0.0
2023-11-22 19:26:32,857 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9650, loss[loss=0.08223, simple_loss=0.1129, pruned_loss=0.01649, audio_tagging_loss=0.009295, over 15746.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09298, pruned_loss=0.01475, audio_tagging_loss=0.009581, over 3051054.43 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:26:37,757 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310250
2023-11-22 19:26:48,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2068360.0, ans=0.1
2023-11-22 19:27:00,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2068426.6666666667, ans=0.0
2023-11-22 19:27:25,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2068560.0, ans=0.2
2023-11-22 19:27:36,289 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9700, loss[loss=0.08449, simple_loss=0.1154, pruned_loss=0.02029, audio_tagging_loss=0.006493, over 15138.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09357, pruned_loss=0.0148, audio_tagging_loss=0.009348, over 3051269.54 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:27:36,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2068626.6666666667, ans=0.1
2023-11-22 19:27:41,789 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310300
2023-11-22 19:27:48,432 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.73 vs. limit=22.5
2023-11-22 19:27:49,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2068693.3333333333, ans=0.125
2023-11-22 19:27:51,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2068693.3333333333, ans=0.0
2023-11-22 19:27:55,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2068693.3333333333, ans=0.125
2023-11-22 19:27:58,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2068693.3333333333, ans=0.1
2023-11-22 19:28:12,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2068760.0, ans=0.125
2023-11-22 19:28:19,482 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.85 vs. limit=15.0
2023-11-22 19:28:29,028 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 8.323e+01 8.859e+01 9.612e+01 1.303e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 19:28:35,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2068893.3333333333, ans=0.0
2023-11-22 19:28:39,875 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9750, loss[loss=0.04639, simple_loss=0.0599, pruned_loss=0.007707, audio_tagging_loss=0.008732, over 15144.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09366, pruned_loss=0.01477, audio_tagging_loss=0.009208, over 3048078.67 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:28:45,389 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310350
2023-11-22 19:28:48,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2068960.0, ans=0.125
2023-11-22 19:28:55,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2069026.6666666667, ans=0.125
2023-11-22 19:28:59,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2069026.6666666667, ans=0.125
2023-11-22 19:29:01,789 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.14 vs. limit=15.0
2023-11-22 19:29:04,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2069093.3333333333, ans=0.0
2023-11-22 19:29:06,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2069093.3333333333, ans=0.2
2023-11-22 19:29:24,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2069160.0, ans=0.125
2023-11-22 19:29:38,957 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.53 vs. limit=22.5
2023-11-22 19:29:43,254 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.69 vs. limit=15.0
2023-11-22 19:29:44,818 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9800, loss[loss=0.07859, simple_loss=0.1042, pruned_loss=0.01784, audio_tagging_loss=0.008645, over 15857.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09341, pruned_loss=0.01466, audio_tagging_loss=0.009286, over 3038633.46 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:29:45,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2069293.3333333333, ans=0.125
2023-11-22 19:29:45,637 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.78 vs. limit=22.5
2023-11-22 19:29:49,794 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310400
2023-11-22 19:29:55,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2069293.3333333333, ans=0.125
2023-11-22 19:30:06,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2069360.0, ans=0.1
2023-11-22 19:30:37,552 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.460e+01 9.204e+01 9.900e+01 1.495e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-22 19:30:39,413 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.17 vs. limit=15.0
2023-11-22 19:30:41,221 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:30:48,376 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9850, loss[loss=0.06682, simple_loss=0.09457, pruned_loss=0.01127, audio_tagging_loss=0.008265, over 16536.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.0935, pruned_loss=0.01468, audio_tagging_loss=0.009149, over 3041682.76 frames. ], batch size: 61, lr: 2.60e-03, grad_scale: 8.0
2023-11-22 19:30:53,385 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310450
2023-11-22 19:31:06,697 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.91 vs. limit=15.0
2023-11-22 19:31:34,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2069826.6666666667, ans=0.125
2023-11-22 19:31:35,259 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.16 vs. limit=15.0
2023-11-22 19:31:48,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2069893.3333333333, ans=0.125
2023-11-22 19:31:51,989 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9900, loss[loss=0.07086, simple_loss=0.09504, pruned_loss=0.01358, audio_tagging_loss=0.00976, over 14057.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09386, pruned_loss=0.01489, audio_tagging_loss=0.009186, over 3042093.63 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 8.0
2023-11-22 19:31:56,928 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310500
2023-11-22 19:32:08,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2070026.6666666667, ans=0.125
2023-11-22 19:32:12,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2070026.6666666667, ans=0.0
2023-11-22 19:32:20,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2070093.3333333333, ans=0.125
2023-11-22 19:32:37,404 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.62 vs. limit=6.0
2023-11-22 19:32:45,207 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.326e+01 8.797e+01 9.739e+01 1.395e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 19:32:49,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2070226.6666666667, ans=0.125
2023-11-22 19:32:53,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2070226.6666666667, ans=0.125
2023-11-22 19:32:56,579 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 9950, loss[loss=0.06846, simple_loss=0.09141, pruned_loss=0.01329, audio_tagging_loss=0.009468, over 14753.00 frames. ], tot_loss[loss=0.07129, simple_loss=0.09415, pruned_loss=0.015, audio_tagging_loss=0.009217, over 3047842.34 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 8.0
2023-11-22 19:33:01,496 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310550
2023-11-22 19:33:16,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.41 vs. limit=15.0
2023-11-22 19:33:28,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2070426.6666666667, ans=0.125
2023-11-22 19:33:28,472 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.34 vs. limit=15.0
2023-11-22 19:33:35,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2070493.3333333333, ans=0.1
2023-11-22 19:33:51,512 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:33:59,655 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10000, loss[loss=0.07888, simple_loss=0.1009, pruned_loss=0.01897, audio_tagging_loss=0.009455, over 14866.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09311, pruned_loss=0.01477, audio_tagging_loss=0.009213, over 3047714.04 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:34:01,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2070626.6666666667, ans=0.1
2023-11-22 19:34:04,701 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310600
2023-11-22 19:34:09,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0
2023-11-22 19:34:30,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.68 vs. limit=6.0
2023-11-22 19:34:53,497 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.266e+01 8.772e+01 9.539e+01 1.390e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 19:35:03,287 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10050, loss[loss=0.06703, simple_loss=0.08927, pruned_loss=0.01436, audio_tagging_loss=0.008035, over 15170.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.0931, pruned_loss=0.01459, audio_tagging_loss=0.009165, over 3051280.52 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:35:08,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310650
2023-11-22 19:35:14,962 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.70 vs. limit=15.0
2023-11-22 19:35:30,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2071093.3333333333, ans=0.07
2023-11-22 19:35:41,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2071160.0, ans=0.2
2023-11-22 19:35:44,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2071160.0, ans=0.0
2023-11-22 19:36:08,963 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10100, loss[loss=0.08425, simple_loss=0.1096, pruned_loss=0.01904, audio_tagging_loss=0.01039, over 15523.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09344, pruned_loss=0.01467, audio_tagging_loss=0.009278, over 3051031.39 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:36:14,507 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310700
2023-11-22 19:36:21,180 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.95 vs. limit=15.0
2023-11-22 19:36:23,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2071360.0, ans=10.0
2023-11-22 19:36:49,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2071493.3333333333, ans=0.0
2023-11-22 19:36:57,795 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.93 vs. limit=10.0
2023-11-22 19:36:58,067 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.17 vs. limit=10.0
2023-11-22 19:36:58,548 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:36:58,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2071560.0, ans=0.95
2023-11-22 19:37:02,773 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.311e+01 8.891e+01 9.853e+01 2.191e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-22 19:37:12,778 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10150, loss[loss=0.05562, simple_loss=0.07932, pruned_loss=0.008936, audio_tagging_loss=0.00703, over 15021.00 frames. ], tot_loss[loss=0.07152, simple_loss=0.09463, pruned_loss=0.01496, audio_tagging_loss=0.009248, over 3050781.94 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:37:17,780 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310750
2023-11-22 19:37:41,915 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:37:48,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2071760.0, ans=0.1
2023-11-22 19:37:59,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2071826.6666666667, ans=0.07
2023-11-22 19:38:03,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2071893.3333333333, ans=0.0
2023-11-22 19:38:05,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2071893.3333333333, ans=0.2
2023-11-22 19:38:15,736 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10200, loss[loss=0.0627, simple_loss=0.07658, pruned_loss=0.01327, audio_tagging_loss=0.01114, over 14493.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.0937, pruned_loss=0.01467, audio_tagging_loss=0.009339, over 3048977.71 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:38:19,590 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:38:20,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310800
2023-11-22 19:38:30,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2072026.6666666667, ans=0.125
2023-11-22 19:38:39,622 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:39:02,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2072160.0, ans=0.0
2023-11-22 19:39:08,473 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 8.131e+01 8.876e+01 9.472e+01 1.289e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-22 19:39:19,374 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10250, loss[loss=0.05103, simple_loss=0.05849, pruned_loss=0.01027, audio_tagging_loss=0.01151, over 14092.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.09233, pruned_loss=0.01443, audio_tagging_loss=0.009526, over 3048789.19 frames. ], batch size: 53, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:39:25,541 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310850
2023-11-22 19:39:36,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2072360.0, ans=0.125
2023-11-22 19:39:40,690 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.65 vs. limit=22.5
2023-11-22 19:39:41,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2072360.0, ans=0.1
2023-11-22 19:39:43,386 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.88 vs. limit=15.0
2023-11-22 19:39:49,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2072426.6666666667, ans=0.125
2023-11-22 19:39:53,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2072426.6666666667, ans=0.125
2023-11-22 19:40:07,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2072493.3333333333, ans=0.0
2023-11-22 19:40:15,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2072560.0, ans=0.125
2023-11-22 19:40:23,801 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10300, loss[loss=0.07902, simple_loss=0.09881, pruned_loss=0.01603, audio_tagging_loss=0.01358, over 15061.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09253, pruned_loss=0.01441, audio_tagging_loss=0.009594, over 3044850.39 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:40:28,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310900
2023-11-22 19:40:29,993 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:40:30,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2072626.6666666667, ans=0.125
2023-11-22 19:40:35,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2072693.3333333333, ans=0.0
2023-11-22 19:40:36,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2072693.3333333333, ans=0.2
2023-11-22 19:40:45,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2072693.3333333333, ans=0.05
2023-11-22 19:41:03,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2072826.6666666667, ans=0.125
2023-11-22 19:41:07,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2072826.6666666667, ans=0.125
2023-11-22 19:41:07,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2072826.6666666667, ans=0.025
2023-11-22 19:41:14,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2072893.3333333333, ans=0.0
2023-11-22 19:41:16,997 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.279e+01 8.744e+01 9.443e+01 1.295e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 19:41:26,755 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10350, loss[loss=0.07315, simple_loss=0.09313, pruned_loss=0.01568, audio_tagging_loss=0.01091, over 15321.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09231, pruned_loss=0.0144, audio_tagging_loss=0.009748, over 3053889.04 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:41:28,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2072960.0, ans=0.1
2023-11-22 19:41:31,725 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 310950
2023-11-22 19:41:34,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2072960.0, ans=0.1
2023-11-22 19:41:51,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2073026.6666666667, ans=0.05
2023-11-22 19:42:00,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2073093.3333333333, ans=0.0
2023-11-22 19:42:30,611 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10400, loss[loss=0.06359, simple_loss=0.07729, pruned_loss=0.01314, audio_tagging_loss=0.0118, over 15442.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09304, pruned_loss=0.01455, audio_tagging_loss=0.009667, over 3043846.50 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:42:35,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2073293.3333333333, ans=0.0
2023-11-22 19:42:36,162 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311000
2023-11-22 19:42:44,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2073360.0, ans=0.2
2023-11-22 19:42:47,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2073360.0, ans=0.09899494936611666
2023-11-22 19:42:48,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2073360.0, ans=0.0
2023-11-22 19:42:53,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff3.min_abs, batch_count=2073360.0, ans=0.2
2023-11-22 19:43:07,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2073426.6666666667, ans=0.125
2023-11-22 19:43:14,695 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.64 vs. limit=15.0
2023-11-22 19:43:14,737 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.69 vs. limit=15.0
2023-11-22 19:43:15,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2073493.3333333333, ans=0.0
2023-11-22 19:43:25,682 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.233e+01 8.739e+01 9.492e+01 1.200e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-22 19:43:35,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=2073626.6666666667, ans=0.02
2023-11-22 19:43:36,098 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10450, loss[loss=0.06382, simple_loss=0.08519, pruned_loss=0.01041, audio_tagging_loss=0.01082, over 14419.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09297, pruned_loss=0.01453, audio_tagging_loss=0.009602, over 3044007.38 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:43:37,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2073626.6666666667, ans=0.125
2023-11-22 19:43:41,078 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311050
2023-11-22 19:43:42,692 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.58 vs. limit=15.0
2023-11-22 19:43:52,813 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.03 vs. limit=22.5
2023-11-22 19:43:56,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2073693.3333333333, ans=0.1
2023-11-22 19:44:02,119 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:44:18,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2073826.6666666667, ans=0.0
2023-11-22 19:44:22,078 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=14.42 vs. limit=15.0
2023-11-22 19:44:22,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2073826.6666666667, ans=0.0
2023-11-22 19:44:26,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2073893.3333333333, ans=0.07
2023-11-22 19:44:34,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2073893.3333333333, ans=0.125
2023-11-22 19:44:37,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2073893.3333333333, ans=0.125
2023-11-22 19:44:39,303 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10500, loss[loss=0.05291, simple_loss=0.06721, pruned_loss=0.006548, audio_tagging_loss=0.01276, over 15305.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09275, pruned_loss=0.0144, audio_tagging_loss=0.009427, over 3041987.83 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:44:44,312 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311100
2023-11-22 19:44:48,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2073960.0, ans=0.0
2023-11-22 19:45:02,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2074026.6666666667, ans=0.125
2023-11-22 19:45:04,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2074093.3333333333, ans=0.2
2023-11-22 19:45:05,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2074093.3333333333, ans=0.125
2023-11-22 19:45:06,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2074093.3333333333, ans=0.5
2023-11-22 19:45:17,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2074160.0, ans=0.125
2023-11-22 19:45:33,293 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.986e+01 7.970e+01 8.549e+01 9.373e+01 1.177e+02, threshold=1.710e+02, percent-clipped=0.0
2023-11-22 19:45:42,217 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10550, loss[loss=0.08887, simple_loss=0.122, pruned_loss=0.02007, audio_tagging_loss=0.00778, over 15459.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09294, pruned_loss=0.01458, audio_tagging_loss=0.009385, over 3040846.89 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:45:47,938 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311150
2023-11-22 19:45:57,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2074360.0, ans=0.0
2023-11-22 19:46:07,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2074360.0, ans=0.125
2023-11-22 19:46:15,036 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.52 vs. limit=22.5
2023-11-22 19:46:26,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2074493.3333333333, ans=0.125
2023-11-22 19:46:31,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2074493.3333333333, ans=0.125
2023-11-22 19:46:47,703 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10600, loss[loss=0.04733, simple_loss=0.05893, pruned_loss=0.008716, audio_tagging_loss=0.009147, over 14892.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09275, pruned_loss=0.01452, audio_tagging_loss=0.009326, over 3044133.09 frames. ], batch size: 60, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:46:51,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2074626.6666666667, ans=0.0
2023-11-22 19:46:53,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311200
2023-11-22 19:47:26,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2074826.6666666667, ans=0.0
2023-11-22 19:47:43,210 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.586e+01 8.272e+01 8.914e+01 9.712e+01 1.198e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-22 19:47:51,577 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10650, loss[loss=0.06685, simple_loss=0.08531, pruned_loss=0.01663, audio_tagging_loss=0.00756, over 15077.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.0921, pruned_loss=0.01448, audio_tagging_loss=0.009304, over 3042866.13 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:47:56,531 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311250
2023-11-22 19:48:22,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2075093.3333333333, ans=0.125
2023-11-22 19:48:33,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2075160.0, ans=0.125
2023-11-22 19:48:38,739 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.12 vs. limit=15.0
2023-11-22 19:48:39,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2075160.0, ans=0.125
2023-11-22 19:48:47,917 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.67 vs. limit=12.0
2023-11-22 19:48:54,720 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10700, loss[loss=0.05916, simple_loss=0.07622, pruned_loss=0.01194, audio_tagging_loss=0.009109, over 14439.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09192, pruned_loss=0.01439, audio_tagging_loss=0.009238, over 3043704.36 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:48:55,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2075293.3333333333, ans=0.05
2023-11-22 19:49:00,349 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311300
2023-11-22 19:49:17,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2075360.0, ans=0.125
2023-11-22 19:49:18,237 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.40 vs. limit=15.0
2023-11-22 19:49:21,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2075426.6666666667, ans=0.2
2023-11-22 19:49:32,220 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.29 vs. limit=15.0
2023-11-22 19:49:33,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2075493.3333333333, ans=0.1
2023-11-22 19:49:40,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.52 vs. limit=15.0
2023-11-22 19:49:50,184 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.953e+01 8.233e+01 8.994e+01 9.619e+01 1.201e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 19:50:00,038 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10750, loss[loss=0.06217, simple_loss=0.08362, pruned_loss=0.0124, audio_tagging_loss=0.007956, over 15110.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.0923, pruned_loss=0.01433, audio_tagging_loss=0.009191, over 3052027.89 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:50:05,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311350
2023-11-22 19:50:12,372 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.64 vs. limit=6.0
2023-11-22 19:50:13,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2075693.3333333333, ans=0.0
2023-11-22 19:50:16,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2075693.3333333333, ans=0.0
2023-11-22 19:50:18,408 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=14.88 vs. limit=15.0
2023-11-22 19:50:21,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2075693.3333333333, ans=0.1
2023-11-22 19:50:52,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2075893.3333333333, ans=0.125
2023-11-22 19:50:52,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2075893.3333333333, ans=0.2
2023-11-22 19:51:04,226 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10800, loss[loss=0.06751, simple_loss=0.09501, pruned_loss=0.01363, audio_tagging_loss=0.006373, over 14760.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09356, pruned_loss=0.01454, audio_tagging_loss=0.009063, over 3045294.39 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:51:08,603 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.90 vs. limit=22.5
2023-11-22 19:51:09,217 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311400
2023-11-22 19:51:14,727 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:51:59,227 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.263e+01 8.884e+01 9.522e+01 1.111e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 19:52:02,570 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.31 vs. limit=15.0
2023-11-22 19:52:07,959 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10850, loss[loss=0.06414, simple_loss=0.07858, pruned_loss=0.01202, audio_tagging_loss=0.01283, over 16154.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.0927, pruned_loss=0.01438, audio_tagging_loss=0.009225, over 3050252.28 frames. ], batch size: 62, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:52:13,048 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311450
2023-11-22 19:52:20,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2076360.0, ans=0.5
2023-11-22 19:52:52,313 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.10 vs. limit=22.5
2023-11-22 19:52:56,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2076493.3333333333, ans=0.0
2023-11-22 19:53:07,034 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:53:13,084 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10900, loss[loss=0.07714, simple_loss=0.1074, pruned_loss=0.01519, audio_tagging_loss=0.008245, over 16042.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09434, pruned_loss=0.01461, audio_tagging_loss=0.009152, over 3053490.51 frames. ], batch size: 61, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:53:15,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2076626.6666666667, ans=0.0
2023-11-22 19:53:17,902 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311500
2023-11-22 19:53:36,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.99 vs. limit=15.0
2023-11-22 19:53:38,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.98 vs. limit=15.0
2023-11-22 19:53:39,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2076760.0, ans=0.125
2023-11-22 19:53:54,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2076826.6666666667, ans=0.0
2023-11-22 19:54:09,630 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.216e+01 8.142e+01 8.836e+01 9.377e+01 1.085e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 19:54:16,920 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 10950, loss[loss=0.0841, simple_loss=0.112, pruned_loss=0.01737, audio_tagging_loss=0.01073, over 15992.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09375, pruned_loss=0.01448, audio_tagging_loss=0.009256, over 3049785.58 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:54:20,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2076960.0, ans=0.125
2023-11-22 19:54:21,921 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311550
2023-11-22 19:54:34,150 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.06 vs. limit=15.0
2023-11-22 19:54:37,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2077026.6666666667, ans=0.2
2023-11-22 19:54:44,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2077093.3333333333, ans=0.0
2023-11-22 19:54:56,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2077160.0, ans=0.1
2023-11-22 19:55:12,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2077226.6666666667, ans=0.125
2023-11-22 19:55:21,227 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11000, loss[loss=0.08299, simple_loss=0.1112, pruned_loss=0.01964, audio_tagging_loss=0.007751, over 15466.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09378, pruned_loss=0.01465, audio_tagging_loss=0.009315, over 3046435.91 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:55:26,187 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311600
2023-11-22 19:55:27,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2077293.3333333333, ans=0.0
2023-11-22 19:55:30,057 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 19:56:02,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.11 vs. limit=15.0
2023-11-22 19:56:06,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2077493.3333333333, ans=0.125
2023-11-22 19:56:17,635 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.294e+01 8.863e+01 9.568e+01 1.311e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 19:56:22,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2077560.0, ans=0.1
2023-11-22 19:56:26,026 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11050, loss[loss=0.06107, simple_loss=0.07642, pruned_loss=0.01187, audio_tagging_loss=0.01098, over 15243.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09384, pruned_loss=0.01471, audio_tagging_loss=0.009402, over 3048697.54 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:56:26,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2077626.6666666667, ans=0.0
2023-11-22 19:56:31,570 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311650
2023-11-22 19:56:32,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2077626.6666666667, ans=0.0
2023-11-22 19:56:51,508 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.51 vs. limit=15.0
2023-11-22 19:57:16,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2077893.3333333333, ans=0.09899494936611666
2023-11-22 19:57:29,638 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11100, loss[loss=0.08062, simple_loss=0.109, pruned_loss=0.01573, audio_tagging_loss=0.01039, over 16055.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09299, pruned_loss=0.01443, audio_tagging_loss=0.009455, over 3051473.92 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:57:32,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2077960.0, ans=0.125
2023-11-22 19:57:32,731 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.32 vs. limit=15.0
2023-11-22 19:57:34,537 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311700
2023-11-22 19:57:37,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2077960.0, ans=0.125
2023-11-22 19:57:38,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2077960.0, ans=0.07
2023-11-22 19:57:43,996 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.98 vs. limit=15.0
2023-11-22 19:57:56,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2078093.3333333333, ans=0.0
2023-11-22 19:57:59,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.31 vs. limit=6.0
2023-11-22 19:58:25,320 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.159e+01 8.862e+01 9.664e+01 1.542e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-22 19:58:32,635 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11150, loss[loss=0.08813, simple_loss=0.1119, pruned_loss=0.02064, audio_tagging_loss=0.01153, over 14652.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09308, pruned_loss=0.0145, audio_tagging_loss=0.009471, over 3056885.32 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 19:58:35,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2078293.3333333333, ans=0.015
2023-11-22 19:58:35,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2078293.3333333333, ans=0.05
2023-11-22 19:58:37,510 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311750
2023-11-22 19:58:38,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2078293.3333333333, ans=0.0
2023-11-22 19:58:56,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2078360.0, ans=0.0
2023-11-22 19:58:57,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2078360.0, ans=0.2
2023-11-22 19:58:59,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2078426.6666666667, ans=0.0
2023-11-22 19:59:07,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2078426.6666666667, ans=10.0
2023-11-22 19:59:13,146 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 19:59:37,909 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11200, loss[loss=0.07773, simple_loss=0.1077, pruned_loss=0.01498, audio_tagging_loss=0.008915, over 15706.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09308, pruned_loss=0.01449, audio_tagging_loss=0.009573, over 3058136.72 frames. ], batch size: 60, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 19:59:43,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311800
2023-11-22 19:59:43,985 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.09 vs. limit=22.5
2023-11-22 19:59:44,107 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.49 vs. limit=22.5
2023-11-22 20:00:12,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2078760.0, ans=0.0
2023-11-22 20:00:28,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2078893.3333333333, ans=0.125
2023-11-22 20:00:34,760 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 7.967e+01 8.796e+01 9.529e+01 1.148e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 20:00:42,237 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11250, loss[loss=0.0799, simple_loss=0.1121, pruned_loss=0.01514, audio_tagging_loss=0.008707, over 15642.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09242, pruned_loss=0.01439, audio_tagging_loss=0.009555, over 3056486.63 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 20:00:42,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2078960.0, ans=0.125
2023-11-22 20:00:44,174 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=22.5
2023-11-22 20:00:47,390 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311850
2023-11-22 20:01:26,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2079160.0, ans=0.125
2023-11-22 20:01:41,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2079226.6666666667, ans=0.1
2023-11-22 20:01:45,785 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11300, loss[loss=0.08781, simple_loss=0.1173, pruned_loss=0.01936, audio_tagging_loss=0.009814, over 15664.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09357, pruned_loss=0.01457, audio_tagging_loss=0.009358, over 3052355.53 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 20:01:50,721 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311900
2023-11-22 20:01:53,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2079293.3333333333, ans=0.125
2023-11-22 20:02:04,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2079360.0, ans=0.025
2023-11-22 20:02:20,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2079426.6666666667, ans=0.07
2023-11-22 20:02:41,675 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.168e+01 8.136e+01 8.665e+01 9.544e+01 1.497e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-22 20:02:47,803 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11350, loss[loss=0.08288, simple_loss=0.1134, pruned_loss=0.01705, audio_tagging_loss=0.009152, over 15658.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09262, pruned_loss=0.01441, audio_tagging_loss=0.009306, over 3051963.09 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:02:53,308 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.60 vs. limit=15.0
2023-11-22 20:02:53,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 311950
2023-11-22 20:03:03,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.02 vs. limit=15.0
2023-11-22 20:03:22,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2079760.0, ans=0.2
2023-11-22 20:03:27,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2079826.6666666667, ans=0.125
2023-11-22 20:03:27,839 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.95 vs. limit=15.0
2023-11-22 20:03:33,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2079826.6666666667, ans=0.125
2023-11-22 20:03:42,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2079893.3333333333, ans=0.2
2023-11-22 20:03:42,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2079893.3333333333, ans=0.0
2023-11-22 20:03:52,226 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11400, loss[loss=0.09188, simple_loss=0.1196, pruned_loss=0.02175, audio_tagging_loss=0.01032, over 14586.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09284, pruned_loss=0.01442, audio_tagging_loss=0.009238, over 3053351.08 frames. ], batch size: 54, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:03:57,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312000
2023-11-22 20:03:58,644 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-312000.pt
2023-11-22 20:04:16,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2080026.6666666667, ans=0.125
2023-11-22 20:04:52,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2080226.6666666667, ans=10.0
2023-11-22 20:04:52,989 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.235e+01 8.932e+01 9.432e+01 1.169e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 20:04:59,056 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11450, loss[loss=0.06063, simple_loss=0.07721, pruned_loss=0.01206, audio_tagging_loss=0.009961, over 15542.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09245, pruned_loss=0.01445, audio_tagging_loss=0.009152, over 3052979.48 frames. ], batch size: 60, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:05:03,994 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312050
2023-11-22 20:05:06,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2080293.3333333333, ans=0.125
2023-11-22 20:05:09,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2080293.3333333333, ans=0.125
2023-11-22 20:05:24,302 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.25 vs. limit=15.0
2023-11-22 20:05:33,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2080426.6666666667, ans=0.125
2023-11-22 20:05:39,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2080493.3333333333, ans=0.0
2023-11-22 20:05:39,901 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=2.563e-03
2023-11-22 20:06:00,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2080560.0, ans=0.0
2023-11-22 20:06:02,413 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11500, loss[loss=0.05714, simple_loss=0.0748, pruned_loss=0.009693, audio_tagging_loss=0.01004, over 15523.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09233, pruned_loss=0.01436, audio_tagging_loss=0.0092, over 3050746.68 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:06:07,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312100
2023-11-22 20:06:23,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2080693.3333333333, ans=10.0
2023-11-22 20:06:43,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2080826.6666666667, ans=0.5
2023-11-22 20:06:46,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.70 vs. limit=15.0
2023-11-22 20:06:50,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2080826.6666666667, ans=0.0
2023-11-22 20:07:00,682 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.062e+01 8.847e+01 9.578e+01 1.218e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-22 20:07:06,752 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11550, loss[loss=0.07338, simple_loss=0.09107, pruned_loss=0.0166, audio_tagging_loss=0.01125, over 15438.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.0936, pruned_loss=0.01457, audio_tagging_loss=0.009142, over 3055881.09 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:07:11,691 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312150
2023-11-22 20:07:30,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2081093.3333333333, ans=0.125
2023-11-22 20:07:43,833 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 20:07:49,986 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.42 vs. limit=22.5
2023-11-22 20:07:53,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.95 vs. limit=6.0
2023-11-22 20:08:04,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.70 vs. limit=15.0
2023-11-22 20:08:09,798 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11600, loss[loss=0.0627, simple_loss=0.08556, pruned_loss=0.01119, audio_tagging_loss=0.008723, over 15061.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09428, pruned_loss=0.01474, audio_tagging_loss=0.009136, over 3057469.36 frames. ], batch size: 56, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 20:08:14,782 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312200
2023-11-22 20:08:39,550 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.41 vs. limit=15.0
2023-11-22 20:08:58,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2081493.3333333333, ans=0.07
2023-11-22 20:09:08,467 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.935e+01 8.330e+01 9.074e+01 9.765e+01 1.554e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-22 20:09:13,389 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11650, loss[loss=0.06385, simple_loss=0.08252, pruned_loss=0.01334, audio_tagging_loss=0.009243, over 15690.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09371, pruned_loss=0.0146, audio_tagging_loss=0.009134, over 3045409.82 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:09:19,024 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312250
2023-11-22 20:09:20,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2081626.6666666667, ans=0.125
2023-11-22 20:09:21,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2081626.6666666667, ans=0.1
2023-11-22 20:10:16,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2081893.3333333333, ans=0.07
2023-11-22 20:10:16,771 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.38 vs. limit=22.5
2023-11-22 20:10:17,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2081960.0, ans=0.125
2023-11-22 20:10:18,465 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11700, loss[loss=0.08249, simple_loss=0.1018, pruned_loss=0.01992, audio_tagging_loss=0.01165, over 15141.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09356, pruned_loss=0.01458, audio_tagging_loss=0.009213, over 3049985.82 frames. ], batch size: 59, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:10:23,985 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312300
2023-11-22 20:10:38,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2082026.6666666667, ans=0.125
2023-11-22 20:10:53,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2082093.3333333333, ans=0.0
2023-11-22 20:11:17,085 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.195e+01 8.722e+01 9.532e+01 1.280e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-22 20:11:19,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2082226.6666666667, ans=0.125
2023-11-22 20:11:22,130 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11750, loss[loss=0.07327, simple_loss=0.09242, pruned_loss=0.01797, audio_tagging_loss=0.009093, over 14701.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09441, pruned_loss=0.01472, audio_tagging_loss=0.009099, over 3052942.93 frames. ], batch size: 55, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:11:27,142 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312350
2023-11-22 20:11:32,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2082293.3333333333, ans=0.125
2023-11-22 20:11:33,480 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:12:03,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2082493.3333333333, ans=0.1
2023-11-22 20:12:25,276 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11800, loss[loss=0.04492, simple_loss=0.05722, pruned_loss=0.005954, audio_tagging_loss=0.01036, over 13165.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09458, pruned_loss=0.01484, audio_tagging_loss=0.009155, over 3047025.26 frames. ], batch size: 52, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:12:29,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2082626.6666666667, ans=0.1
2023-11-22 20:12:30,355 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312400
2023-11-22 20:12:56,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2082760.0, ans=0.1
2023-11-22 20:13:24,248 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.495e+01 8.351e+01 8.869e+01 9.580e+01 1.621e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-22 20:13:24,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2082893.3333333333, ans=0.125
2023-11-22 20:13:30,308 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11850, loss[loss=0.07872, simple_loss=0.1034, pruned_loss=0.01895, audio_tagging_loss=0.008066, over 16996.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09392, pruned_loss=0.01474, audio_tagging_loss=0.009255, over 3049909.01 frames. ], batch size: 61, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:13:35,363 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312450
2023-11-22 20:14:05,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2083093.3333333333, ans=0.1
2023-11-22 20:14:13,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2083160.0, ans=0.1
2023-11-22 20:14:15,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2083160.0, ans=0.125
2023-11-22 20:14:34,575 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11900, loss[loss=0.0588, simple_loss=0.06125, pruned_loss=0.01283, audio_tagging_loss=0.01535, over 14350.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09265, pruned_loss=0.01449, audio_tagging_loss=0.009556, over 3042864.33 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:14:34,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2083293.3333333333, ans=0.125
2023-11-22 20:14:39,593 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312500
2023-11-22 20:14:39,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2083293.3333333333, ans=0.2
2023-11-22 20:14:57,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2083426.6666666667, ans=0.125
2023-11-22 20:14:59,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2083426.6666666667, ans=0.125
2023-11-22 20:15:11,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2083493.3333333333, ans=0.125
2023-11-22 20:15:16,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2083493.3333333333, ans=0.0
2023-11-22 20:15:26,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2083560.0, ans=0.125
2023-11-22 20:15:32,656 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.017e+01 8.415e+01 9.018e+01 9.656e+01 1.551e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-22 20:15:35,580 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.90 vs. limit=15.0
2023-11-22 20:15:37,475 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 11950, loss[loss=0.07981, simple_loss=0.1058, pruned_loss=0.01806, audio_tagging_loss=0.008873, over 15871.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09264, pruned_loss=0.01461, audio_tagging_loss=0.009594, over 3042436.74 frames. ], batch size: 58, lr: 2.60e-03, grad_scale: 16.0
2023-11-22 20:15:42,414 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312550
2023-11-22 20:15:55,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2083693.3333333333, ans=0.0
2023-11-22 20:15:57,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2083693.3333333333, ans=0.07
2023-11-22 20:16:00,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2083693.3333333333, ans=0.1
2023-11-22 20:16:08,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2083760.0, ans=0.125
2023-11-22 20:16:16,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2083826.6666666667, ans=0.0
2023-11-22 20:16:28,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2083893.3333333333, ans=0.0
2023-11-22 20:16:38,902 INFO [train_asr.py:1221] (0/4) Epoch 26, batch 12000, loss[loss=0.08109, simple_loss=0.1109, pruned_loss=0.01833, audio_tagging_loss=0.007305, over 16004.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09288, pruned_loss=0.01464, audio_tagging_loss=0.009634, over 3042702.70 frames. ], batch size: 57, lr: 2.60e-03, grad_scale: 32.0
2023-11-22 20:16:38,906 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 20:17:01,723 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([6.4271, 6.0362, 6.3513, 5.8329], device='cuda:0')
2023-11-22 20:17:07,701 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.0607, 1.7309, 3.3928, 2.8164, 3.6584, 3.7126, 3.1883, 3.1567],
       device='cuda:0')
2023-11-22 20:17:18,657 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3455, 5.0268, 4.7358, 5.1816], device='cuda:0')
2023-11-22 20:17:20,963 INFO [train_asr.py:1253] (0/4) Epoch 26, validation: loss=0.05885, simple_loss=0.05139, pruned_loss=0.00512, audio_tagging_loss=0.02803, over 4681554.00 frames. 
2023-11-22 20:17:20,964 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 20:17:23,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2083960.0, ans=0.2
2023-11-22 20:17:25,642 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312600
2023-11-22 20:17:48,793 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-26.pt
2023-11-22 20:18:24,315 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 0, loss[loss=0.07322, simple_loss=0.08145, pruned_loss=0.009611, audio_tagging_loss=0.02288, over 14881.00 frames. ], tot_loss[loss=0.07322, simple_loss=0.08145, pruned_loss=0.009611, audio_tagging_loss=0.02288, over 14881.00 frames. ], batch size: 55, lr: 2.55e-03, grad_scale: 32.0
2023-11-22 20:18:24,318 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 20:18:42,798 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3798, 5.0890, 4.7375, 4.9631], device='cuda:0')
2023-11-22 20:19:01,962 INFO [train_asr.py:1253] (0/4) Epoch 27, validation: loss=0.05818, simple_loss=0.05133, pruned_loss=0.005046, audio_tagging_loss=0.02747, over 4681554.00 frames. 
2023-11-22 20:19:01,963 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 20:19:07,608 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0
2023-11-22 20:19:10,110 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.25 vs. limit=15.0
2023-11-22 20:19:22,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2084180.0, ans=0.1
2023-11-22 20:19:29,559 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.30 vs. limit=12.0
2023-11-22 20:19:32,310 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.281e+01 9.255e+01 1.009e+02 1.305e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-22 20:19:33,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2084246.6666666667, ans=0.015
2023-11-22 20:19:39,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2084246.6666666667, ans=22.5
2023-11-22 20:19:43,059 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312650
2023-11-22 20:19:43,175 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:20:01,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2084380.0, ans=0.125
2023-11-22 20:20:06,137 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 50, loss[loss=0.1, simple_loss=0.1198, pruned_loss=0.02255, audio_tagging_loss=0.01756, over 15704.00 frames. ], tot_loss[loss=0.078, simple_loss=0.09051, pruned_loss=0.01441, audio_tagging_loss=0.01833, over 694979.97 frames. ], batch size: 56, lr: 2.55e-03, grad_scale: 16.0
2023-11-22 20:20:35,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2084580.0, ans=0.1
2023-11-22 20:20:47,040 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312700
2023-11-22 20:21:03,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2084713.3333333333, ans=0.125
2023-11-22 20:21:04,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2084713.3333333333, ans=0.125
2023-11-22 20:21:12,352 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 100, loss[loss=0.06885, simple_loss=0.08204, pruned_loss=0.01174, audio_tagging_loss=0.0161, over 16112.00 frames. ], tot_loss[loss=0.07858, simple_loss=0.09282, pruned_loss=0.0148, audio_tagging_loss=0.01736, over 1215552.28 frames. ], batch size: 60, lr: 2.55e-03, grad_scale: 16.0
2023-11-22 20:21:18,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2084780.0, ans=0.0
2023-11-22 20:21:35,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2084846.6666666667, ans=0.0
2023-11-22 20:21:42,715 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.400e+01 8.923e+01 9.571e+01 1.006e+02 1.303e+02, threshold=1.914e+02, percent-clipped=0.0
2023-11-22 20:21:51,993 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312750
2023-11-22 20:22:01,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2084980.0, ans=0.125
2023-11-22 20:22:04,040 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.53 vs. limit=15.0
2023-11-22 20:22:10,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2085046.6666666667, ans=0.125
2023-11-22 20:22:17,549 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 150, loss[loss=0.08261, simple_loss=0.1109, pruned_loss=0.01715, audio_tagging_loss=0.01003, over 14822.00 frames. ], tot_loss[loss=0.07649, simple_loss=0.09282, pruned_loss=0.01473, audio_tagging_loss=0.01535, over 1625491.79 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:22:28,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2085180.0, ans=0.015
2023-11-22 20:22:55,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2085313.3333333333, ans=0.1
2023-11-22 20:22:57,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312800
2023-11-22 20:23:07,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2085313.3333333333, ans=0.125
2023-11-22 20:23:22,036 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 200, loss[loss=0.08388, simple_loss=0.1136, pruned_loss=0.01799, audio_tagging_loss=0.009086, over 15972.00 frames. ], tot_loss[loss=0.0748, simple_loss=0.09269, pruned_loss=0.01477, audio_tagging_loss=0.01368, over 1941330.84 frames. ], batch size: 60, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:23:23,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2085446.6666666667, ans=0.125
2023-11-22 20:23:35,688 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=11.12 vs. limit=15.0
2023-11-22 20:23:51,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2085580.0, ans=0.1
2023-11-22 20:23:53,577 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.118e+01 8.443e+01 8.985e+01 9.584e+01 1.364e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 20:24:03,062 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312850
2023-11-22 20:24:19,164 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.86 vs. limit=15.0
2023-11-22 20:24:23,813 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.44 vs. limit=22.5
2023-11-22 20:24:24,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2085713.3333333333, ans=0.2
2023-11-22 20:24:28,008 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 250, loss[loss=0.06733, simple_loss=0.09769, pruned_loss=0.01159, audio_tagging_loss=0.0069, over 15058.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.09354, pruned_loss=0.01498, audio_tagging_loss=0.01233, over 2191765.89 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:24:30,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2085780.0, ans=0.0
2023-11-22 20:24:51,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2085846.6666666667, ans=0.0
2023-11-22 20:25:03,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2085913.3333333333, ans=0.125
2023-11-22 20:25:07,761 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312900
2023-11-22 20:25:27,682 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.05 vs. limit=15.0
2023-11-22 20:25:31,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.38 vs. limit=22.5
2023-11-22 20:25:31,869 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 300, loss[loss=0.08057, simple_loss=0.1054, pruned_loss=0.01814, audio_tagging_loss=0.00973, over 16331.00 frames. ], tot_loss[loss=0.07375, simple_loss=0.09422, pruned_loss=0.01519, audio_tagging_loss=0.01145, over 2383786.59 frames. ], batch size: 61, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:25:38,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2086113.3333333333, ans=0.09899494936611666
2023-11-22 20:25:59,670 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.01 vs. limit=12.0
2023-11-22 20:26:03,572 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.485e+01 8.248e+01 9.049e+01 9.988e+01 1.263e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 20:26:05,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2086246.6666666667, ans=0.2
2023-11-22 20:26:06,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2086246.6666666667, ans=0.0
2023-11-22 20:26:08,068 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.26 vs. limit=22.5
2023-11-22 20:26:12,594 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 312950
2023-11-22 20:26:18,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2086313.3333333333, ans=0.0
2023-11-22 20:26:36,497 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 350, loss[loss=0.07496, simple_loss=0.1039, pruned_loss=0.01292, audio_tagging_loss=0.0101, over 15451.00 frames. ], tot_loss[loss=0.0731, simple_loss=0.09482, pruned_loss=0.01499, audio_tagging_loss=0.0107, over 2536176.69 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:26:39,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2086446.6666666667, ans=0.2
2023-11-22 20:26:46,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2086446.6666666667, ans=0.125
2023-11-22 20:26:53,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2086513.3333333333, ans=0.0
2023-11-22 20:26:53,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2086513.3333333333, ans=0.0
2023-11-22 20:26:59,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2086513.3333333333, ans=0.1
2023-11-22 20:27:06,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2086580.0, ans=0.0
2023-11-22 20:27:15,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2086646.6666666667, ans=0.125
2023-11-22 20:27:17,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313000
2023-11-22 20:27:21,972 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.10 vs. limit=15.0
2023-11-22 20:27:25,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2086646.6666666667, ans=0.125
2023-11-22 20:27:35,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2086713.3333333333, ans=0.1
2023-11-22 20:27:35,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2086713.3333333333, ans=0.0
2023-11-22 20:27:42,143 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 400, loss[loss=0.07364, simple_loss=0.09233, pruned_loss=0.02119, audio_tagging_loss=0.006289, over 15360.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09217, pruned_loss=0.01456, audio_tagging_loss=0.01047, over 2642478.18 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:27:57,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2086846.6666666667, ans=0.0
2023-11-22 20:27:57,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2086846.6666666667, ans=0.125
2023-11-22 20:28:12,336 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.142e+01 8.725e+01 9.349e+01 1.086e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 20:28:21,035 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313050
2023-11-22 20:28:24,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2086980.0, ans=0.125
2023-11-22 20:28:32,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2087046.6666666667, ans=0.125
2023-11-22 20:28:33,009 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.91 vs. limit=15.0
2023-11-22 20:28:41,614 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.94 vs. limit=15.0
2023-11-22 20:28:45,809 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 450, loss[loss=0.07131, simple_loss=0.09024, pruned_loss=0.01772, audio_tagging_loss=0.008459, over 15097.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09251, pruned_loss=0.01445, audio_tagging_loss=0.01012, over 2738133.29 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:28:51,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2087113.3333333333, ans=0.0
2023-11-22 20:29:25,395 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313100
2023-11-22 20:29:26,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2087313.3333333333, ans=0.1
2023-11-22 20:29:49,315 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 500, loss[loss=0.06237, simple_loss=0.08556, pruned_loss=0.009667, audio_tagging_loss=0.009923, over 14860.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09167, pruned_loss=0.01417, audio_tagging_loss=0.009982, over 2799523.18 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:29:49,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2087446.6666666667, ans=0.125
2023-11-22 20:29:57,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2087446.6666666667, ans=0.1
2023-11-22 20:29:57,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2087446.6666666667, ans=0.125
2023-11-22 20:29:57,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2087446.6666666667, ans=0.2
2023-11-22 20:30:06,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2087513.3333333333, ans=0.025
2023-11-22 20:30:09,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2087513.3333333333, ans=0.125
2023-11-22 20:30:19,928 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.170e+01 8.718e+01 9.505e+01 1.331e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-22 20:30:20,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2087580.0, ans=0.125
2023-11-22 20:30:22,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2087580.0, ans=0.035
2023-11-22 20:30:27,735 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=17.63 vs. limit=22.5
2023-11-22 20:30:28,571 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313150
2023-11-22 20:30:54,212 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 550, loss[loss=0.06478, simple_loss=0.09232, pruned_loss=0.01053, audio_tagging_loss=0.008087, over 15783.00 frames. ], tot_loss[loss=0.07061, simple_loss=0.09277, pruned_loss=0.01444, audio_tagging_loss=0.009783, over 2851011.61 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:31:05,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2087846.6666666667, ans=0.125
2023-11-22 20:31:32,779 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313200
2023-11-22 20:31:34,900 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=14.13 vs. limit=15.0
2023-11-22 20:31:57,887 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 600, loss[loss=0.04566, simple_loss=0.05537, pruned_loss=0.006651, audio_tagging_loss=0.01133, over 15679.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09307, pruned_loss=0.01464, audio_tagging_loss=0.009704, over 2897069.71 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:31:58,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2088113.3333333333, ans=0.1
2023-11-22 20:32:05,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2088113.3333333333, ans=0.125
2023-11-22 20:32:06,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2088113.3333333333, ans=0.125
2023-11-22 20:32:10,924 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.29 vs. limit=15.0
2023-11-22 20:32:28,509 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.661e+01 8.589e+01 9.222e+01 1.004e+02 1.377e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-22 20:32:36,001 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.77 vs. limit=15.0
2023-11-22 20:32:37,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313250
2023-11-22 20:33:01,059 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 650, loss[loss=0.05074, simple_loss=0.06928, pruned_loss=0.008195, audio_tagging_loss=0.007907, over 14490.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09281, pruned_loss=0.01467, audio_tagging_loss=0.009662, over 2930143.99 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:33:14,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2088513.3333333333, ans=0.07
2023-11-22 20:33:42,042 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313300
2023-11-22 20:33:42,755 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.44 vs. limit=15.0
2023-11-22 20:33:48,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2088646.6666666667, ans=0.0
2023-11-22 20:34:06,055 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 700, loss[loss=0.0629, simple_loss=0.09145, pruned_loss=0.009762, audio_tagging_loss=0.007413, over 15442.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09228, pruned_loss=0.01457, audio_tagging_loss=0.00966, over 2956858.99 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:34:06,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2088780.0, ans=0.0
2023-11-22 20:34:33,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2088913.3333333333, ans=0.07
2023-11-22 20:34:35,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2088913.3333333333, ans=0.1
2023-11-22 20:34:36,746 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.830e+01 8.373e+01 8.817e+01 9.744e+01 1.427e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 20:34:36,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2088913.3333333333, ans=0.125
2023-11-22 20:34:39,926 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.82 vs. limit=12.0
2023-11-22 20:34:45,571 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313350
2023-11-22 20:35:11,630 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 750, loss[loss=0.0891, simple_loss=0.114, pruned_loss=0.02539, audio_tagging_loss=0.006717, over 15317.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09359, pruned_loss=0.0149, audio_tagging_loss=0.009504, over 2981346.20 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:35:12,298 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.34 vs. limit=15.0
2023-11-22 20:35:22,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2089180.0, ans=0.05
2023-11-22 20:35:52,109 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313400
2023-11-22 20:36:03,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2089380.0, ans=0.125
2023-11-22 20:36:15,354 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 800, loss[loss=0.05795, simple_loss=0.07191, pruned_loss=0.01297, audio_tagging_loss=0.009024, over 14397.00 frames. ], tot_loss[loss=0.07123, simple_loss=0.09386, pruned_loss=0.0149, audio_tagging_loss=0.009409, over 2996024.28 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:36:47,050 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.864e+01 8.197e+01 8.696e+01 9.566e+01 1.219e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-22 20:36:55,831 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313450
2023-11-22 20:36:58,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2089646.6666666667, ans=0.0
2023-11-22 20:36:59,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2089646.6666666667, ans=0.04949747468305833
2023-11-22 20:37:02,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2089646.6666666667, ans=0.125
2023-11-22 20:37:04,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2089646.6666666667, ans=0.1
2023-11-22 20:37:18,940 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 850, loss[loss=0.06165, simple_loss=0.07649, pruned_loss=0.0142, audio_tagging_loss=0.009208, over 15679.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09421, pruned_loss=0.01483, audio_tagging_loss=0.009435, over 3008345.75 frames. ], batch size: 61, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:37:30,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2089780.0, ans=0.0
2023-11-22 20:37:59,146 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313500
2023-11-22 20:38:02,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2089980.0, ans=0.125
2023-11-22 20:38:09,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2090046.6666666667, ans=0.125
2023-11-22 20:38:09,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2090046.6666666667, ans=0.1
2023-11-22 20:38:23,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2090113.3333333333, ans=0.1
2023-11-22 20:38:24,673 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 900, loss[loss=0.05608, simple_loss=0.07053, pruned_loss=0.009153, audio_tagging_loss=0.01166, over 15391.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09261, pruned_loss=0.01449, audio_tagging_loss=0.009507, over 3017056.75 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:38:25,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2090113.3333333333, ans=0.05
2023-11-22 20:38:31,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2090113.3333333333, ans=0.0
2023-11-22 20:38:38,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2090180.0, ans=0.125
2023-11-22 20:38:42,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2090180.0, ans=0.1
2023-11-22 20:38:53,873 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.563e+01 8.151e+01 8.989e+01 9.712e+01 1.407e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-22 20:38:55,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2090246.6666666667, ans=0.0
2023-11-22 20:39:00,109 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.56 vs. limit=15.0
2023-11-22 20:39:04,395 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313550
2023-11-22 20:39:19,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2090380.0, ans=0.1
2023-11-22 20:39:27,633 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 950, loss[loss=0.07458, simple_loss=0.09624, pruned_loss=0.01915, audio_tagging_loss=0.007304, over 13932.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.0935, pruned_loss=0.01472, audio_tagging_loss=0.009392, over 3019377.74 frames. ], batch size: 52, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:39:33,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2090446.6666666667, ans=0.125
2023-11-22 20:39:42,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2090513.3333333333, ans=0.1
2023-11-22 20:39:58,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2090580.0, ans=0.0
2023-11-22 20:40:07,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313600
2023-11-22 20:40:10,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2090646.6666666667, ans=0.1
2023-11-22 20:40:13,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2090646.6666666667, ans=0.0
2023-11-22 20:40:18,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2090713.3333333333, ans=0.04949747468305833
2023-11-22 20:40:28,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2090713.3333333333, ans=0.125
2023-11-22 20:40:31,709 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1000, loss[loss=0.07529, simple_loss=0.1067, pruned_loss=0.01429, audio_tagging_loss=0.007628, over 16585.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09338, pruned_loss=0.0147, audio_tagging_loss=0.009274, over 3027063.65 frames. ], batch size: 62, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:40:57,972 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 20:40:58,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2090913.3333333333, ans=0.1
2023-11-22 20:41:03,831 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.703e+01 8.088e+01 8.617e+01 9.308e+01 1.248e+02, threshold=1.723e+02, percent-clipped=0.0
2023-11-22 20:41:11,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313650
2023-11-22 20:41:12,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2090980.0, ans=0.125
2023-11-22 20:41:36,659 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1050, loss[loss=0.08438, simple_loss=0.1215, pruned_loss=0.01529, audio_tagging_loss=0.008363, over 15321.00 frames. ], tot_loss[loss=0.07103, simple_loss=0.09415, pruned_loss=0.01481, audio_tagging_loss=0.009154, over 3031434.89 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:41:49,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2091180.0, ans=0.1
2023-11-22 20:41:56,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2091180.0, ans=0.0
2023-11-22 20:41:56,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2091180.0, ans=0.1
2023-11-22 20:41:56,901 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.88 vs. limit=22.5
2023-11-22 20:42:14,263 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0
2023-11-22 20:42:15,407 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313700
2023-11-22 20:42:15,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2091313.3333333333, ans=0.0
2023-11-22 20:42:18,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2091313.3333333333, ans=0.125
2023-11-22 20:42:26,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2091380.0, ans=0.1
2023-11-22 20:42:38,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2091446.6666666667, ans=0.05
2023-11-22 20:42:39,808 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1100, loss[loss=0.06993, simple_loss=0.09424, pruned_loss=0.01529, audio_tagging_loss=0.007523, over 14261.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09335, pruned_loss=0.01469, audio_tagging_loss=0.009265, over 3033474.13 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:42:42,278 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 20:43:04,363 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.97 vs. limit=12.0
2023-11-22 20:43:05,955 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.12 vs. limit=6.0
2023-11-22 20:43:12,481 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.984e+01 8.123e+01 8.810e+01 9.567e+01 1.299e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-22 20:43:14,610 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0
2023-11-22 20:43:20,182 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313750
2023-11-22 20:43:28,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.33 vs. limit=15.0
2023-11-22 20:43:35,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2091713.3333333333, ans=0.95
2023-11-22 20:43:43,631 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1150, loss[loss=0.09078, simple_loss=0.1294, pruned_loss=0.0214, audio_tagging_loss=0.004691, over 15330.00 frames. ], tot_loss[loss=0.07034, simple_loss=0.09314, pruned_loss=0.01461, audio_tagging_loss=0.009163, over 3034932.74 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:43:59,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2091846.6666666667, ans=0.2
2023-11-22 20:43:59,158 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:44:11,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2091913.3333333333, ans=0.1
2023-11-22 20:44:14,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2091913.3333333333, ans=0.125
2023-11-22 20:44:14,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.51 vs. limit=10.0
2023-11-22 20:44:22,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2091980.0, ans=0.2
2023-11-22 20:44:23,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313800
2023-11-22 20:44:29,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2091980.0, ans=0.1
2023-11-22 20:44:31,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2091980.0, ans=0.125
2023-11-22 20:44:34,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.14 vs. limit=15.0
2023-11-22 20:44:49,093 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1200, loss[loss=0.05856, simple_loss=0.07581, pruned_loss=0.009599, audio_tagging_loss=0.01105, over 14335.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.093, pruned_loss=0.01451, audio_tagging_loss=0.009114, over 3036089.18 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:44:53,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2092113.3333333333, ans=0.125
2023-11-22 20:45:19,787 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.227e+01 8.331e+01 9.096e+01 9.827e+01 1.345e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-22 20:45:23,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2092246.6666666667, ans=0.125
2023-11-22 20:45:27,204 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313850
2023-11-22 20:45:32,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2092313.3333333333, ans=0.1
2023-11-22 20:45:36,436 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 20:45:38,161 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.34 vs. limit=15.0
2023-11-22 20:45:46,111 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.37 vs. limit=10.0
2023-11-22 20:45:48,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2092380.0, ans=0.0
2023-11-22 20:45:52,588 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1250, loss[loss=0.06074, simple_loss=0.08073, pruned_loss=0.01159, audio_tagging_loss=0.008785, over 15439.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09334, pruned_loss=0.01457, audio_tagging_loss=0.009092, over 3034578.94 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:45:59,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2092446.6666666667, ans=0.0
2023-11-22 20:46:05,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2092513.3333333333, ans=0.125
2023-11-22 20:46:21,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2092580.0, ans=0.0
2023-11-22 20:46:32,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313900
2023-11-22 20:46:50,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2092713.3333333333, ans=0.1
2023-11-22 20:46:51,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.46 vs. limit=12.0
2023-11-22 20:46:56,848 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1300, loss[loss=0.05933, simple_loss=0.08231, pruned_loss=0.009692, audio_tagging_loss=0.008483, over 15307.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09376, pruned_loss=0.01469, audio_tagging_loss=0.009054, over 3033532.59 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:47:01,356 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0
2023-11-22 20:47:03,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.93 vs. limit=15.0
2023-11-22 20:47:22,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2092913.3333333333, ans=0.04949747468305833
2023-11-22 20:47:27,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2092913.3333333333, ans=0.125
2023-11-22 20:47:28,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2092913.3333333333, ans=0.2
2023-11-22 20:47:29,476 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.264e+01 8.633e+01 9.270e+01 1.307e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-22 20:47:32,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2092913.3333333333, ans=0.2
2023-11-22 20:47:33,803 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.76 vs. limit=10.0
2023-11-22 20:47:36,230 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.50 vs. limit=22.5
2023-11-22 20:47:36,913 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 313950
2023-11-22 20:47:55,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2093046.6666666667, ans=0.125
2023-11-22 20:48:01,422 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1350, loss[loss=0.07999, simple_loss=0.1094, pruned_loss=0.01726, audio_tagging_loss=0.008034, over 15026.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09437, pruned_loss=0.0147, audio_tagging_loss=0.009007, over 3033348.23 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:48:34,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2093246.6666666667, ans=0.0
2023-11-22 20:48:40,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314000
2023-11-22 20:48:47,378 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 20:48:50,110 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=14.45 vs. limit=15.0
2023-11-22 20:48:56,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2093380.0, ans=0.1
2023-11-22 20:49:05,257 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1400, loss[loss=0.07165, simple_loss=0.0934, pruned_loss=0.01501, audio_tagging_loss=0.00994, over 14132.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09459, pruned_loss=0.01489, audio_tagging_loss=0.00902, over 3038187.45 frames. ], batch size: 53, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:49:20,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2093513.3333333333, ans=0.125
2023-11-22 20:49:22,229 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.15 vs. limit=15.0
2023-11-22 20:49:25,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2093513.3333333333, ans=0.0
2023-11-22 20:49:25,696 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.73 vs. limit=15.0
2023-11-22 20:49:29,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2093580.0, ans=0.125
2023-11-22 20:49:34,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.43 vs. limit=15.0
2023-11-22 20:49:35,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2093580.0, ans=0.0
2023-11-22 20:49:38,503 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.151e+01 8.829e+01 9.680e+01 1.142e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 20:49:39,220 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.61 vs. limit=12.0
2023-11-22 20:49:44,908 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314050
2023-11-22 20:49:56,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_na.min_abs, batch_count=2093713.3333333333, ans=0.02
2023-11-22 20:50:08,428 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.43 vs. limit=10.0
2023-11-22 20:50:08,777 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1450, loss[loss=0.07188, simple_loss=0.1017, pruned_loss=0.01334, audio_tagging_loss=0.007688, over 16420.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09437, pruned_loss=0.01477, audio_tagging_loss=0.009002, over 3046089.02 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:50:49,258 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314100
2023-11-22 20:50:49,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2093980.0, ans=0.2
2023-11-22 20:51:03,315 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.16 vs. limit=15.0
2023-11-22 20:51:12,969 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1500, loss[loss=0.05984, simple_loss=0.07339, pruned_loss=0.01193, audio_tagging_loss=0.01122, over 15663.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09412, pruned_loss=0.01474, audio_tagging_loss=0.009167, over 3047207.24 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:51:25,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2094180.0, ans=0.125
2023-11-22 20:51:25,513 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.46 vs. limit=15.0
2023-11-22 20:51:29,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2094180.0, ans=0.0
2023-11-22 20:51:35,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2094180.0, ans=0.125
2023-11-22 20:51:46,474 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.611e+01 8.220e+01 8.857e+01 9.629e+01 1.228e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-22 20:51:52,798 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314150
2023-11-22 20:51:54,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2094313.3333333333, ans=0.0
2023-11-22 20:52:11,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2094380.0, ans=0.0
2023-11-22 20:52:17,403 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1550, loss[loss=0.04619, simple_loss=0.05524, pruned_loss=0.007562, audio_tagging_loss=0.01101, over 16493.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09387, pruned_loss=0.01475, audio_tagging_loss=0.009259, over 3041986.31 frames. ], batch size: 64, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 20:52:19,111 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.54 vs. limit=15.0
2023-11-22 20:52:22,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2094446.6666666667, ans=0.1
2023-11-22 20:52:47,103 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.85 vs. limit=15.0
2023-11-22 20:52:57,446 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314200
2023-11-22 20:52:57,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2094646.6666666667, ans=0.125
2023-11-22 20:53:05,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.71 vs. limit=15.0
2023-11-22 20:53:10,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2094713.3333333333, ans=0.0
2023-11-22 20:53:21,389 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.24 vs. limit=15.0
2023-11-22 20:53:21,763 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1600, loss[loss=0.08438, simple_loss=0.112, pruned_loss=0.02167, audio_tagging_loss=0.006691, over 15370.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09321, pruned_loss=0.01471, audio_tagging_loss=0.009407, over 3041198.82 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:53:33,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2094846.6666666667, ans=0.0
2023-11-22 20:53:44,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.00 vs. limit=15.0
2023-11-22 20:53:54,983 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.862e+01 8.130e+01 8.806e+01 9.580e+01 1.162e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-22 20:53:56,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2094913.3333333333, ans=0.125
2023-11-22 20:54:01,175 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314250
2023-11-22 20:54:08,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn1.whiten.whitening_limit, batch_count=2094980.0, ans=22.5
2023-11-22 20:54:22,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2095046.6666666667, ans=0.125
2023-11-22 20:54:25,232 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1650, loss[loss=0.0688, simple_loss=0.09715, pruned_loss=0.01242, audio_tagging_loss=0.007805, over 15046.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09301, pruned_loss=0.0145, audio_tagging_loss=0.009387, over 3040849.26 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:54:28,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2095113.3333333333, ans=0.0
2023-11-22 20:55:05,922 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314300
2023-11-22 20:55:08,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2095313.3333333333, ans=0.1
2023-11-22 20:55:22,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2095380.0, ans=0.125
2023-11-22 20:55:28,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2095380.0, ans=0.125
2023-11-22 20:55:30,022 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1700, loss[loss=0.04381, simple_loss=0.05201, pruned_loss=0.006475, audio_tagging_loss=0.01133, over 15823.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09226, pruned_loss=0.01417, audio_tagging_loss=0.009482, over 3042067.25 frames. ], batch size: 61, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:55:35,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2095446.6666666667, ans=0.125
2023-11-22 20:55:35,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2095446.6666666667, ans=0.0
2023-11-22 20:55:49,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2095513.3333333333, ans=0.0
2023-11-22 20:55:51,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2095513.3333333333, ans=0.2
2023-11-22 20:56:02,457 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.152e+01 8.274e+01 8.793e+01 9.384e+01 1.139e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-22 20:56:10,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314350
2023-11-22 20:56:10,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=14.71 vs. limit=15.0
2023-11-22 20:56:33,714 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1750, loss[loss=0.08373, simple_loss=0.114, pruned_loss=0.01674, audio_tagging_loss=0.009982, over 14816.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09209, pruned_loss=0.01421, audio_tagging_loss=0.009423, over 3038990.01 frames. ], batch size: 55, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:56:42,310 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.66 vs. limit=15.0
2023-11-22 20:56:43,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2095780.0, ans=0.125
2023-11-22 20:56:48,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2095846.6666666667, ans=0.125
2023-11-22 20:56:54,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2095846.6666666667, ans=0.1
2023-11-22 20:56:56,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2095846.6666666667, ans=0.1
2023-11-22 20:57:04,774 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.12 vs. limit=15.0
2023-11-22 20:57:08,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2095913.3333333333, ans=0.125
2023-11-22 20:57:14,014 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314400
2023-11-22 20:57:14,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2095980.0, ans=0.125
2023-11-22 20:57:38,167 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1800, loss[loss=0.07338, simple_loss=0.1039, pruned_loss=0.01339, audio_tagging_loss=0.008027, over 15342.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09261, pruned_loss=0.0143, audio_tagging_loss=0.009318, over 3040667.35 frames. ], batch size: 60, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:57:39,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2096113.3333333333, ans=0.125
2023-11-22 20:57:47,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2096113.3333333333, ans=0.0
2023-11-22 20:57:49,897 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.47 vs. limit=12.0
2023-11-22 20:57:58,530 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.27 vs. limit=15.0
2023-11-22 20:57:59,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2096180.0, ans=0.125
2023-11-22 20:58:10,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2096246.6666666667, ans=0.125
2023-11-22 20:58:11,557 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.462e+01 8.025e+01 8.684e+01 9.480e+01 1.547e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-22 20:58:17,938 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314450
2023-11-22 20:58:20,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.96 vs. limit=15.0
2023-11-22 20:58:23,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2096313.3333333333, ans=0.125
2023-11-22 20:58:42,925 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1850, loss[loss=0.06578, simple_loss=0.09398, pruned_loss=0.009672, audio_tagging_loss=0.009111, over 15482.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09291, pruned_loss=0.01427, audio_tagging_loss=0.009149, over 3038182.55 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:58:49,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2096446.6666666667, ans=0.125
2023-11-22 20:58:51,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2096446.6666666667, ans=0.1
2023-11-22 20:59:06,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2096580.0, ans=0.0
2023-11-22 20:59:22,850 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314500
2023-11-22 20:59:31,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2096646.6666666667, ans=0.0
2023-11-22 20:59:35,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2096713.3333333333, ans=0.2
2023-11-22 20:59:46,111 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1900, loss[loss=0.06857, simple_loss=0.08429, pruned_loss=0.01776, audio_tagging_loss=0.008668, over 15157.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09291, pruned_loss=0.01434, audio_tagging_loss=0.00914, over 3042209.85 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 20:59:50,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2096780.0, ans=0.1
2023-11-22 21:00:04,724 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.65 vs. limit=15.0
2023-11-22 21:00:19,873 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 7.994e+01 8.463e+01 9.067e+01 1.382e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-22 21:00:26,054 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314550
2023-11-22 21:00:31,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2096980.0, ans=0.125
2023-11-22 21:00:37,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2097046.6666666667, ans=0.125
2023-11-22 21:00:44,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2097046.6666666667, ans=0.1
2023-11-22 21:00:49,515 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 1950, loss[loss=0.05757, simple_loss=0.06514, pruned_loss=0.01393, audio_tagging_loss=0.01107, over 15115.00 frames. ], tot_loss[loss=0.07009, simple_loss=0.09297, pruned_loss=0.01457, audio_tagging_loss=0.00903, over 3042478.20 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:00:55,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2097113.3333333333, ans=0.125
2023-11-22 21:01:25,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2097246.6666666665, ans=0.2
2023-11-22 21:01:27,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2097313.3333333335, ans=0.0
2023-11-22 21:01:28,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314600
2023-11-22 21:01:43,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2097380.0, ans=0.125
2023-11-22 21:01:54,199 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2000, loss[loss=0.06485, simple_loss=0.0854, pruned_loss=0.01314, audio_tagging_loss=0.009002, over 15507.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09279, pruned_loss=0.01445, audio_tagging_loss=0.009025, over 3047141.97 frames. ], batch size: 58, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:01:55,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2097446.6666666665, ans=0.1
2023-11-22 21:01:56,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2097446.6666666665, ans=0.0
2023-11-22 21:01:59,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2097446.6666666665, ans=0.125
2023-11-22 21:02:06,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2097513.3333333335, ans=0.1
2023-11-22 21:02:21,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2097580.0, ans=0.1
2023-11-22 21:02:26,119 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.310e+01 8.293e+01 8.864e+01 9.588e+01 1.359e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 21:02:30,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2097646.6666666665, ans=0.0
2023-11-22 21:02:32,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314650
2023-11-22 21:02:50,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2097713.3333333335, ans=0.0
2023-11-22 21:02:57,055 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2050, loss[loss=0.06893, simple_loss=0.09354, pruned_loss=0.01512, audio_tagging_loss=0.007047, over 14977.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09415, pruned_loss=0.01477, audio_tagging_loss=0.008997, over 3042781.62 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:03:05,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2097780.0, ans=0.2
2023-11-22 21:03:09,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2097846.6666666665, ans=0.1
2023-11-22 21:03:26,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2097913.3333333335, ans=0.125
2023-11-22 21:03:28,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2097913.3333333335, ans=0.125
2023-11-22 21:03:33,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2097913.3333333335, ans=0.125
2023-11-22 21:03:37,206 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314700
2023-11-22 21:03:48,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2098046.6666666665, ans=0.125
2023-11-22 21:03:50,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2098046.6666666665, ans=0.2
2023-11-22 21:04:00,567 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2100, loss[loss=0.07275, simple_loss=0.09735, pruned_loss=0.0162, audio_tagging_loss=0.007878, over 15886.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09461, pruned_loss=0.01481, audio_tagging_loss=0.008978, over 3047207.18 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:04:09,156 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:04:23,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2098180.0, ans=0.035
2023-11-22 21:04:26,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2098246.6666666665, ans=0.0
2023-11-22 21:04:34,494 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.684e+01 9.132e+01 9.760e+01 1.245e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-22 21:04:37,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2098246.6666666665, ans=0.0
2023-11-22 21:04:39,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2098313.3333333335, ans=0.0
2023-11-22 21:04:40,035 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.94 vs. limit=15.0
2023-11-22 21:04:40,786 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314750
2023-11-22 21:04:51,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2098380.0, ans=0.0
2023-11-22 21:05:05,788 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2150, loss[loss=0.07237, simple_loss=0.09579, pruned_loss=0.01554, audio_tagging_loss=0.008929, over 14587.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09429, pruned_loss=0.0148, audio_tagging_loss=0.009062, over 3034946.57 frames. ], batch size: 53, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:05:18,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2098513.3333333335, ans=0.125
2023-11-22 21:05:42,529 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:05:43,785 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314800
2023-11-22 21:05:43,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2098646.6666666665, ans=0.1
2023-11-22 21:05:50,980 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.78 vs. limit=15.0
2023-11-22 21:06:08,665 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2200, loss[loss=0.07334, simple_loss=0.1, pruned_loss=0.01608, audio_tagging_loss=0.007253, over 15363.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09414, pruned_loss=0.01477, audio_tagging_loss=0.009161, over 3037763.60 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:06:42,193 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.590e+01 8.202e+01 8.834e+01 9.532e+01 1.382e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-22 21:06:44,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2098913.3333333335, ans=0.125
2023-11-22 21:06:48,429 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314850
2023-11-22 21:06:52,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2098980.0, ans=0.125
2023-11-22 21:06:55,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2098980.0, ans=0.125
2023-11-22 21:07:07,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2099046.6666666665, ans=0.05
2023-11-22 21:07:09,941 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.70 vs. limit=15.0
2023-11-22 21:07:11,824 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2250, loss[loss=0.07991, simple_loss=0.09899, pruned_loss=0.01786, audio_tagging_loss=0.01256, over 15189.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09429, pruned_loss=0.01478, audio_tagging_loss=0.009192, over 3037381.83 frames. ], batch size: 59, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:07:16,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2099113.3333333335, ans=0.0
2023-11-22 21:07:29,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff3.min_abs, batch_count=2099180.0, ans=0.2
2023-11-22 21:07:51,694 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314900
2023-11-22 21:08:08,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2099380.0, ans=0.2
2023-11-22 21:08:15,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2099446.6666666665, ans=0.125
2023-11-22 21:08:15,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2099446.6666666665, ans=0.2
2023-11-22 21:08:16,705 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2300, loss[loss=0.06665, simple_loss=0.08824, pruned_loss=0.01086, audio_tagging_loss=0.01167, over 14797.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09426, pruned_loss=0.01479, audio_tagging_loss=0.009199, over 3041630.19 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:08:26,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2099446.6666666665, ans=0.1
2023-11-22 21:08:44,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2099580.0, ans=0.05
2023-11-22 21:08:50,534 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.386e+01 8.086e+01 8.698e+01 9.161e+01 1.218e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-22 21:08:54,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2099646.6666666665, ans=0.125
2023-11-22 21:08:55,690 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 314950
2023-11-22 21:09:13,520 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:09:20,838 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2350, loss[loss=0.08952, simple_loss=0.1277, pruned_loss=0.01978, audio_tagging_loss=0.005875, over 16091.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.09403, pruned_loss=0.01459, audio_tagging_loss=0.009263, over 3042647.05 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:09:29,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2099780.0, ans=0.0
2023-11-22 21:09:37,216 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.12 vs. limit=15.0
2023-11-22 21:09:39,373 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:10:00,561 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315000
2023-11-22 21:10:08,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2099980.0, ans=0.125
2023-11-22 21:10:11,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.67 vs. limit=15.0
2023-11-22 21:10:18,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2100046.6666666665, ans=0.1
2023-11-22 21:10:24,697 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2400, loss[loss=0.07443, simple_loss=0.09547, pruned_loss=0.01874, audio_tagging_loss=0.00795, over 14083.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09368, pruned_loss=0.01456, audio_tagging_loss=0.009351, over 3042233.02 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 32.0
2023-11-22 21:10:42,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2100180.0, ans=0.125
2023-11-22 21:10:58,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2100246.6666666665, ans=0.09899494936611666
2023-11-22 21:11:00,962 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.260e+01 8.417e+01 9.057e+01 9.650e+01 1.239e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 21:11:01,433 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:11:04,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315050
2023-11-22 21:11:22,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2100380.0, ans=0.125
2023-11-22 21:11:27,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=2100446.6666666665, ans=0.02
2023-11-22 21:11:28,997 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2450, loss[loss=0.06644, simple_loss=0.08824, pruned_loss=0.01119, audio_tagging_loss=0.01113, over 15274.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09268, pruned_loss=0.0144, audio_tagging_loss=0.009468, over 3037487.48 frames. ], batch size: 57, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:11:32,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2100446.6666666665, ans=0.0
2023-11-22 21:11:42,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2100513.3333333335, ans=0.125
2023-11-22 21:11:57,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2100580.0, ans=0.0
2023-11-22 21:12:08,070 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315100
2023-11-22 21:12:09,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.87 vs. limit=15.0
2023-11-22 21:12:10,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2100646.6666666665, ans=0.1
2023-11-22 21:12:33,362 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2500, loss[loss=0.05593, simple_loss=0.07424, pruned_loss=0.008767, audio_tagging_loss=0.01004, over 14244.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09344, pruned_loss=0.01458, audio_tagging_loss=0.009364, over 3031649.93 frames. ], batch size: 56, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:12:39,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2100780.0, ans=0.125
2023-11-22 21:12:55,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=2100846.6666666665, ans=0.05
2023-11-22 21:13:08,896 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.903e+01 8.214e+01 8.930e+01 9.863e+01 1.228e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 21:13:13,306 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315150
2023-11-22 21:13:24,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2101046.6666666665, ans=0.125
2023-11-22 21:13:36,985 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2550, loss[loss=0.04737, simple_loss=0.06925, pruned_loss=0.005318, audio_tagging_loss=0.00743, over 14311.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09274, pruned_loss=0.01455, audio_tagging_loss=0.009373, over 3035645.46 frames. ], batch size: 54, lr: 2.54e-03, grad_scale: 16.0
2023-11-22 21:13:39,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2101113.3333333335, ans=0.2
2023-11-22 21:13:48,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2101180.0, ans=0.125
2023-11-22 21:13:51,055 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.66 vs. limit=15.0
2023-11-22 21:13:53,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2101180.0, ans=0.0
2023-11-22 21:14:12,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2101246.6666666665, ans=0.0
2023-11-22 21:14:17,287 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315200
2023-11-22 21:14:23,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.43 vs. limit=15.0
2023-11-22 21:14:40,774 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2600, loss[loss=0.09922, simple_loss=0.1311, pruned_loss=0.0257, audio_tagging_loss=0.007971, over 17431.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09265, pruned_loss=0.0146, audio_tagging_loss=0.009286, over 3042660.47 frames. ], batch size: 61, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:15:00,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2101513.3333333335, ans=0.125
2023-11-22 21:15:03,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2101513.3333333335, ans=0.125
2023-11-22 21:15:13,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2101580.0, ans=0.07
2023-11-22 21:15:17,722 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 5.989e+01 8.300e+01 8.753e+01 9.403e+01 1.364e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-22 21:15:21,591 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315250
2023-11-22 21:15:21,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2101646.6666666665, ans=0.0
2023-11-22 21:15:46,462 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2650, loss[loss=0.05303, simple_loss=0.06536, pruned_loss=0.009251, audio_tagging_loss=0.0111, over 14694.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.0924, pruned_loss=0.01436, audio_tagging_loss=0.009326, over 3040710.50 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:16:06,077 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.10 vs. limit=22.5
2023-11-22 21:16:14,618 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.42 vs. limit=15.0
2023-11-22 21:16:16,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2101913.3333333335, ans=0.04949747468305833
2023-11-22 21:16:26,919 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315300
2023-11-22 21:16:30,856 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:16:35,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2101980.0, ans=0.0
2023-11-22 21:16:41,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2102046.6666666665, ans=0.1
2023-11-22 21:16:50,645 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2700, loss[loss=0.06751, simple_loss=0.09101, pruned_loss=0.01226, audio_tagging_loss=0.009748, over 15009.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09258, pruned_loss=0.01443, audio_tagging_loss=0.009152, over 3046661.58 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:16:56,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2102113.3333333335, ans=0.0
2023-11-22 21:17:20,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.51 vs. limit=10.0
2023-11-22 21:17:24,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2102246.6666666665, ans=0.125
2023-11-22 21:17:26,857 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.486e+01 8.121e+01 8.686e+01 9.401e+01 1.913e+02, threshold=1.737e+02, percent-clipped=1.0
2023-11-22 21:17:30,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315350
2023-11-22 21:17:54,225 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2750, loss[loss=0.08416, simple_loss=0.1144, pruned_loss=0.0183, audio_tagging_loss=0.008684, over 15348.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09276, pruned_loss=0.01444, audio_tagging_loss=0.009165, over 3048833.10 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:17:55,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2102446.6666666665, ans=0.0
2023-11-22 21:18:14,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2102513.3333333335, ans=0.1
2023-11-22 21:18:34,855 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315400
2023-11-22 21:18:35,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2102646.6666666665, ans=0.125
2023-11-22 21:18:37,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2102646.6666666665, ans=0.05
2023-11-22 21:18:42,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2102646.6666666665, ans=0.125
2023-11-22 21:18:49,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2102713.3333333335, ans=0.125
2023-11-22 21:18:50,965 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:18:52,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2102713.3333333335, ans=15.0
2023-11-22 21:19:00,021 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2800, loss[loss=0.06263, simple_loss=0.0776, pruned_loss=0.01256, audio_tagging_loss=0.01127, over 14912.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09238, pruned_loss=0.01449, audio_tagging_loss=0.009163, over 3040787.43 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:19:01,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2102780.0, ans=0.125
2023-11-22 21:19:13,780 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:19:28,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2102913.3333333335, ans=0.0
2023-11-22 21:19:31,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2102913.3333333335, ans=0.5
2023-11-22 21:19:37,008 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.680e+01 8.174e+01 8.854e+01 9.674e+01 2.365e+02, threshold=1.771e+02, percent-clipped=1.0
2023-11-22 21:19:39,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2102980.0, ans=0.0
2023-11-22 21:19:40,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315450
2023-11-22 21:19:44,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2102980.0, ans=0.2
2023-11-22 21:20:01,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2103046.6666666665, ans=0.1
2023-11-22 21:20:03,954 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2850, loss[loss=0.1018, simple_loss=0.1416, pruned_loss=0.02162, audio_tagging_loss=0.009393, over 14483.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09259, pruned_loss=0.01446, audio_tagging_loss=0.009144, over 3041156.81 frames. ], batch size: 52, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:20:24,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2103180.0, ans=0.125
2023-11-22 21:20:37,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2103246.6666666665, ans=0.125
2023-11-22 21:20:44,681 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315500
2023-11-22 21:20:54,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2103380.0, ans=0.0
2023-11-22 21:21:08,328 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2900, loss[loss=0.07551, simple_loss=0.09836, pruned_loss=0.01545, audio_tagging_loss=0.01088, over 16013.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.093, pruned_loss=0.01445, audio_tagging_loss=0.009207, over 3045919.32 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:21:13,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2103446.6666666665, ans=0.125
2023-11-22 21:21:37,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2103580.0, ans=0.0
2023-11-22 21:21:37,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2103580.0, ans=0.1
2023-11-22 21:21:45,633 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.599e+01 8.319e+01 9.006e+01 9.775e+01 1.156e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 21:21:48,846 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315550
2023-11-22 21:21:51,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2103646.6666666665, ans=0.1
2023-11-22 21:21:55,126 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:22:13,401 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 2950, loss[loss=0.05362, simple_loss=0.07065, pruned_loss=0.009198, audio_tagging_loss=0.009101, over 14440.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.0934, pruned_loss=0.01454, audio_tagging_loss=0.009251, over 3046834.94 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:22:15,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.71 vs. limit=15.0
2023-11-22 21:22:17,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2103780.0, ans=0.0
2023-11-22 21:22:21,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2103780.0, ans=0.0
2023-11-22 21:22:25,151 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.33 vs. limit=15.0
2023-11-22 21:22:25,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2103846.6666666665, ans=0.125
2023-11-22 21:22:45,508 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.61 vs. limit=15.0
2023-11-22 21:22:52,510 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315600
2023-11-22 21:23:10,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2104046.6666666665, ans=0.125
2023-11-22 21:23:17,747 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3000, loss[loss=0.1095, simple_loss=0.1449, pruned_loss=0.03062, audio_tagging_loss=0.006391, over 15996.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09372, pruned_loss=0.01467, audio_tagging_loss=0.009272, over 3045201.75 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:23:17,751 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 21:23:55,256 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9151, 3.7155, 4.8719, 4.3419], device='cuda:0')
2023-11-22 21:23:56,434 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.9215, 3.9545, 3.7828, 3.2968], device='cuda:0')
2023-11-22 21:23:59,580 INFO [train_asr.py:1253] (0/4) Epoch 27, validation: loss=0.058, simple_loss=0.05133, pruned_loss=0.005079, audio_tagging_loss=0.02726, over 4681554.00 frames. 
2023-11-22 21:23:59,581 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 21:24:05,109 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.08 vs. limit=15.0
2023-11-22 21:24:16,902 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.57 vs. limit=15.0
2023-11-22 21:24:36,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2104246.6666666665, ans=0.07
2023-11-22 21:24:36,798 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.824e+01 8.281e+01 8.900e+01 9.558e+01 2.196e+02, threshold=1.780e+02, percent-clipped=1.0
2023-11-22 21:24:39,360 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315650
2023-11-22 21:24:40,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2104313.3333333335, ans=0.0
2023-11-22 21:24:44,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2104313.3333333335, ans=0.0
2023-11-22 21:25:04,722 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3050, loss[loss=0.07626, simple_loss=0.1121, pruned_loss=0.01538, audio_tagging_loss=0.004826, over 15820.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09451, pruned_loss=0.01485, audio_tagging_loss=0.009262, over 3043660.99 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:25:09,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2104446.6666666665, ans=0.125
2023-11-22 21:25:10,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.56 vs. limit=15.0
2023-11-22 21:25:18,818 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.70 vs. limit=6.0
2023-11-22 21:25:20,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2104513.3333333335, ans=0.125
2023-11-22 21:25:41,310 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:25:43,921 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315700
2023-11-22 21:26:07,723 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3100, loss[loss=0.07306, simple_loss=0.1062, pruned_loss=0.01036, audio_tagging_loss=0.009607, over 14921.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09418, pruned_loss=0.01473, audio_tagging_loss=0.009329, over 3049780.68 frames. ], batch size: 54, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:26:22,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2104846.6666666665, ans=0.125
2023-11-22 21:26:24,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.32 vs. limit=15.0
2023-11-22 21:26:25,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2104846.6666666665, ans=0.5
2023-11-22 21:26:34,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2104913.3333333335, ans=0.1
2023-11-22 21:26:44,229 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.222e+01 8.175e+01 8.886e+01 9.764e+01 1.279e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 21:26:46,796 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315750
2023-11-22 21:26:47,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2104980.0, ans=0.2
2023-11-22 21:27:09,961 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3150, loss[loss=0.06725, simple_loss=0.08973, pruned_loss=0.01435, audio_tagging_loss=0.008037, over 16504.00 frames. ], tot_loss[loss=0.07081, simple_loss=0.09359, pruned_loss=0.01457, audio_tagging_loss=0.009438, over 3050303.45 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:27:37,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2105246.6666666665, ans=0.125
2023-11-22 21:27:50,014 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315800
2023-11-22 21:27:59,298 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.89 vs. limit=22.5
2023-11-22 21:28:06,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2105380.0, ans=0.125
2023-11-22 21:28:16,029 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3200, loss[loss=0.06898, simple_loss=0.0898, pruned_loss=0.01298, audio_tagging_loss=0.0111, over 14561.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09346, pruned_loss=0.01459, audio_tagging_loss=0.009503, over 3047112.20 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:28:16,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2105446.6666666665, ans=0.0
2023-11-22 21:28:18,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2105446.6666666665, ans=0.125
2023-11-22 21:28:18,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2105446.6666666665, ans=0.125
2023-11-22 21:28:25,650 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0
2023-11-22 21:28:35,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2105513.3333333335, ans=0.0
2023-11-22 21:28:51,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.60 vs. limit=15.0
2023-11-22 21:28:53,855 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.364e+01 8.971e+01 9.873e+01 1.649e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-22 21:28:55,871 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315850
2023-11-22 21:29:10,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2105713.3333333335, ans=0.125
2023-11-22 21:29:20,007 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3250, loss[loss=0.09728, simple_loss=0.1248, pruned_loss=0.02395, audio_tagging_loss=0.01091, over 14856.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09341, pruned_loss=0.01452, audio_tagging_loss=0.0095, over 3050579.35 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:29:25,704 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.70 vs. limit=15.0
2023-11-22 21:29:30,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2105780.0, ans=0.1
2023-11-22 21:29:37,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2105846.6666666665, ans=0.0
2023-11-22 21:29:44,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2105913.3333333335, ans=0.0
2023-11-22 21:29:53,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2105913.3333333335, ans=0.0
2023-11-22 21:30:00,791 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315900
2023-11-22 21:30:04,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2105980.0, ans=0.0
2023-11-22 21:30:13,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2106046.6666666665, ans=0.125
2023-11-22 21:30:23,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2106113.3333333335, ans=0.04949747468305833
2023-11-22 21:30:24,049 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3300, loss[loss=0.05783, simple_loss=0.06937, pruned_loss=0.01214, audio_tagging_loss=0.011, over 14304.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09371, pruned_loss=0.01463, audio_tagging_loss=0.009502, over 3048612.27 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:30:39,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2106180.0, ans=0.0
2023-11-22 21:30:43,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2106180.0, ans=0.1
2023-11-22 21:30:57,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2106246.6666666665, ans=0.0
2023-11-22 21:31:02,498 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.403e+01 9.072e+01 9.867e+01 1.352e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-22 21:31:03,948 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 315950
2023-11-22 21:31:04,613 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.03 vs. limit=15.0
2023-11-22 21:31:06,898 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.98 vs. limit=15.0
2023-11-22 21:31:12,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2106313.3333333335, ans=0.0
2023-11-22 21:31:15,576 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.06 vs. limit=15.0
2023-11-22 21:31:27,787 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3350, loss[loss=0.05947, simple_loss=0.07788, pruned_loss=0.01118, audio_tagging_loss=0.009356, over 14634.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09455, pruned_loss=0.01464, audio_tagging_loss=0.009427, over 3054160.03 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:31:38,496 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.30 vs. limit=22.5
2023-11-22 21:31:39,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.34 vs. limit=12.0
2023-11-22 21:31:54,106 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.01 vs. limit=22.5
2023-11-22 21:32:00,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2106580.0, ans=0.125
2023-11-22 21:32:07,810 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316000
2023-11-22 21:32:07,878 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:32:09,321 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-316000.pt
2023-11-22 21:32:18,909 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.49 vs. limit=22.5
2023-11-22 21:32:30,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2106713.3333333335, ans=0.0
2023-11-22 21:32:35,463 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3400, loss[loss=0.09055, simple_loss=0.1204, pruned_loss=0.02275, audio_tagging_loss=0.007593, over 14969.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09427, pruned_loss=0.01466, audio_tagging_loss=0.009296, over 3055759.10 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:32:55,976 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.67 vs. limit=22.5
2023-11-22 21:33:14,295 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.192e+01 8.214e+01 8.729e+01 9.303e+01 1.198e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-22 21:33:15,700 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316050
2023-11-22 21:33:34,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2107046.6666666665, ans=0.1
2023-11-22 21:33:39,065 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3450, loss[loss=0.08685, simple_loss=0.1183, pruned_loss=0.02102, audio_tagging_loss=0.006677, over 15284.00 frames. ], tot_loss[loss=0.07146, simple_loss=0.09496, pruned_loss=0.01479, audio_tagging_loss=0.009186, over 3054946.58 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:33:43,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2107113.3333333335, ans=0.1
2023-11-22 21:33:57,327 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.87 vs. limit=15.0
2023-11-22 21:34:13,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2107246.6666666665, ans=0.1
2023-11-22 21:34:18,740 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316100
2023-11-22 21:34:43,093 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3500, loss[loss=0.07435, simple_loss=0.1027, pruned_loss=0.01296, audio_tagging_loss=0.01004, over 16731.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09443, pruned_loss=0.01468, audio_tagging_loss=0.009123, over 3055036.48 frames. ], batch size: 63, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:35:03,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2107513.3333333335, ans=0.035
2023-11-22 21:35:04,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2107513.3333333335, ans=0.0
2023-11-22 21:35:16,219 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:35:20,936 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.375e+01 8.106e+01 8.631e+01 9.458e+01 1.678e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 21:35:22,332 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316150
2023-11-22 21:35:31,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.91 vs. limit=22.5
2023-11-22 21:35:38,991 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.59 vs. limit=15.0
2023-11-22 21:35:47,508 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3550, loss[loss=0.04808, simple_loss=0.06791, pruned_loss=0.006059, audio_tagging_loss=0.008063, over 15925.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09437, pruned_loss=0.01463, audio_tagging_loss=0.009107, over 3047667.09 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:35:50,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2107780.0, ans=0.125
2023-11-22 21:36:02,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2107846.6666666665, ans=0.0
2023-11-22 21:36:16,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2107913.3333333335, ans=0.025
2023-11-22 21:36:21,490 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.24 vs. limit=10.0
2023-11-22 21:36:27,528 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316200
2023-11-22 21:36:39,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2108046.6666666665, ans=0.1
2023-11-22 21:36:41,045 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:36:45,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2108046.6666666665, ans=0.125
2023-11-22 21:36:51,768 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3600, loss[loss=0.06638, simple_loss=0.08583, pruned_loss=0.01356, audio_tagging_loss=0.009905, over 15081.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09369, pruned_loss=0.01445, audio_tagging_loss=0.009146, over 3041685.61 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:36:57,102 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.69 vs. limit=15.0
2023-11-22 21:37:00,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2108113.3333333335, ans=0.1
2023-11-22 21:37:12,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2108180.0, ans=0.125
2023-11-22 21:37:17,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2108246.6666666665, ans=0.125
2023-11-22 21:37:21,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2108246.6666666665, ans=0.125
2023-11-22 21:37:30,463 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.329e+01 8.258e+01 8.715e+01 9.542e+01 1.505e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 21:37:31,912 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316250
2023-11-22 21:37:54,338 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=7.542e-03
2023-11-22 21:37:56,475 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3650, loss[loss=0.06706, simple_loss=0.09156, pruned_loss=0.01187, audio_tagging_loss=0.009414, over 15295.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.0931, pruned_loss=0.01424, audio_tagging_loss=0.009073, over 3050422.29 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:38:22,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2108580.0, ans=0.125
2023-11-22 21:38:26,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2108580.0, ans=0.125
2023-11-22 21:38:35,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316300
2023-11-22 21:38:47,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2108713.3333333335, ans=0.0
2023-11-22 21:38:49,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2108713.3333333335, ans=0.125
2023-11-22 21:38:52,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2108713.3333333335, ans=0.2
2023-11-22 21:38:52,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2108713.3333333335, ans=0.0
2023-11-22 21:39:00,755 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3700, loss[loss=0.07035, simple_loss=0.09222, pruned_loss=0.01754, audio_tagging_loss=0.006706, over 14766.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09356, pruned_loss=0.0145, audio_tagging_loss=0.009109, over 3046680.70 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:39:09,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2108780.0, ans=0.125
2023-11-22 21:39:16,907 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.61 vs. limit=15.0
2023-11-22 21:39:24,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2108913.3333333335, ans=0.0
2023-11-22 21:39:26,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2108913.3333333335, ans=0.125
2023-11-22 21:39:33,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2108913.3333333335, ans=0.0
2023-11-22 21:39:34,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2108913.3333333335, ans=0.0
2023-11-22 21:39:37,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2108913.3333333335, ans=0.125
2023-11-22 21:39:40,495 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.949e+01 8.416e+01 9.143e+01 1.024e+02 1.510e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-22 21:39:40,640 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316350
2023-11-22 21:39:45,292 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:39:59,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2109046.6666666665, ans=0.125
2023-11-22 21:40:05,182 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3750, loss[loss=0.05378, simple_loss=0.07055, pruned_loss=0.008585, audio_tagging_loss=0.009922, over 15598.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09398, pruned_loss=0.01459, audio_tagging_loss=0.009097, over 3048853.75 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:40:14,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2109113.3333333335, ans=0.1
2023-11-22 21:40:37,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2109246.6666666665, ans=0.0
2023-11-22 21:40:45,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316400
2023-11-22 21:40:48,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2109313.3333333335, ans=0.125
2023-11-22 21:40:50,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2109313.3333333335, ans=0.125
2023-11-22 21:40:50,988 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:40:52,734 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-22 21:40:53,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2109313.3333333335, ans=10.0
2023-11-22 21:41:03,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2109380.0, ans=0.025
2023-11-22 21:41:10,533 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3800, loss[loss=0.07396, simple_loss=0.1007, pruned_loss=0.01291, audio_tagging_loss=0.01071, over 15795.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09423, pruned_loss=0.0145, audio_tagging_loss=0.009207, over 3055273.45 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:41:36,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2109580.0, ans=0.1
2023-11-22 21:41:40,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2109580.0, ans=0.125
2023-11-22 21:41:41,095 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.71 vs. limit=15.0
2023-11-22 21:41:47,267 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.54 vs. limit=22.5
2023-11-22 21:41:50,095 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.525e+01 8.963e+01 9.873e+01 1.311e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-22 21:41:50,256 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316450
2023-11-22 21:42:14,514 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3850, loss[loss=0.08403, simple_loss=0.1161, pruned_loss=0.01525, audio_tagging_loss=0.01073, over 15276.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09399, pruned_loss=0.01446, audio_tagging_loss=0.009175, over 3047333.76 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:42:31,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2109846.6666666665, ans=0.125
2023-11-22 21:42:43,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2109913.3333333335, ans=0.0
2023-11-22 21:42:44,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2109913.3333333335, ans=0.1
2023-11-22 21:42:49,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2109913.3333333335, ans=0.125
2023-11-22 21:42:53,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316500
2023-11-22 21:42:58,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2109980.0, ans=0.05
2023-11-22 21:43:03,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2109980.0, ans=0.0
2023-11-22 21:43:04,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1.whitening_limit, batch_count=2110046.6666666665, ans=10.0
2023-11-22 21:43:08,262 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.88 vs. limit=15.0
2023-11-22 21:43:17,557 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3900, loss[loss=0.07767, simple_loss=0.1048, pruned_loss=0.01609, audio_tagging_loss=0.009175, over 14890.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.0936, pruned_loss=0.01445, audio_tagging_loss=0.009263, over 3044504.28 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:43:20,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2110113.3333333335, ans=0.0
2023-11-22 21:43:30,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2110180.0, ans=0.125
2023-11-22 21:43:57,204 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.364e+01 8.359e+01 8.893e+01 9.456e+01 1.423e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-22 21:43:57,356 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316550
2023-11-22 21:44:00,274 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.85 vs. limit=22.5
2023-11-22 21:44:17,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2110380.0, ans=0.1
2023-11-22 21:44:18,833 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0
2023-11-22 21:44:21,469 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 3950, loss[loss=0.07632, simple_loss=0.1103, pruned_loss=0.01395, audio_tagging_loss=0.007198, over 14589.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09437, pruned_loss=0.01472, audio_tagging_loss=0.009277, over 3042425.49 frames. ], batch size: 54, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:44:28,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2110446.6666666665, ans=0.125
2023-11-22 21:44:31,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2110446.6666666665, ans=0.2
2023-11-22 21:44:35,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2110513.3333333335, ans=0.0
2023-11-22 21:44:39,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2110513.3333333335, ans=0.0
2023-11-22 21:44:48,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2110580.0, ans=0.0
2023-11-22 21:44:48,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2110580.0, ans=0.125
2023-11-22 21:45:00,675 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316600
2023-11-22 21:45:23,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2110713.3333333335, ans=0.1
2023-11-22 21:45:25,355 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4000, loss[loss=0.05499, simple_loss=0.06914, pruned_loss=0.00996, audio_tagging_loss=0.01046, over 15595.00 frames. ], tot_loss[loss=0.0713, simple_loss=0.09455, pruned_loss=0.01461, audio_tagging_loss=0.009416, over 3043695.49 frames. ], batch size: 60, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:45:29,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2110780.0, ans=0.09899494936611666
2023-11-22 21:45:36,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2110846.6666666665, ans=0.0
2023-11-22 21:45:43,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2110846.6666666665, ans=0.0
2023-11-22 21:45:49,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2110913.3333333335, ans=0.1
2023-11-22 21:45:51,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2110913.3333333335, ans=0.1
2023-11-22 21:46:02,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2110980.0, ans=0.2
2023-11-22 21:46:04,311 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.309e+01 8.921e+01 9.654e+01 1.157e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-22 21:46:04,454 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316650
2023-11-22 21:46:09,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2110980.0, ans=0.04949747468305833
2023-11-22 21:46:13,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2110980.0, ans=0.125
2023-11-22 21:46:19,346 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.02 vs. limit=12.0
2023-11-22 21:46:28,460 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4050, loss[loss=0.082, simple_loss=0.1178, pruned_loss=0.01367, audio_tagging_loss=0.009406, over 15817.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09356, pruned_loss=0.01439, audio_tagging_loss=0.009423, over 3040583.51 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:46:32,058 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:46:40,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2111180.0, ans=0.125
2023-11-22 21:46:51,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2111180.0, ans=0.125
2023-11-22 21:47:02,385 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.97 vs. limit=22.5
2023-11-22 21:47:06,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2111313.3333333335, ans=0.0
2023-11-22 21:47:07,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316700
2023-11-22 21:47:26,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2111380.0, ans=0.1
2023-11-22 21:47:27,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2111380.0, ans=0.125
2023-11-22 21:47:27,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2111380.0, ans=0.125
2023-11-22 21:47:31,763 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4100, loss[loss=0.06371, simple_loss=0.07713, pruned_loss=0.01185, audio_tagging_loss=0.0133, over 14595.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09337, pruned_loss=0.01428, audio_tagging_loss=0.009453, over 3035065.85 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:47:52,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2111513.3333333335, ans=0.125
2023-11-22 21:48:10,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2111646.6666666665, ans=0.125
2023-11-22 21:48:10,814 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.312e+01 9.101e+01 1.011e+02 1.374e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-22 21:48:10,975 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316750
2023-11-22 21:48:15,488 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.57 vs. limit=6.0
2023-11-22 21:48:20,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2111646.6666666665, ans=0.125
2023-11-22 21:48:36,916 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4150, loss[loss=0.08632, simple_loss=0.1203, pruned_loss=0.01884, audio_tagging_loss=0.007333, over 15241.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09356, pruned_loss=0.01444, audio_tagging_loss=0.009253, over 3037557.86 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:48:42,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2111780.0, ans=0.1
2023-11-22 21:48:52,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2111846.6666666665, ans=0.0
2023-11-22 21:48:55,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2111846.6666666665, ans=0.125
2023-11-22 21:49:14,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2111980.0, ans=0.1
2023-11-22 21:49:14,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2111980.0, ans=0.1
2023-11-22 21:49:16,616 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316800
2023-11-22 21:49:23,141 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:49:31,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2112046.6666666665, ans=0.125
2023-11-22 21:49:32,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2112046.6666666665, ans=0.125
2023-11-22 21:49:32,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2112046.6666666665, ans=0.0
2023-11-22 21:49:40,888 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4200, loss[loss=0.05863, simple_loss=0.07826, pruned_loss=0.01019, audio_tagging_loss=0.009306, over 14878.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09352, pruned_loss=0.01451, audio_tagging_loss=0.009062, over 3038321.36 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:49:47,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2112113.3333333335, ans=0.0
2023-11-22 21:50:04,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2112180.0, ans=0.125
2023-11-22 21:50:20,408 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.029e+01 8.143e+01 8.631e+01 9.392e+01 1.233e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 21:50:20,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316850
2023-11-22 21:50:41,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2112380.0, ans=0.125
2023-11-22 21:50:43,605 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4250, loss[loss=0.07481, simple_loss=0.1051, pruned_loss=0.01439, audio_tagging_loss=0.00787, over 14895.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09429, pruned_loss=0.01473, audio_tagging_loss=0.009023, over 3047511.80 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:50:47,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2112446.6666666665, ans=0.125
2023-11-22 21:50:51,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2112446.6666666665, ans=0.0
2023-11-22 21:51:00,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2112513.3333333335, ans=0.125
2023-11-22 21:51:23,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316900
2023-11-22 21:51:23,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2112646.6666666665, ans=0.1
2023-11-22 21:51:24,794 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 21:51:47,714 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4300, loss[loss=0.05794, simple_loss=0.07202, pruned_loss=0.0122, audio_tagging_loss=0.009734, over 15152.00 frames. ], tot_loss[loss=0.07105, simple_loss=0.09443, pruned_loss=0.01481, audio_tagging_loss=0.00903, over 3046030.06 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:51:48,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2112780.0, ans=0.125
2023-11-22 21:51:54,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2112780.0, ans=0.125
2023-11-22 21:51:59,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2112846.6666666665, ans=0.125
2023-11-22 21:52:26,448 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.241e+01 8.941e+01 9.538e+01 1.182e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-22 21:52:26,604 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 316950
2023-11-22 21:52:26,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2112980.0, ans=0.07
2023-11-22 21:52:51,108 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4350, loss[loss=0.07341, simple_loss=0.09895, pruned_loss=0.015, audio_tagging_loss=0.008936, over 16294.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09392, pruned_loss=0.0147, audio_tagging_loss=0.009047, over 3044044.61 frames. ], batch size: 62, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:52:57,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2113113.3333333335, ans=0.125
2023-11-22 21:52:59,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2113113.3333333335, ans=0.5
2023-11-22 21:53:31,168 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.25 vs. limit=15.0
2023-11-22 21:53:31,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317000
2023-11-22 21:53:55,197 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4400, loss[loss=0.07723, simple_loss=0.09586, pruned_loss=0.0172, audio_tagging_loss=0.01209, over 14781.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09324, pruned_loss=0.01455, audio_tagging_loss=0.009107, over 3048372.99 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:54:07,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2113513.3333333335, ans=0.0
2023-11-22 21:54:34,944 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.698e+01 8.507e+01 9.216e+01 9.885e+01 1.286e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-22 21:54:35,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317050
2023-11-22 21:54:51,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2113713.3333333335, ans=0.0
2023-11-22 21:54:54,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2113713.3333333335, ans=0.1
2023-11-22 21:54:55,194 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.45 vs. limit=15.0
2023-11-22 21:54:59,491 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4450, loss[loss=0.0801, simple_loss=0.108, pruned_loss=0.01758, audio_tagging_loss=0.008504, over 14989.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09434, pruned_loss=0.0147, audio_tagging_loss=0.009028, over 3053524.51 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:55:00,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2113780.0, ans=0.125
2023-11-22 21:55:09,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2113780.0, ans=0.0
2023-11-22 21:55:18,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2113846.6666666665, ans=0.04949747468305833
2023-11-22 21:55:29,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2113913.3333333335, ans=0.125
2023-11-22 21:55:31,953 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=26.94 vs. limit=22.5
2023-11-22 21:55:39,466 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317100
2023-11-22 21:55:39,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2113980.0, ans=0.0
2023-11-22 21:56:03,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.29 vs. limit=15.0
2023-11-22 21:56:04,050 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4500, loss[loss=0.06975, simple_loss=0.09511, pruned_loss=0.01509, audio_tagging_loss=0.007104, over 14782.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09447, pruned_loss=0.0148, audio_tagging_loss=0.009058, over 3048278.39 frames. ], batch size: 56, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:56:06,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2114113.3333333335, ans=0.125
2023-11-22 21:56:08,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2114113.3333333335, ans=0.04949747468305833
2023-11-22 21:56:08,671 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.38 vs. limit=15.0
2023-11-22 21:56:09,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2114113.3333333335, ans=0.0
2023-11-22 21:56:44,905 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.650e+01 8.228e+01 8.995e+01 9.708e+01 1.219e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 21:56:45,046 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317150
2023-11-22 21:56:48,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2114313.3333333335, ans=0.0
2023-11-22 21:57:08,316 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4550, loss[loss=0.05954, simple_loss=0.07345, pruned_loss=0.01459, audio_tagging_loss=0.008218, over 14717.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09438, pruned_loss=0.01473, audio_tagging_loss=0.009098, over 3044982.82 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 21:57:45,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.15 vs. limit=22.5
2023-11-22 21:57:47,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2114646.6666666665, ans=0.0
2023-11-22 21:57:48,243 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.31 vs. limit=10.0
2023-11-22 21:57:48,851 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317200
2023-11-22 21:57:50,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2114646.6666666665, ans=0.125
2023-11-22 21:57:56,119 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.85 vs. limit=6.0
2023-11-22 21:57:57,934 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 21:58:09,033 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.78 vs. limit=22.5
2023-11-22 21:58:13,827 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4600, loss[loss=0.07523, simple_loss=0.09843, pruned_loss=0.01624, audio_tagging_loss=0.009765, over 15662.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.0947, pruned_loss=0.0148, audio_tagging_loss=0.0091, over 3045682.64 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:58:16,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2114780.0, ans=0.125
2023-11-22 21:58:33,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2114846.6666666665, ans=0.0
2023-11-22 21:58:48,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2114913.3333333335, ans=0.0
2023-11-22 21:58:52,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2114980.0, ans=0.1
2023-11-22 21:58:53,232 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317250
2023-11-22 21:58:54,291 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.032e+01 8.605e+01 9.322e+01 1.270e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-22 21:59:18,563 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4650, loss[loss=0.07946, simple_loss=0.1135, pruned_loss=0.01498, audio_tagging_loss=0.007728, over 14864.00 frames. ], tot_loss[loss=0.07153, simple_loss=0.09464, pruned_loss=0.01494, audio_tagging_loss=0.009273, over 3046771.05 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 21:59:22,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.36 vs. limit=22.5
2023-11-22 21:59:41,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2115246.6666666665, ans=0.2
2023-11-22 21:59:56,711 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.24 vs. limit=22.5
2023-11-22 21:59:57,319 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317300
2023-11-22 22:00:02,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2115313.3333333335, ans=0.125
2023-11-22 22:00:08,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2115380.0, ans=0.1
2023-11-22 22:00:19,650 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.95 vs. limit=15.0
2023-11-22 22:00:21,168 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4700, loss[loss=0.06252, simple_loss=0.08039, pruned_loss=0.01474, audio_tagging_loss=0.007587, over 14577.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09373, pruned_loss=0.01485, audio_tagging_loss=0.009362, over 3042916.91 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:00:48,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2115580.0, ans=0.2
2023-11-22 22:00:48,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2115580.0, ans=0.125
2023-11-22 22:00:50,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2115580.0, ans=0.0
2023-11-22 22:01:00,931 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317350
2023-11-22 22:01:02,001 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.187e+01 8.736e+01 9.527e+01 1.154e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 22:01:03,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2115646.6666666665, ans=0.125
2023-11-22 22:01:07,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2115646.6666666665, ans=0.1
2023-11-22 22:01:11,153 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.51 vs. limit=15.0
2023-11-22 22:01:25,032 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4750, loss[loss=0.08159, simple_loss=0.1137, pruned_loss=0.01653, audio_tagging_loss=0.00819, over 15740.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09281, pruned_loss=0.01466, audio_tagging_loss=0.00944, over 3044379.75 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:01:25,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2115780.0, ans=0.125
2023-11-22 22:01:49,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2115913.3333333335, ans=0.0
2023-11-22 22:02:03,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317400
2023-11-22 22:02:28,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2116113.3333333335, ans=0.125
2023-11-22 22:02:29,392 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4800, loss[loss=0.07059, simple_loss=0.09349, pruned_loss=0.01343, audio_tagging_loss=0.01042, over 15362.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09212, pruned_loss=0.01435, audio_tagging_loss=0.009514, over 3050826.58 frames. ], batch size: 59, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 22:02:32,519 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.35 vs. limit=12.0
2023-11-22 22:02:33,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2116113.3333333335, ans=0.125
2023-11-22 22:02:52,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2116180.0, ans=0.125
2023-11-22 22:02:55,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2116246.6666666665, ans=0.0
2023-11-22 22:03:09,301 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317450
2023-11-22 22:03:10,366 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.616e+01 8.423e+01 9.043e+01 9.752e+01 1.234e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-22 22:03:24,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2116380.0, ans=0.125
2023-11-22 22:03:27,238 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.52 vs. limit=15.0
2023-11-22 22:03:31,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2116380.0, ans=0.2
2023-11-22 22:03:33,809 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4850, loss[loss=0.08499, simple_loss=0.1133, pruned_loss=0.0197, audio_tagging_loss=0.008622, over 15950.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.0923, pruned_loss=0.01452, audio_tagging_loss=0.009632, over 3052522.97 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 32.0
2023-11-22 22:03:37,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2116446.6666666665, ans=0.0
2023-11-22 22:03:48,517 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.59 vs. limit=15.0
2023-11-22 22:03:49,682 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.55 vs. limit=22.5
2023-11-22 22:03:49,829 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.80 vs. limit=15.0
2023-11-22 22:04:07,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2116580.0, ans=0.1
2023-11-22 22:04:13,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317500
2023-11-22 22:04:15,483 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.98 vs. limit=15.0
2023-11-22 22:04:27,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2116713.3333333335, ans=0.1
2023-11-22 22:04:38,107 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4900, loss[loss=0.06078, simple_loss=0.08044, pruned_loss=0.01207, audio_tagging_loss=0.008484, over 15510.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09262, pruned_loss=0.01462, audio_tagging_loss=0.009622, over 3050231.63 frames. ], batch size: 57, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:04:38,639 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.57 vs. limit=15.0
2023-11-22 22:04:42,747 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:05:16,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2116980.0, ans=0.125
2023-11-22 22:05:18,294 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317550
2023-11-22 22:05:18,820 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.48 vs. limit=6.0
2023-11-22 22:05:19,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2116980.0, ans=0.125
2023-11-22 22:05:19,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2116980.0, ans=0.0
2023-11-22 22:05:20,624 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.184e+01 8.772e+01 9.580e+01 1.151e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-22 22:05:43,001 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 4950, loss[loss=0.07738, simple_loss=0.1123, pruned_loss=0.01343, audio_tagging_loss=0.007812, over 15457.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09409, pruned_loss=0.01466, audio_tagging_loss=0.009438, over 3052277.32 frames. ], batch size: 58, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:05:47,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.35 vs. limit=15.0
2023-11-22 22:05:55,077 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.24 vs. limit=15.0
2023-11-22 22:05:59,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2117180.0, ans=0.125
2023-11-22 22:06:00,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2117180.0, ans=0.1
2023-11-22 22:06:22,926 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317600
2023-11-22 22:06:32,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2117313.3333333335, ans=0.0
2023-11-22 22:06:46,924 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5000, loss[loss=0.07955, simple_loss=0.1053, pruned_loss=0.01856, audio_tagging_loss=0.00833, over 14763.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09444, pruned_loss=0.01463, audio_tagging_loss=0.009218, over 3045817.42 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:07:22,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2117580.0, ans=0.0
2023-11-22 22:07:26,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2117646.6666666665, ans=0.025
2023-11-22 22:07:27,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317650
2023-11-22 22:07:29,601 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.065e+01 8.747e+01 9.520e+01 1.397e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 22:07:51,025 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5050, loss[loss=0.06255, simple_loss=0.0858, pruned_loss=0.01192, audio_tagging_loss=0.007732, over 14397.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09363, pruned_loss=0.01452, audio_tagging_loss=0.009255, over 3044978.94 frames. ], batch size: 55, lr: 2.53e-03, grad_scale: 16.0
2023-11-22 22:08:05,091 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.19 vs. limit=15.0
2023-11-22 22:08:10,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2117846.6666666665, ans=0.125
2023-11-22 22:08:18,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2117913.3333333335, ans=0.125
2023-11-22 22:08:18,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2117913.3333333335, ans=0.125
2023-11-22 22:08:20,592 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.40 vs. limit=15.0
2023-11-22 22:08:23,129 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.09 vs. limit=15.0
2023-11-22 22:08:25,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2117913.3333333335, ans=0.0
2023-11-22 22:08:26,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2117913.3333333335, ans=0.125
2023-11-22 22:08:29,085 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.29 vs. limit=15.0
2023-11-22 22:08:31,706 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317700
2023-11-22 22:08:48,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2118046.6666666665, ans=0.0
2023-11-22 22:08:55,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.63 vs. limit=22.5
2023-11-22 22:08:56,522 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5100, loss[loss=0.06274, simple_loss=0.07534, pruned_loss=0.01708, audio_tagging_loss=0.007988, over 14776.00 frames. ], tot_loss[loss=0.07134, simple_loss=0.09463, pruned_loss=0.01489, audio_tagging_loss=0.009139, over 3038494.11 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:08:56,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2118113.3333333335, ans=0.125
2023-11-22 22:09:01,874 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.35 vs. limit=15.0
2023-11-22 22:09:24,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2118246.6666666665, ans=0.0
2023-11-22 22:09:36,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317750
2023-11-22 22:09:37,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2118313.3333333335, ans=0.125
2023-11-22 22:09:37,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2118313.3333333335, ans=0.125
2023-11-22 22:09:38,774 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.041e+01 8.209e+01 8.781e+01 9.360e+01 1.201e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 22:09:43,024 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.99 vs. limit=15.0
2023-11-22 22:09:55,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2118380.0, ans=0.95
2023-11-22 22:10:00,178 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5150, loss[loss=0.07119, simple_loss=0.092, pruned_loss=0.01555, audio_tagging_loss=0.009645, over 13351.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09354, pruned_loss=0.01462, audio_tagging_loss=0.009209, over 3040595.32 frames. ], batch size: 52, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:10:02,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2118446.6666666665, ans=0.0
2023-11-22 22:10:39,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2118646.6666666665, ans=0.125
2023-11-22 22:10:40,814 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317800
2023-11-22 22:11:04,851 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5200, loss[loss=0.06685, simple_loss=0.09091, pruned_loss=0.01335, audio_tagging_loss=0.008047, over 14738.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09251, pruned_loss=0.01447, audio_tagging_loss=0.009261, over 3044116.38 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:11:29,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2118913.3333333335, ans=0.1
2023-11-22 22:11:44,352 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317850
2023-11-22 22:11:46,669 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.652e+01 9.262e+01 9.962e+01 1.336e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-22 22:12:07,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2119046.6666666665, ans=0.125
2023-11-22 22:12:09,282 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5250, loss[loss=0.05775, simple_loss=0.07676, pruned_loss=0.009808, audio_tagging_loss=0.009565, over 14761.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09372, pruned_loss=0.01479, audio_tagging_loss=0.009082, over 3044673.66 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:12:09,945 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.44 vs. limit=22.5
2023-11-22 22:12:10,220 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.98 vs. limit=10.0
2023-11-22 22:12:14,346 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.14 vs. limit=15.0
2023-11-22 22:12:18,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2119113.3333333335, ans=0.125
2023-11-22 22:12:36,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.20 vs. limit=22.5
2023-11-22 22:12:48,392 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317900
2023-11-22 22:12:57,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2119313.3333333335, ans=0.0
2023-11-22 22:13:12,504 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5300, loss[loss=0.07817, simple_loss=0.1054, pruned_loss=0.01805, audio_tagging_loss=0.007427, over 15068.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09324, pruned_loss=0.01466, audio_tagging_loss=0.008994, over 3046192.46 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:13:30,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.31 vs. limit=15.0
2023-11-22 22:13:43,402 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.50 vs. limit=15.0
2023-11-22 22:13:53,101 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 317950
2023-11-22 22:13:56,555 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.353e+01 8.983e+01 9.482e+01 1.198e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 22:14:04,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2119713.3333333335, ans=0.0
2023-11-22 22:14:16,244 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5350, loss[loss=0.05135, simple_loss=0.06327, pruned_loss=0.008853, audio_tagging_loss=0.01086, over 16321.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09331, pruned_loss=0.01465, audio_tagging_loss=0.008973, over 3043719.19 frames. ], batch size: 64, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:14:56,199 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318000
2023-11-22 22:15:05,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2119980.0, ans=0.125
2023-11-22 22:15:10,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2120046.6666666665, ans=0.025
2023-11-22 22:15:19,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2120046.6666666665, ans=0.07
2023-11-22 22:15:21,776 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5400, loss[loss=0.07351, simple_loss=0.09852, pruned_loss=0.01604, audio_tagging_loss=0.008215, over 14747.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09344, pruned_loss=0.01468, audio_tagging_loss=0.009148, over 3045884.64 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:15:26,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2120113.3333333335, ans=0.125
2023-11-22 22:16:00,401 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318050
2023-11-22 22:16:04,452 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.681e+01 8.392e+01 8.997e+01 9.617e+01 1.276e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 22:16:17,902 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.94 vs. limit=6.0
2023-11-22 22:16:25,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.06 vs. limit=6.0
2023-11-22 22:16:25,803 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5450, loss[loss=0.0643, simple_loss=0.08499, pruned_loss=0.01437, audio_tagging_loss=0.007442, over 15274.00 frames. ], tot_loss[loss=0.07111, simple_loss=0.09406, pruned_loss=0.01492, audio_tagging_loss=0.009161, over 3051373.59 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:16:42,158 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:16:48,682 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.27 vs. limit=10.0
2023-11-22 22:17:06,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318100
2023-11-22 22:17:09,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2120646.6666666665, ans=0.2
2023-11-22 22:17:12,229 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.91 vs. limit=22.5
2023-11-22 22:17:19,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2120713.3333333335, ans=0.0
2023-11-22 22:17:25,353 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.49 vs. limit=22.5
2023-11-22 22:17:29,791 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5500, loss[loss=0.0665, simple_loss=0.08337, pruned_loss=0.01487, audio_tagging_loss=0.009938, over 15619.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09477, pruned_loss=0.01489, audio_tagging_loss=0.009191, over 3048026.20 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:17:48,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2120846.6666666665, ans=0.2
2023-11-22 22:17:52,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2120846.6666666665, ans=0.1
2023-11-22 22:17:53,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2120846.6666666665, ans=0.0
2023-11-22 22:17:55,783 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.31 vs. limit=22.5
2023-11-22 22:18:01,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2120913.3333333335, ans=0.025
2023-11-22 22:18:09,907 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318150
2023-11-22 22:18:13,410 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.216e+01 8.175e+01 8.992e+01 9.514e+01 1.142e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-22 22:18:19,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2121046.6666666665, ans=0.125
2023-11-22 22:18:33,987 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5550, loss[loss=0.07736, simple_loss=0.1087, pruned_loss=0.01272, audio_tagging_loss=0.01029, over 15274.00 frames. ], tot_loss[loss=0.07138, simple_loss=0.09453, pruned_loss=0.01479, audio_tagging_loss=0.00932, over 3048823.63 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:18:40,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2121113.3333333335, ans=0.125
2023-11-22 22:19:07,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2121246.6666666665, ans=0.05
2023-11-22 22:19:08,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2121246.6666666665, ans=0.0
2023-11-22 22:19:11,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2121313.3333333335, ans=0.125
2023-11-22 22:19:13,484 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318200
2023-11-22 22:19:19,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2121313.3333333335, ans=0.2
2023-11-22 22:19:25,349 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.22 vs. limit=15.0
2023-11-22 22:19:31,959 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.01 vs. limit=15.0
2023-11-22 22:19:34,802 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.59 vs. limit=10.0
2023-11-22 22:19:39,031 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5600, loss[loss=0.07448, simple_loss=0.101, pruned_loss=0.01505, audio_tagging_loss=0.008954, over 14735.00 frames. ], tot_loss[loss=0.07228, simple_loss=0.09563, pruned_loss=0.01515, audio_tagging_loss=0.009309, over 3049852.23 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:19:46,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2121446.6666666665, ans=0.05
2023-11-22 22:20:03,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2121580.0, ans=0.125
2023-11-22 22:20:18,193 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318250
2023-11-22 22:20:22,783 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.341e+01 8.364e+01 9.057e+01 9.975e+01 1.357e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-22 22:20:25,273 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 22:20:32,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2121713.3333333335, ans=0.035
2023-11-22 22:20:37,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2121713.3333333335, ans=0.0
2023-11-22 22:20:40,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2121713.3333333335, ans=0.125
2023-11-22 22:20:42,427 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5650, loss[loss=0.06751, simple_loss=0.08923, pruned_loss=0.01284, audio_tagging_loss=0.01005, over 15135.00 frames. ], tot_loss[loss=0.07192, simple_loss=0.09512, pruned_loss=0.01496, audio_tagging_loss=0.009403, over 3047112.15 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:20:46,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2121780.0, ans=0.0
2023-11-22 22:20:46,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2121780.0, ans=0.125
2023-11-22 22:20:51,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2121780.0, ans=0.0
2023-11-22 22:21:15,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2121913.3333333335, ans=0.125
2023-11-22 22:21:20,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2121980.0, ans=0.0
2023-11-22 22:21:22,559 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318300
2023-11-22 22:21:23,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2121980.0, ans=0.125
2023-11-22 22:21:46,027 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5700, loss[loss=0.07352, simple_loss=0.0916, pruned_loss=0.01685, audio_tagging_loss=0.01087, over 15696.00 frames. ], tot_loss[loss=0.07193, simple_loss=0.0952, pruned_loss=0.01495, audio_tagging_loss=0.009383, over 3044776.19 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:21:48,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2122113.3333333335, ans=0.0
2023-11-22 22:22:07,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2122180.0, ans=0.125
2023-11-22 22:22:13,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2122246.6666666665, ans=0.125
2023-11-22 22:22:23,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2122313.3333333335, ans=0.0
2023-11-22 22:22:24,032 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.32 vs. limit=15.0
2023-11-22 22:22:25,763 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318350
2023-11-22 22:22:29,266 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.048e+01 8.246e+01 8.801e+01 9.549e+01 1.194e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-22 22:22:32,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2122313.3333333335, ans=0.0
2023-11-22 22:22:45,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2122380.0, ans=0.125
2023-11-22 22:22:47,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2122380.0, ans=0.125
2023-11-22 22:22:50,413 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5750, loss[loss=0.07535, simple_loss=0.1103, pruned_loss=0.01119, audio_tagging_loss=0.008994, over 15962.00 frames. ], tot_loss[loss=0.07166, simple_loss=0.09488, pruned_loss=0.01497, audio_tagging_loss=0.00925, over 3043202.25 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:23:02,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2122513.3333333335, ans=0.125
2023-11-22 22:23:09,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2122513.3333333335, ans=0.125
2023-11-22 22:23:14,732 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.87 vs. limit=15.0
2023-11-22 22:23:27,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2122646.6666666665, ans=0.0
2023-11-22 22:23:29,924 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318400
2023-11-22 22:23:38,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2122646.6666666665, ans=0.05
2023-11-22 22:23:42,932 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.38 vs. limit=6.0
2023-11-22 22:23:49,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2122713.3333333335, ans=0.0
2023-11-22 22:23:54,257 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5800, loss[loss=0.08376, simple_loss=0.1073, pruned_loss=0.02259, audio_tagging_loss=0.007495, over 15747.00 frames. ], tot_loss[loss=0.07115, simple_loss=0.09417, pruned_loss=0.01487, audio_tagging_loss=0.009186, over 3040341.46 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:23:56,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2122780.0, ans=0.125
2023-11-22 22:24:00,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2122780.0, ans=0.05
2023-11-22 22:24:03,310 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.93 vs. limit=12.0
2023-11-22 22:24:04,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2122780.0, ans=0.125
2023-11-22 22:24:06,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.90 vs. limit=15.0
2023-11-22 22:24:20,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2122913.3333333335, ans=0.0
2023-11-22 22:24:21,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0
2023-11-22 22:24:34,769 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318450
2023-11-22 22:24:38,377 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.593e+01 8.198e+01 8.735e+01 9.439e+01 1.113e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-22 22:24:38,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2122980.0, ans=0.125
2023-11-22 22:24:58,499 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5850, loss[loss=0.0768, simple_loss=0.1012, pruned_loss=0.01969, audio_tagging_loss=0.0065, over 15152.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09338, pruned_loss=0.0146, audio_tagging_loss=0.009172, over 3035980.97 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:25:07,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2123113.3333333335, ans=0.035
2023-11-22 22:25:38,300 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318500
2023-11-22 22:25:38,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2123313.3333333335, ans=0.5
2023-11-22 22:25:39,022 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.24 vs. limit=10.0
2023-11-22 22:26:03,123 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5900, loss[loss=0.04479, simple_loss=0.04932, pruned_loss=0.00785, audio_tagging_loss=0.01228, over 15178.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09272, pruned_loss=0.0144, audio_tagging_loss=0.00912, over 3038170.69 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:26:25,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2123513.3333333335, ans=0.125
2023-11-22 22:26:37,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2123580.0, ans=0.125
2023-11-22 22:26:42,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318550
2023-11-22 22:26:47,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2123646.6666666665, ans=0.125
2023-11-22 22:26:48,713 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.253e+01 8.842e+01 9.499e+01 1.797e+02, threshold=1.768e+02, percent-clipped=1.0
2023-11-22 22:26:56,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2123713.3333333335, ans=0.125
2023-11-22 22:27:06,817 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 5950, loss[loss=0.07039, simple_loss=0.0932, pruned_loss=0.01651, audio_tagging_loss=0.007283, over 14731.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09345, pruned_loss=0.01452, audio_tagging_loss=0.009073, over 3040014.13 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:27:31,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2123913.3333333335, ans=0.125
2023-11-22 22:27:34,578 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.92 vs. limit=22.5
2023-11-22 22:27:46,893 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318600
2023-11-22 22:28:09,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.79 vs. limit=22.5
2023-11-22 22:28:10,917 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6000, loss[loss=0.06493, simple_loss=0.08383, pruned_loss=0.01506, audio_tagging_loss=0.00795, over 16174.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09443, pruned_loss=0.01462, audio_tagging_loss=0.008922, over 3041460.40 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:28:10,920 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 22:28:54,214 INFO [train_asr.py:1253] (0/4) Epoch 27, validation: loss=0.05853, simple_loss=0.05134, pruned_loss=0.005103, audio_tagging_loss=0.02775, over 4681554.00 frames. 
2023-11-22 22:28:54,214 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 22:29:02,292 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.70 vs. limit=10.0
2023-11-22 22:29:18,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2124246.6666666665, ans=0.125
2023-11-22 22:29:29,858 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.81 vs. limit=22.5
2023-11-22 22:29:34,118 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318650
2023-11-22 22:29:39,673 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.79 vs. limit=10.0
2023-11-22 22:29:40,614 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.043e+01 8.303e+01 8.890e+01 9.466e+01 1.522e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-22 22:29:41,915 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 22:29:57,687 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6050, loss[loss=0.05068, simple_loss=0.06372, pruned_loss=0.007646, audio_tagging_loss=0.01118, over 14835.00 frames. ], tot_loss[loss=0.07069, simple_loss=0.09422, pruned_loss=0.01462, audio_tagging_loss=0.008958, over 3041179.17 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:30:02,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2124446.6666666665, ans=0.1
2023-11-22 22:30:38,127 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318700
2023-11-22 22:30:51,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2124713.3333333335, ans=0.125
2023-11-22 22:31:02,043 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6100, loss[loss=0.07877, simple_loss=0.1057, pruned_loss=0.01993, audio_tagging_loss=0.005985, over 15165.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09348, pruned_loss=0.01449, audio_tagging_loss=0.009001, over 3037464.58 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:31:09,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2124780.0, ans=0.125
2023-11-22 22:31:22,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2124846.6666666665, ans=0.05
2023-11-22 22:31:41,955 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318750
2023-11-22 22:31:48,033 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.696e+01 8.282e+01 8.867e+01 9.593e+01 1.255e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-22 22:31:52,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2125046.6666666665, ans=22.5
2023-11-22 22:31:56,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2125046.6666666665, ans=0.125
2023-11-22 22:32:06,860 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6150, loss[loss=0.05833, simple_loss=0.07649, pruned_loss=0.01023, audio_tagging_loss=0.009847, over 15174.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09303, pruned_loss=0.01438, audio_tagging_loss=0.009078, over 3050947.23 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:32:24,679 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:32:47,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318800
2023-11-22 22:32:54,409 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=15.0
2023-11-22 22:33:11,365 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6200, loss[loss=0.07967, simple_loss=0.09982, pruned_loss=0.0182, audio_tagging_loss=0.01157, over 15424.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09152, pruned_loss=0.01418, audio_tagging_loss=0.009232, over 3049092.48 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:33:52,127 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318850
2023-11-22 22:33:57,957 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.282e+01 8.300e+01 8.948e+01 9.929e+01 3.016e+02, threshold=1.790e+02, percent-clipped=1.0
2023-11-22 22:34:12,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.80 vs. limit=10.0
2023-11-22 22:34:15,997 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6250, loss[loss=0.07561, simple_loss=0.1024, pruned_loss=0.01764, audio_tagging_loss=0.006786, over 15727.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09215, pruned_loss=0.01423, audio_tagging_loss=0.009317, over 3052083.33 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:34:21,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2125780.0, ans=0.0
2023-11-22 22:34:25,581 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.61 vs. limit=8.0
2023-11-22 22:34:55,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318900
2023-11-22 22:34:59,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2125980.0, ans=0.0
2023-11-22 22:35:20,699 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6300, loss[loss=0.06756, simple_loss=0.09138, pruned_loss=0.01268, audio_tagging_loss=0.009194, over 13985.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09238, pruned_loss=0.01423, audio_tagging_loss=0.009419, over 3049861.32 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:35:30,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2126113.3333333335, ans=0.0
2023-11-22 22:35:31,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2126113.3333333335, ans=0.0
2023-11-22 22:35:38,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2126180.0, ans=0.2
2023-11-22 22:35:39,514 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.14 vs. limit=5.0
2023-11-22 22:35:43,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2126180.0, ans=0.125
2023-11-22 22:35:55,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2126246.6666666665, ans=0.125
2023-11-22 22:36:01,577 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 318950
2023-11-22 22:36:07,417 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.041e+01 8.820e+01 9.703e+01 1.205e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 22:36:25,231 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6350, loss[loss=0.05639, simple_loss=0.07553, pruned_loss=0.01134, audio_tagging_loss=0.007281, over 15249.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09243, pruned_loss=0.01415, audio_tagging_loss=0.009449, over 3051270.27 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:36:25,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2126446.6666666665, ans=10.0
2023-11-22 22:36:26,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2126446.6666666665, ans=0.1
2023-11-22 22:37:03,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2126646.6666666665, ans=0.125
2023-11-22 22:37:05,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319000
2023-11-22 22:37:09,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2126646.6666666665, ans=0.0
2023-11-22 22:37:11,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2126646.6666666665, ans=0.125
2023-11-22 22:37:14,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2126646.6666666665, ans=0.0
2023-11-22 22:37:23,797 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.62 vs. limit=15.0
2023-11-22 22:37:29,147 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6400, loss[loss=0.05816, simple_loss=0.06566, pruned_loss=0.01469, audio_tagging_loss=0.01064, over 15586.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09243, pruned_loss=0.01424, audio_tagging_loss=0.009516, over 3045985.41 frames. ], batch size: 62, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:37:30,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2126780.0, ans=0.125
2023-11-22 22:37:42,185 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.19 vs. limit=15.0
2023-11-22 22:37:44,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2126846.6666666665, ans=0.1
2023-11-22 22:38:04,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2126913.3333333335, ans=0.0
2023-11-22 22:38:08,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319050
2023-11-22 22:38:12,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2126980.0, ans=0.1
2023-11-22 22:38:15,685 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.384e+01 8.479e+01 8.884e+01 9.505e+01 1.366e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 22:38:23,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2127046.6666666665, ans=0.125
2023-11-22 22:38:25,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2127046.6666666665, ans=0.125
2023-11-22 22:38:30,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2127046.6666666665, ans=0.025
2023-11-22 22:38:31,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2127113.3333333335, ans=0.125
2023-11-22 22:38:32,770 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6450, loss[loss=0.07751, simple_loss=0.112, pruned_loss=0.01586, audio_tagging_loss=0.005632, over 15643.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09274, pruned_loss=0.01427, audio_tagging_loss=0.009504, over 3042706.86 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:38:54,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2127180.0, ans=0.09899494936611666
2023-11-22 22:38:54,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.72 vs. limit=15.0
2023-11-22 22:39:11,151 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319100
2023-11-22 22:39:19,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2127313.3333333335, ans=0.125
2023-11-22 22:39:20,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2127313.3333333335, ans=0.2
2023-11-22 22:39:23,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2127380.0, ans=0.0
2023-11-22 22:39:24,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2127380.0, ans=0.125
2023-11-22 22:39:36,771 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6500, loss[loss=0.06695, simple_loss=0.08695, pruned_loss=0.01254, audio_tagging_loss=0.01093, over 15928.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09209, pruned_loss=0.01426, audio_tagging_loss=0.009568, over 3040049.09 frames. ], batch size: 62, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:39:50,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2127513.3333333335, ans=0.125
2023-11-22 22:39:56,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2127513.3333333335, ans=0.0
2023-11-22 22:40:08,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2127580.0, ans=0.0
2023-11-22 22:40:17,210 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319150
2023-11-22 22:40:24,337 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.580e+01 8.470e+01 8.995e+01 9.586e+01 1.282e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-22 22:40:24,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2127646.6666666665, ans=0.1
2023-11-22 22:40:31,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2127713.3333333335, ans=0.0
2023-11-22 22:40:33,514 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.57 vs. limit=22.5
2023-11-22 22:40:40,329 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6550, loss[loss=0.05534, simple_loss=0.06826, pruned_loss=0.01395, audio_tagging_loss=0.007262, over 14029.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09291, pruned_loss=0.01441, audio_tagging_loss=0.009329, over 3039025.75 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:40:44,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2127780.0, ans=0.0
2023-11-22 22:40:45,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2127780.0, ans=0.1
2023-11-22 22:40:54,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.45 vs. limit=6.0
2023-11-22 22:41:19,036 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.36 vs. limit=15.0
2023-11-22 22:41:21,060 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319200
2023-11-22 22:41:33,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2128046.6666666665, ans=0.0
2023-11-22 22:41:45,842 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6600, loss[loss=0.09044, simple_loss=0.1161, pruned_loss=0.0242, audio_tagging_loss=0.008212, over 15685.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.0927, pruned_loss=0.01438, audio_tagging_loss=0.009235, over 3036248.93 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:41:51,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2128113.3333333335, ans=0.125
2023-11-22 22:41:57,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2128113.3333333335, ans=0.125
2023-11-22 22:42:25,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319250
2023-11-22 22:42:29,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2128313.3333333335, ans=0.0
2023-11-22 22:42:33,339 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.441e+01 8.485e+01 9.023e+01 9.901e+01 1.531e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-22 22:42:49,978 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6650, loss[loss=0.0625, simple_loss=0.07889, pruned_loss=0.01285, audio_tagging_loss=0.01021, over 15751.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09245, pruned_loss=0.01435, audio_tagging_loss=0.009182, over 3036097.82 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:43:07,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2128513.3333333335, ans=0.125
2023-11-22 22:43:09,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2128513.3333333335, ans=0.04949747468305833
2023-11-22 22:43:14,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2128580.0, ans=0.1
2023-11-22 22:43:26,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2128646.6666666665, ans=0.1
2023-11-22 22:43:27,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2128646.6666666665, ans=0.0
2023-11-22 22:43:29,494 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319300
2023-11-22 22:43:37,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2128646.6666666665, ans=0.1
2023-11-22 22:43:53,684 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6700, loss[loss=0.05315, simple_loss=0.06099, pruned_loss=0.009882, audio_tagging_loss=0.01277, over 14616.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.0924, pruned_loss=0.01434, audio_tagging_loss=0.009073, over 3034380.28 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:44:05,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2128846.6666666665, ans=0.2
2023-11-22 22:44:07,212 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.11 vs. limit=15.0
2023-11-22 22:44:28,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2128913.3333333335, ans=0.0
2023-11-22 22:44:34,189 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319350
2023-11-22 22:44:41,537 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.124e+01 8.630e+01 9.282e+01 1.213e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-22 22:44:56,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2129046.6666666665, ans=0.0
2023-11-22 22:44:58,821 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6750, loss[loss=0.07067, simple_loss=0.09378, pruned_loss=0.01442, audio_tagging_loss=0.009359, over 15314.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09194, pruned_loss=0.01424, audio_tagging_loss=0.009123, over 3036341.98 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 22:45:02,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2129113.3333333335, ans=0.09899494936611666
2023-11-22 22:45:02,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2129113.3333333335, ans=0.0
2023-11-22 22:45:02,884 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:45:03,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2129113.3333333335, ans=0.04949747468305833
2023-11-22 22:45:14,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2129180.0, ans=0.0
2023-11-22 22:45:14,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2129180.0, ans=0.125
2023-11-22 22:45:20,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2129180.0, ans=0.0
2023-11-22 22:45:37,697 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319400
2023-11-22 22:45:46,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2129313.3333333335, ans=0.09899494936611666
2023-11-22 22:46:00,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2129380.0, ans=0.125
2023-11-22 22:46:03,698 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6800, loss[loss=0.06464, simple_loss=0.08381, pruned_loss=0.0128, audio_tagging_loss=0.009942, over 15106.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09247, pruned_loss=0.01445, audio_tagging_loss=0.00906, over 3035140.20 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:46:12,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2129446.6666666665, ans=0.1
2023-11-22 22:46:29,832 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:46:38,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2129580.0, ans=0.125
2023-11-22 22:46:42,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2129646.6666666665, ans=0.05
2023-11-22 22:46:43,185 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319450
2023-11-22 22:46:50,912 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.871e+01 8.253e+01 9.028e+01 9.739e+01 1.351e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-22 22:47:07,630 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6850, loss[loss=0.05434, simple_loss=0.0768, pruned_loss=0.008108, audio_tagging_loss=0.007834, over 13733.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09258, pruned_loss=0.01435, audio_tagging_loss=0.009001, over 3033690.60 frames. ], batch size: 53, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:47:08,341 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.32 vs. limit=22.5
2023-11-22 22:47:09,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2129780.0, ans=0.125
2023-11-22 22:47:17,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_na.min_abs, batch_count=2129780.0, ans=0.02
2023-11-22 22:47:21,963 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.92 vs. limit=10.0
2023-11-22 22:47:25,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2129846.6666666665, ans=0.125
2023-11-22 22:47:26,596 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.11 vs. limit=6.0
2023-11-22 22:47:27,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2129846.6666666665, ans=0.2
2023-11-22 22:47:28,882 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.84 vs. limit=10.0
2023-11-22 22:47:35,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.29 vs. limit=22.5
2023-11-22 22:47:43,789 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.78 vs. limit=8.0
2023-11-22 22:47:47,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319500
2023-11-22 22:47:54,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2129980.0, ans=0.0
2023-11-22 22:48:10,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2130113.3333333335, ans=0.125
2023-11-22 22:48:11,929 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6900, loss[loss=0.07543, simple_loss=0.1007, pruned_loss=0.01606, audio_tagging_loss=0.009035, over 16435.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09218, pruned_loss=0.01431, audio_tagging_loss=0.008981, over 3035191.93 frames. ], batch size: 61, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:48:28,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.03 vs. limit=15.0
2023-11-22 22:48:40,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2130246.6666666665, ans=0.125
2023-11-22 22:48:44,107 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.28 vs. limit=15.0
2023-11-22 22:48:51,316 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.54 vs. limit=15.0
2023-11-22 22:48:51,953 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319550
2023-11-22 22:48:59,091 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.191e+01 8.934e+01 9.580e+01 1.102e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-22 22:49:02,190 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 22:49:15,978 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 6950, loss[loss=0.07696, simple_loss=0.1046, pruned_loss=0.01645, audio_tagging_loss=0.008205, over 17474.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09296, pruned_loss=0.01442, audio_tagging_loss=0.008951, over 3035848.59 frames. ], batch size: 63, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:49:22,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2130446.6666666665, ans=0.125
2023-11-22 22:49:40,514 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.14 vs. limit=15.0
2023-11-22 22:49:55,232 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319600
2023-11-22 22:50:05,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2130713.3333333335, ans=0.125
2023-11-22 22:50:09,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2130713.3333333335, ans=0.0
2023-11-22 22:50:13,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2130713.3333333335, ans=0.125
2023-11-22 22:50:19,650 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7000, loss[loss=0.0651, simple_loss=0.08831, pruned_loss=0.01201, audio_tagging_loss=0.008936, over 15548.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09352, pruned_loss=0.01443, audio_tagging_loss=0.009005, over 3035985.40 frames. ], batch size: 57, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:50:24,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2130780.0, ans=0.125
2023-11-22 22:50:28,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2130780.0, ans=0.125
2023-11-22 22:50:44,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2130913.3333333335, ans=0.125
2023-11-22 22:50:59,606 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319650
2023-11-22 22:51:06,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2130980.0, ans=0.1
2023-11-22 22:51:07,659 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.087e+01 8.954e+01 9.728e+01 1.359e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 22:51:16,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2131046.6666666665, ans=0.0
2023-11-22 22:51:23,714 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7050, loss[loss=0.07479, simple_loss=0.09417, pruned_loss=0.0177, audio_tagging_loss=0.01001, over 15483.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09354, pruned_loss=0.01458, audio_tagging_loss=0.009058, over 3042770.47 frames. ], batch size: 60, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:51:28,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2131113.3333333335, ans=0.125
2023-11-22 22:51:37,609 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.58 vs. limit=15.0
2023-11-22 22:51:41,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2131180.0, ans=0.0
2023-11-22 22:51:44,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2131180.0, ans=0.125
2023-11-22 22:51:46,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2131180.0, ans=0.1
2023-11-22 22:51:47,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2131180.0, ans=0.0
2023-11-22 22:52:02,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2131313.3333333335, ans=0.125
2023-11-22 22:52:02,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2131313.3333333335, ans=0.0
2023-11-22 22:52:03,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319700
2023-11-22 22:52:12,104 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.37 vs. limit=15.0
2023-11-22 22:52:15,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2131380.0, ans=0.125
2023-11-22 22:52:20,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2131380.0, ans=0.125
2023-11-22 22:52:24,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2131380.0, ans=0.125
2023-11-22 22:52:28,770 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7100, loss[loss=0.05341, simple_loss=0.0678, pruned_loss=0.008153, audio_tagging_loss=0.01136, over 13815.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09311, pruned_loss=0.01461, audio_tagging_loss=0.009217, over 3037357.14 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:52:37,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2131446.6666666665, ans=0.1
2023-11-22 22:53:07,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319750
2023-11-22 22:53:11,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2131646.6666666665, ans=0.125
2023-11-22 22:53:15,650 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.625e+01 7.978e+01 8.785e+01 9.427e+01 1.365e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-22 22:53:28,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2131713.3333333335, ans=0.125
2023-11-22 22:53:31,760 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7150, loss[loss=0.07192, simple_loss=0.08951, pruned_loss=0.01409, audio_tagging_loss=0.01307, over 14537.00 frames. ], tot_loss[loss=0.07099, simple_loss=0.09403, pruned_loss=0.01473, audio_tagging_loss=0.009242, over 3044185.01 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:53:34,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2131780.0, ans=0.0
2023-11-22 22:53:41,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2131780.0, ans=0.125
2023-11-22 22:53:46,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2131846.6666666665, ans=0.2
2023-11-22 22:53:52,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2131846.6666666665, ans=0.125
2023-11-22 22:53:53,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2131846.6666666665, ans=0.125
2023-11-22 22:54:11,784 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319800
2023-11-22 22:54:22,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2132046.6666666665, ans=0.0
2023-11-22 22:54:26,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2132046.6666666665, ans=0.125
2023-11-22 22:54:35,963 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7200, loss[loss=0.07377, simple_loss=0.08411, pruned_loss=0.02211, audio_tagging_loss=0.009608, over 15540.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.0938, pruned_loss=0.0146, audio_tagging_loss=0.009283, over 3038172.18 frames. ], batch size: 59, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:54:46,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2132113.3333333335, ans=0.125
2023-11-22 22:54:47,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2132180.0, ans=0.125
2023-11-22 22:55:05,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2132246.6666666665, ans=0.2
2023-11-22 22:55:09,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2132246.6666666665, ans=0.0
2023-11-22 22:55:15,839 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319850
2023-11-22 22:55:18,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2132313.3333333335, ans=10.0
2023-11-22 22:55:23,094 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.344e+01 8.207e+01 8.744e+01 9.394e+01 1.258e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-22 22:55:26,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2132380.0, ans=0.0
2023-11-22 22:55:39,869 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7250, loss[loss=0.06907, simple_loss=0.09028, pruned_loss=0.01323, audio_tagging_loss=0.0107, over 15239.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09375, pruned_loss=0.0148, audio_tagging_loss=0.009447, over 3037158.60 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 32.0
2023-11-22 22:55:48,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2132446.6666666665, ans=0.0
2023-11-22 22:55:57,683 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 22:56:10,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2132580.0, ans=0.125
2023-11-22 22:56:18,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319900
2023-11-22 22:56:43,083 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7300, loss[loss=0.0718, simple_loss=0.09969, pruned_loss=0.01372, audio_tagging_loss=0.008231, over 15975.00 frames. ], tot_loss[loss=0.07064, simple_loss=0.09346, pruned_loss=0.01458, audio_tagging_loss=0.009334, over 3037113.84 frames. ], batch size: 58, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:56:45,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2132780.0, ans=0.0
2023-11-22 22:56:47,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2132780.0, ans=0.125
2023-11-22 22:57:22,985 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 319950
2023-11-22 22:57:28,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2132980.0, ans=0.09899494936611666
2023-11-22 22:57:32,784 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.257e+01 8.750e+01 9.620e+01 1.168e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-22 22:57:45,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2133113.3333333335, ans=0.1
2023-11-22 22:57:46,963 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7350, loss[loss=0.1017, simple_loss=0.1362, pruned_loss=0.0272, audio_tagging_loss=0.006359, over 13813.00 frames. ], tot_loss[loss=0.0711, simple_loss=0.09412, pruned_loss=0.01478, audio_tagging_loss=0.009262, over 3039124.94 frames. ], batch size: 52, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:57:49,129 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.77 vs. limit=15.0
2023-11-22 22:57:52,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2133113.3333333335, ans=0.0
2023-11-22 22:58:26,145 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320000
2023-11-22 22:58:27,684 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-320000.pt
2023-11-22 22:58:32,422 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.02 vs. limit=15.0
2023-11-22 22:58:46,785 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.95 vs. limit=6.0
2023-11-22 22:58:51,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2133380.0, ans=0.025
2023-11-22 22:58:53,952 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7400, loss[loss=0.06472, simple_loss=0.08254, pruned_loss=0.01195, audio_tagging_loss=0.0115, over 14868.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09406, pruned_loss=0.01472, audio_tagging_loss=0.009249, over 3044609.74 frames. ], batch size: 56, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 22:58:58,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2133446.6666666665, ans=0.0
2023-11-22 22:59:24,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2133580.0, ans=0.2
2023-11-22 22:59:33,172 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320050
2023-11-22 22:59:43,444 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.110e+01 8.793e+01 9.338e+01 3.637e+02, threshold=1.759e+02, percent-clipped=1.0
2023-11-22 22:59:47,387 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0
2023-11-22 22:59:57,615 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7450, loss[loss=0.06534, simple_loss=0.09148, pruned_loss=0.009952, audio_tagging_loss=0.009642, over 14608.00 frames. ], tot_loss[loss=0.07113, simple_loss=0.09436, pruned_loss=0.01477, audio_tagging_loss=0.009181, over 3040761.31 frames. ], batch size: 54, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 23:00:00,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.72 vs. limit=22.5
2023-11-22 23:00:04,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2133780.0, ans=0.0
2023-11-22 23:00:05,714 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.38 vs. limit=15.0
2023-11-22 23:00:12,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2133846.6666666665, ans=0.0
2023-11-22 23:00:30,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2133913.3333333335, ans=0.0
2023-11-22 23:00:38,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320100
2023-11-22 23:00:42,106 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:00:43,519 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0
2023-11-22 23:00:54,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2134046.6666666665, ans=10.0
2023-11-22 23:01:00,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2134113.3333333335, ans=0.125
2023-11-22 23:01:01,252 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7500, loss[loss=0.07024, simple_loss=0.08198, pruned_loss=0.01815, audio_tagging_loss=0.0111, over 15940.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09298, pruned_loss=0.01467, audio_tagging_loss=0.009055, over 3043138.80 frames. ], batch size: 63, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 23:01:03,013 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.10 vs. limit=22.5
2023-11-22 23:01:16,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2134180.0, ans=0.125
2023-11-22 23:01:41,521 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320150
2023-11-22 23:01:51,162 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.201e+01 8.778e+01 9.335e+01 1.224e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 23:02:05,934 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7550, loss[loss=0.0668, simple_loss=0.09109, pruned_loss=0.012, audio_tagging_loss=0.009254, over 14826.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09317, pruned_loss=0.01457, audio_tagging_loss=0.009109, over 3042719.77 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 8.0
2023-11-22 23:02:35,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2134580.0, ans=0.125
2023-11-22 23:02:41,169 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.45 vs. limit=22.5
2023-11-22 23:02:45,356 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320200
2023-11-22 23:03:10,729 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7600, loss[loss=0.08585, simple_loss=0.1261, pruned_loss=0.01651, audio_tagging_loss=0.006277, over 15303.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09303, pruned_loss=0.01452, audio_tagging_loss=0.009098, over 3043058.33 frames. ], batch size: 55, lr: 2.52e-03, grad_scale: 16.0
2023-11-22 23:03:24,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2134846.6666666665, ans=0.125
2023-11-22 23:03:31,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2134846.6666666665, ans=0.125
2023-11-22 23:03:45,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2134913.3333333335, ans=0.125
2023-11-22 23:03:49,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2134980.0, ans=0.125
2023-11-22 23:03:50,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320250
2023-11-22 23:03:50,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2134980.0, ans=0.125
2023-11-22 23:03:51,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2134980.0, ans=0.2
2023-11-22 23:04:00,562 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.533e+01 8.089e+01 8.781e+01 9.511e+01 1.232e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-22 23:04:05,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2135046.6666666665, ans=0.2
2023-11-22 23:04:10,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2135046.6666666665, ans=0.125
2023-11-22 23:04:13,896 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7650, loss[loss=0.06686, simple_loss=0.08244, pruned_loss=0.01329, audio_tagging_loss=0.01236, over 14410.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09332, pruned_loss=0.01463, audio_tagging_loss=0.009003, over 3049701.30 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:04:23,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2135113.3333333335, ans=0.025
2023-11-22 23:04:32,977 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.10 vs. limit=15.0
2023-11-22 23:04:46,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2135246.6666666665, ans=0.07
2023-11-22 23:04:47,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2135246.6666666665, ans=0.0
2023-11-22 23:04:53,120 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.90 vs. limit=10.0
2023-11-22 23:04:53,717 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320300
2023-11-22 23:05:18,677 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7700, loss[loss=0.09041, simple_loss=0.1261, pruned_loss=0.02075, audio_tagging_loss=0.006627, over 14485.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.0929, pruned_loss=0.01448, audio_tagging_loss=0.008974, over 3042312.26 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:05:19,332 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.13 vs. limit=15.0
2023-11-22 23:05:43,872 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.84 vs. limit=12.0
2023-11-22 23:05:44,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2135580.0, ans=0.125
2023-11-22 23:05:52,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2135580.0, ans=0.2
2023-11-22 23:05:59,447 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320350
2023-11-22 23:06:10,430 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.326e+01 9.020e+01 9.879e+01 1.267e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-22 23:06:10,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2135713.3333333335, ans=0.1
2023-11-22 23:06:25,291 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7750, loss[loss=0.08042, simple_loss=0.1021, pruned_loss=0.01851, audio_tagging_loss=0.01084, over 15578.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09254, pruned_loss=0.01433, audio_tagging_loss=0.009121, over 3041858.88 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:06:47,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2135846.6666666665, ans=0.125
2023-11-22 23:07:05,778 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320400
2023-11-22 23:07:16,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2136046.6666666665, ans=0.125
2023-11-22 23:07:17,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2136046.6666666665, ans=0.125
2023-11-22 23:07:19,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.99 vs. limit=10.0
2023-11-22 23:07:29,758 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7800, loss[loss=0.06353, simple_loss=0.07893, pruned_loss=0.01305, audio_tagging_loss=0.01103, over 14254.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.0925, pruned_loss=0.0144, audio_tagging_loss=0.009188, over 3038314.95 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:07:42,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2136180.0, ans=0.125
2023-11-22 23:07:45,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2136180.0, ans=0.2
2023-11-22 23:07:53,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=2136180.0, ans=0.025
2023-11-22 23:08:10,012 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320450
2023-11-22 23:08:19,801 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.736e+01 8.253e+01 8.929e+01 9.795e+01 1.214e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 23:08:33,322 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7850, loss[loss=0.05305, simple_loss=0.06067, pruned_loss=0.008876, audio_tagging_loss=0.01384, over 14540.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09159, pruned_loss=0.01405, audio_tagging_loss=0.00925, over 3048619.75 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:09:13,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2136646.6666666665, ans=0.2
2023-11-22 23:09:14,163 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320500
2023-11-22 23:09:14,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2136646.6666666665, ans=0.1
2023-11-22 23:09:15,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2136646.6666666665, ans=0.125
2023-11-22 23:09:18,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2136646.6666666665, ans=0.05
2023-11-22 23:09:29,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2136713.3333333335, ans=0.0
2023-11-22 23:09:33,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.21 vs. limit=10.0
2023-11-22 23:09:38,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2136780.0, ans=0.125
2023-11-22 23:09:39,732 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7900, loss[loss=0.06724, simple_loss=0.08857, pruned_loss=0.01193, audio_tagging_loss=0.01103, over 14780.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09246, pruned_loss=0.01417, audio_tagging_loss=0.00928, over 3045743.62 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:09:44,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2136780.0, ans=0.1
2023-11-22 23:10:00,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2136846.6666666665, ans=0.125
2023-11-22 23:10:01,295 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0
2023-11-22 23:10:07,429 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.88 vs. limit=22.5
2023-11-22 23:10:09,776 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:10:10,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2136913.3333333335, ans=0.125
2023-11-22 23:10:19,160 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320550
2023-11-22 23:10:20,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2136980.0, ans=0.2
2023-11-22 23:10:29,991 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.096e+01 8.900e+01 9.590e+01 1.237e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-22 23:10:30,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2137046.6666666665, ans=0.125
2023-11-22 23:10:35,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.38 vs. limit=12.0
2023-11-22 23:10:43,603 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 7950, loss[loss=0.05833, simple_loss=0.08268, pruned_loss=0.00889, audio_tagging_loss=0.008103, over 15212.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09204, pruned_loss=0.01421, audio_tagging_loss=0.009383, over 3041479.73 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:10:48,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2137113.3333333335, ans=0.125
2023-11-22 23:10:58,110 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:10:58,301 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:10:59,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2137180.0, ans=0.125
2023-11-22 23:11:11,003 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.04 vs. limit=12.0
2023-11-22 23:11:24,002 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320600
2023-11-22 23:11:31,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2137313.3333333335, ans=0.0
2023-11-22 23:11:47,660 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8000, loss[loss=0.06371, simple_loss=0.08069, pruned_loss=0.01583, audio_tagging_loss=0.007533, over 14660.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09144, pruned_loss=0.01421, audio_tagging_loss=0.009445, over 3033353.31 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:11:47,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2137446.6666666665, ans=0.0
2023-11-22 23:11:53,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2137446.6666666665, ans=0.125
2023-11-22 23:11:55,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2137446.6666666665, ans=0.125
2023-11-22 23:11:55,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2137446.6666666665, ans=0.125
2023-11-22 23:12:06,002 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.75 vs. limit=22.5
2023-11-22 23:12:17,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2137580.0, ans=0.0
2023-11-22 23:12:27,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320650
2023-11-22 23:12:34,886 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.38 vs. limit=6.0
2023-11-22 23:12:37,614 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.720e+01 8.271e+01 8.710e+01 9.655e+01 1.155e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-22 23:12:48,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2137713.3333333335, ans=0.0
2023-11-22 23:12:52,950 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8050, loss[loss=0.08012, simple_loss=0.1069, pruned_loss=0.01826, audio_tagging_loss=0.008438, over 15775.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09211, pruned_loss=0.01449, audio_tagging_loss=0.009494, over 3041930.42 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:12:58,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2137780.0, ans=0.0
2023-11-22 23:13:07,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2137846.6666666665, ans=0.0
2023-11-22 23:13:07,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2137846.6666666665, ans=0.0
2023-11-22 23:13:15,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2137846.6666666665, ans=0.1
2023-11-22 23:13:18,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2137913.3333333335, ans=0.125
2023-11-22 23:13:22,491 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.72 vs. limit=15.0
2023-11-22 23:13:31,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2137980.0, ans=0.0
2023-11-22 23:13:32,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320700
2023-11-22 23:13:57,274 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8100, loss[loss=0.0802, simple_loss=0.1162, pruned_loss=0.01517, audio_tagging_loss=0.006945, over 16475.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.0923, pruned_loss=0.01448, audio_tagging_loss=0.009454, over 3048292.11 frames. ], batch size: 62, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:14:05,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2138113.3333333335, ans=0.0
2023-11-22 23:14:21,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2138246.6666666665, ans=0.1
2023-11-22 23:14:28,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2138246.6666666665, ans=0.0
2023-11-22 23:14:37,233 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320750
2023-11-22 23:14:47,702 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.347e+01 8.289e+01 8.980e+01 9.464e+01 1.714e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-22 23:14:51,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2138380.0, ans=0.0
2023-11-22 23:15:01,315 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8150, loss[loss=0.08506, simple_loss=0.1153, pruned_loss=0.01788, audio_tagging_loss=0.009513, over 15282.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09234, pruned_loss=0.01462, audio_tagging_loss=0.009355, over 3046349.28 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:15:09,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2138446.6666666665, ans=0.125
2023-11-22 23:15:23,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2138513.3333333335, ans=0.0
2023-11-22 23:15:41,499 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320800
2023-11-22 23:15:56,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2138713.3333333335, ans=0.125
2023-11-22 23:16:02,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2138713.3333333335, ans=0.125
2023-11-22 23:16:04,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2138713.3333333335, ans=0.2
2023-11-22 23:16:06,091 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8200, loss[loss=0.09088, simple_loss=0.1259, pruned_loss=0.02065, audio_tagging_loss=0.007278, over 15314.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09247, pruned_loss=0.01456, audio_tagging_loss=0.009305, over 3046808.21 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:16:06,155 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:16:32,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2138913.3333333335, ans=0.0
2023-11-22 23:16:37,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2138913.3333333335, ans=0.125
2023-11-22 23:16:45,550 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320850
2023-11-22 23:16:47,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2138980.0, ans=0.0
2023-11-22 23:16:56,806 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.281e+01 9.046e+01 9.469e+01 1.170e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-22 23:17:06,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2139046.6666666665, ans=0.0
2023-11-22 23:17:11,168 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8250, loss[loss=0.06956, simple_loss=0.09346, pruned_loss=0.01124, audio_tagging_loss=0.01159, over 15634.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09193, pruned_loss=0.01445, audio_tagging_loss=0.009295, over 3041801.93 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:17:15,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2139113.3333333335, ans=0.125
2023-11-22 23:17:44,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2139246.6666666665, ans=0.125
2023-11-22 23:17:51,877 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320900
2023-11-22 23:18:15,864 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8300, loss[loss=0.0746, simple_loss=0.09753, pruned_loss=0.01733, audio_tagging_loss=0.008506, over 14115.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09212, pruned_loss=0.01447, audio_tagging_loss=0.009324, over 3044564.97 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:18:52,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2139580.0, ans=0.0
2023-11-22 23:18:56,506 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 320950
2023-11-22 23:19:05,961 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.263e+01 8.400e+01 9.007e+01 9.767e+01 1.180e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-22 23:19:08,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2139713.3333333335, ans=0.0
2023-11-22 23:19:18,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2139713.3333333335, ans=0.0
2023-11-22 23:19:20,887 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8350, loss[loss=0.06837, simple_loss=0.09248, pruned_loss=0.01364, audio_tagging_loss=0.008493, over 14981.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09166, pruned_loss=0.01433, audio_tagging_loss=0.009142, over 3049744.54 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:19:23,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2139780.0, ans=0.0
2023-11-22 23:19:55,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2139913.3333333335, ans=0.125
2023-11-22 23:19:59,975 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321000
2023-11-22 23:20:22,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2140046.6666666665, ans=0.0
2023-11-22 23:20:24,980 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8400, loss[loss=0.04904, simple_loss=0.0584, pruned_loss=0.009187, audio_tagging_loss=0.01065, over 14802.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09145, pruned_loss=0.01416, audio_tagging_loss=0.00914, over 3043349.05 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:20:25,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2140113.3333333335, ans=0.125
2023-11-22 23:20:30,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2140113.3333333335, ans=0.125
2023-11-22 23:21:00,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2140246.6666666665, ans=0.125
2023-11-22 23:21:00,381 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.82 vs. limit=15.0
2023-11-22 23:21:01,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2140246.6666666665, ans=0.125
2023-11-22 23:21:06,312 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.60 vs. limit=15.0
2023-11-22 23:21:06,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321050
2023-11-22 23:21:08,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2140313.3333333335, ans=0.125
2023-11-22 23:21:08,551 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.59 vs. limit=15.0
2023-11-22 23:21:13,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2140313.3333333335, ans=0.0
2023-11-22 23:21:17,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.27 vs. limit=15.0
2023-11-22 23:21:17,704 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.749e+01 8.024e+01 8.884e+01 9.515e+01 1.445e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-22 23:21:29,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2140446.6666666665, ans=0.125
2023-11-22 23:21:30,756 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8450, loss[loss=0.05816, simple_loss=0.08331, pruned_loss=0.007824, audio_tagging_loss=0.008682, over 16467.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09145, pruned_loss=0.01411, audio_tagging_loss=0.009113, over 3047444.97 frames. ], batch size: 60, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:21:31,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2140446.6666666665, ans=0.2
2023-11-22 23:22:11,052 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321100
2023-11-22 23:22:18,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.71 vs. limit=15.0
2023-11-22 23:22:21,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.68 vs. limit=15.0
2023-11-22 23:22:28,146 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.87 vs. limit=12.0
2023-11-22 23:22:36,201 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8500, loss[loss=0.07464, simple_loss=0.096, pruned_loss=0.01583, audio_tagging_loss=0.01082, over 14484.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09169, pruned_loss=0.01427, audio_tagging_loss=0.009152, over 3050171.34 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:22:36,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2140780.0, ans=0.125
2023-11-22 23:23:10,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2140913.3333333335, ans=0.125
2023-11-22 23:23:15,167 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321150
2023-11-22 23:23:27,510 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.243e+01 8.956e+01 9.576e+01 1.335e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-22 23:23:39,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2141113.3333333335, ans=10.0
2023-11-22 23:23:39,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2141113.3333333335, ans=0.125
2023-11-22 23:23:40,545 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8550, loss[loss=0.06817, simple_loss=0.09442, pruned_loss=0.01177, audio_tagging_loss=0.009194, over 14589.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09206, pruned_loss=0.01439, audio_tagging_loss=0.009212, over 3053291.38 frames. ], batch size: 53, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:23:40,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2141113.3333333335, ans=0.0
2023-11-22 23:23:44,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2141113.3333333335, ans=0.1
2023-11-22 23:23:50,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2141113.3333333335, ans=0.125
2023-11-22 23:24:00,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.22 vs. limit=15.0
2023-11-22 23:24:20,955 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321200
2023-11-22 23:24:22,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2141313.3333333335, ans=0.125
2023-11-22 23:24:37,930 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.54 vs. limit=15.0
2023-11-22 23:24:44,468 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8600, loss[loss=0.1036, simple_loss=0.1445, pruned_loss=0.024, audio_tagging_loss=0.007406, over 16162.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09239, pruned_loss=0.01446, audio_tagging_loss=0.009254, over 3060662.89 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:24:44,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2141446.6666666665, ans=0.025
2023-11-22 23:24:50,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.04 vs. limit=6.0
2023-11-22 23:24:55,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2141446.6666666665, ans=0.125
2023-11-22 23:25:07,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2141513.3333333335, ans=0.0
2023-11-22 23:25:25,090 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321250
2023-11-22 23:25:25,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2141646.6666666665, ans=0.2
2023-11-22 23:25:29,258 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.14 vs. limit=15.0
2023-11-22 23:25:36,062 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.429e+01 8.395e+01 8.985e+01 9.646e+01 1.201e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-22 23:25:49,430 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8650, loss[loss=0.08871, simple_loss=0.1277, pruned_loss=0.01814, audio_tagging_loss=0.006742, over 16313.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09353, pruned_loss=0.01444, audio_tagging_loss=0.009264, over 3055197.93 frames. ], batch size: 60, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:25:52,638 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.53 vs. limit=22.5
2023-11-22 23:26:20,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2141913.3333333335, ans=0.125
2023-11-22 23:26:28,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321300
2023-11-22 23:26:29,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=2141980.0, ans=0.5
2023-11-22 23:26:29,487 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.81 vs. limit=22.5
2023-11-22 23:26:31,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2141980.0, ans=0.125
2023-11-22 23:26:32,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2141980.0, ans=0.125
2023-11-22 23:26:35,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2141980.0, ans=0.125
2023-11-22 23:26:43,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2142046.6666666665, ans=0.125
2023-11-22 23:26:46,359 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.59 vs. limit=15.0
2023-11-22 23:26:50,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2142046.6666666665, ans=0.0
2023-11-22 23:26:54,812 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8700, loss[loss=0.08885, simple_loss=0.1289, pruned_loss=0.01944, audio_tagging_loss=0.004965, over 15897.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09321, pruned_loss=0.01431, audio_tagging_loss=0.009292, over 3056666.16 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:27:13,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2142180.0, ans=0.125
2023-11-22 23:27:17,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2142180.0, ans=0.025
2023-11-22 23:27:34,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321350
2023-11-22 23:27:44,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.13 vs. limit=15.0
2023-11-22 23:27:45,844 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.902e+01 8.323e+01 8.903e+01 9.633e+01 3.250e+02, threshold=1.781e+02, percent-clipped=1.0
2023-11-22 23:27:58,157 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8750, loss[loss=0.07973, simple_loss=0.1054, pruned_loss=0.01808, audio_tagging_loss=0.008941, over 15453.00 frames. ], tot_loss[loss=0.07117, simple_loss=0.09456, pruned_loss=0.01465, audio_tagging_loss=0.009235, over 3058121.10 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:28:34,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2142580.0, ans=0.07
2023-11-22 23:28:39,041 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321400
2023-11-22 23:28:43,809 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0
2023-11-22 23:29:03,162 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8800, loss[loss=0.05493, simple_loss=0.0728, pruned_loss=0.009989, audio_tagging_loss=0.008537, over 17074.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.09439, pruned_loss=0.01476, audio_tagging_loss=0.009329, over 3059755.08 frames. ], batch size: 64, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:29:05,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2142780.0, ans=0.125
2023-11-22 23:29:10,722 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.81 vs. limit=10.0
2023-11-22 23:29:35,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2142913.3333333335, ans=0.125
2023-11-22 23:29:36,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2142913.3333333335, ans=0.015
2023-11-22 23:29:44,341 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321450
2023-11-22 23:29:50,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2142980.0, ans=0.125
2023-11-22 23:29:54,926 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0
2023-11-22 23:29:56,623 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.814e+01 8.244e+01 8.962e+01 9.489e+01 1.229e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-22 23:30:10,467 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8850, loss[loss=0.07934, simple_loss=0.116, pruned_loss=0.01579, audio_tagging_loss=0.005563, over 14464.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09369, pruned_loss=0.0147, audio_tagging_loss=0.009406, over 3055050.05 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:30:19,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2143113.3333333335, ans=0.0
2023-11-22 23:30:21,725 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:30:50,365 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321500
2023-11-22 23:30:58,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2143313.3333333335, ans=0.125
2023-11-22 23:31:14,531 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8900, loss[loss=0.07493, simple_loss=0.1032, pruned_loss=0.01512, audio_tagging_loss=0.008212, over 15074.00 frames. ], tot_loss[loss=0.07184, simple_loss=0.09505, pruned_loss=0.01502, audio_tagging_loss=0.009294, over 3056397.89 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:31:28,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2143513.3333333335, ans=0.1
2023-11-22 23:31:55,151 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321550
2023-11-22 23:32:05,885 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.623e+01 8.181e+01 8.758e+01 9.474e+01 1.266e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-22 23:32:18,173 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 8950, loss[loss=0.08588, simple_loss=0.1133, pruned_loss=0.02102, audio_tagging_loss=0.008193, over 15387.00 frames. ], tot_loss[loss=0.07204, simple_loss=0.09564, pruned_loss=0.01509, audio_tagging_loss=0.009129, over 3049770.49 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:32:22,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2143780.0, ans=0.0
2023-11-22 23:32:33,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2143846.6666666665, ans=0.125
2023-11-22 23:33:00,151 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321600
2023-11-22 23:33:00,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2143980.0, ans=0.125
2023-11-22 23:33:12,528 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.57 vs. limit=10.0
2023-11-22 23:33:18,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2144046.6666666665, ans=0.0
2023-11-22 23:33:19,958 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.10 vs. limit=12.0
2023-11-22 23:33:26,997 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9000, loss[loss=0.0667, simple_loss=0.09183, pruned_loss=0.01358, audio_tagging_loss=0.007203, over 16797.00 frames. ], tot_loss[loss=0.07132, simple_loss=0.0947, pruned_loss=0.01488, audio_tagging_loss=0.009095, over 3045674.88 frames. ], batch size: 64, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:33:27,000 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-22 23:33:53,072 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.9538, 3.9539, 3.7766, 3.0425], device='cuda:0')
2023-11-22 23:33:56,366 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3541, 5.0200, 4.6931, 5.2007], device='cuda:0')
2023-11-22 23:34:00,516 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.8204, 5.8595, 5.9139, 5.8818], device='cuda:0')
2023-11-22 23:34:03,952 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4830, 3.2220, 3.6421, 3.4704], device='cuda:0')
2023-11-22 23:34:08,306 INFO [train_asr.py:1253] (0/4) Epoch 27, validation: loss=0.05906, simple_loss=0.05129, pruned_loss=0.005052, audio_tagging_loss=0.02836, over 4681554.00 frames. 
2023-11-22 23:34:08,307 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-22 23:34:37,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2144246.6666666665, ans=0.0
2023-11-22 23:34:44,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2144246.6666666665, ans=0.125
2023-11-22 23:34:49,970 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321650
2023-11-22 23:34:54,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.12 vs. limit=15.0
2023-11-22 23:34:55,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.52 vs. limit=15.0
2023-11-22 23:35:01,329 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.041e+01 8.419e+01 9.018e+01 9.927e+01 1.306e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-22 23:35:13,839 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9050, loss[loss=0.06108, simple_loss=0.07678, pruned_loss=0.01137, audio_tagging_loss=0.01132, over 15881.00 frames. ], tot_loss[loss=0.07159, simple_loss=0.09512, pruned_loss=0.01498, audio_tagging_loss=0.009055, over 3048659.06 frames. ], batch size: 60, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:35:41,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2144580.0, ans=0.0
2023-11-22 23:35:41,302 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:35:49,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2144580.0, ans=0.0
2023-11-22 23:35:54,728 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321700
2023-11-22 23:36:19,473 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9100, loss[loss=0.08138, simple_loss=0.1098, pruned_loss=0.01887, audio_tagging_loss=0.007613, over 15152.00 frames. ], tot_loss[loss=0.07124, simple_loss=0.09479, pruned_loss=0.01478, audio_tagging_loss=0.009062, over 3054275.71 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:36:51,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2144913.3333333335, ans=0.125
2023-11-22 23:36:56,410 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.50 vs. limit=15.0
2023-11-22 23:36:58,371 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321750
2023-11-22 23:37:11,733 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.773e+01 8.202e+01 8.707e+01 9.347e+01 1.081e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-22 23:37:22,642 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9150, loss[loss=0.05745, simple_loss=0.08046, pruned_loss=0.01082, audio_tagging_loss=0.006402, over 15457.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09394, pruned_loss=0.01465, audio_tagging_loss=0.00903, over 3053570.82 frames. ], batch size: 58, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:37:50,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2145246.6666666665, ans=0.0
2023-11-22 23:38:00,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2145313.3333333335, ans=0.09899494936611666
2023-11-22 23:38:02,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321800
2023-11-22 23:38:16,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2145380.0, ans=0.0
2023-11-22 23:38:26,078 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9200, loss[loss=0.07544, simple_loss=0.1024, pruned_loss=0.017, audio_tagging_loss=0.007217, over 15604.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09368, pruned_loss=0.01445, audio_tagging_loss=0.008938, over 3053455.20 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:38:30,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2145446.6666666665, ans=0.2
2023-11-22 23:39:06,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321850
2023-11-22 23:39:07,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2145646.6666666665, ans=0.0
2023-11-22 23:39:15,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.89 vs. limit=22.5
2023-11-22 23:39:18,718 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.111e+01 8.155e+01 8.815e+01 9.566e+01 1.382e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-22 23:39:30,953 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9250, loss[loss=0.07534, simple_loss=0.1031, pruned_loss=0.01456, audio_tagging_loss=0.009232, over 15821.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09331, pruned_loss=0.01441, audio_tagging_loss=0.008933, over 3057014.09 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:39:31,661 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.83 vs. limit=15.0
2023-11-22 23:39:55,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2145913.3333333335, ans=0.125
2023-11-22 23:40:01,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2145913.3333333335, ans=0.0
2023-11-22 23:40:07,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2145980.0, ans=0.0
2023-11-22 23:40:10,930 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321900
2023-11-22 23:40:35,085 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9300, loss[loss=0.0764, simple_loss=0.1042, pruned_loss=0.01615, audio_tagging_loss=0.008149, over 15369.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09255, pruned_loss=0.01425, audio_tagging_loss=0.009036, over 3055697.36 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:40:38,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2146113.3333333335, ans=0.125
2023-11-22 23:40:44,040 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.06 vs. limit=15.0
2023-11-22 23:40:58,166 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.95 vs. limit=15.0
2023-11-22 23:41:14,785 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 321950
2023-11-22 23:41:26,766 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.225e+01 8.828e+01 9.380e+01 1.148e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-22 23:41:37,955 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9350, loss[loss=0.0576, simple_loss=0.07453, pruned_loss=0.01014, audio_tagging_loss=0.0102, over 16772.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09253, pruned_loss=0.01444, audio_tagging_loss=0.00917, over 3048191.04 frames. ], batch size: 63, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:41:38,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.26 vs. limit=12.0
2023-11-22 23:42:14,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2146580.0, ans=0.2
2023-11-22 23:42:17,928 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322000
2023-11-22 23:42:21,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2146646.6666666665, ans=0.125
2023-11-22 23:42:23,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2146646.6666666665, ans=0.0
2023-11-22 23:42:23,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2146646.6666666665, ans=0.2
2023-11-22 23:42:32,211 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.54 vs. limit=22.5
2023-11-22 23:42:38,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2146713.3333333335, ans=0.125
2023-11-22 23:42:43,223 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9400, loss[loss=0.07132, simple_loss=0.0948, pruned_loss=0.01498, audio_tagging_loss=0.00894, over 14589.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09221, pruned_loss=0.01447, audio_tagging_loss=0.009297, over 3045186.96 frames. ], batch size: 54, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:42:44,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2146780.0, ans=0.125
2023-11-22 23:42:57,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2146846.6666666665, ans=0.2
2023-11-22 23:43:10,057 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.91 vs. limit=6.0
2023-11-22 23:43:20,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2146980.0, ans=0.95
2023-11-22 23:43:21,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2146980.0, ans=0.125
2023-11-22 23:43:22,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2146980.0, ans=0.125
2023-11-22 23:43:23,205 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322050
2023-11-22 23:43:24,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.12 vs. limit=6.0
2023-11-22 23:43:26,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2146980.0, ans=0.125
2023-11-22 23:43:36,904 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.391e+01 8.949e+01 9.601e+01 1.196e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-22 23:43:46,449 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:43:49,035 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9450, loss[loss=0.07063, simple_loss=0.09713, pruned_loss=0.01156, audio_tagging_loss=0.01051, over 16096.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09236, pruned_loss=0.01435, audio_tagging_loss=0.009397, over 3052440.03 frames. ], batch size: 61, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:43:54,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2147113.3333333335, ans=0.2
2023-11-22 23:43:55,497 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:43:58,585 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.47 vs. limit=15.0
2023-11-22 23:43:59,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2147113.3333333335, ans=0.0
2023-11-22 23:44:00,869 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.13 vs. limit=15.0
2023-11-22 23:44:29,025 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322100
2023-11-22 23:44:32,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2147313.3333333335, ans=0.125
2023-11-22 23:44:44,221 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.14 vs. limit=6.0
2023-11-22 23:44:53,054 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9500, loss[loss=0.07602, simple_loss=0.1102, pruned_loss=0.01329, audio_tagging_loss=0.007621, over 16862.00 frames. ], tot_loss[loss=0.07034, simple_loss=0.09287, pruned_loss=0.01451, audio_tagging_loss=0.0094, over 3054314.50 frames. ], batch size: 62, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:45:00,205 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.62 vs. limit=10.0
2023-11-22 23:45:02,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2147446.6666666665, ans=0.125
2023-11-22 23:45:06,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2147513.3333333335, ans=0.125
2023-11-22 23:45:32,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2147646.6666666665, ans=0.125
2023-11-22 23:45:33,399 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322150
2023-11-22 23:45:45,469 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.180e+01 8.349e+01 8.991e+01 9.634e+01 1.651e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-22 23:45:52,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2147713.3333333335, ans=0.0
2023-11-22 23:45:57,755 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9550, loss[loss=0.05208, simple_loss=0.0621, pruned_loss=0.009934, audio_tagging_loss=0.0111, over 16062.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09315, pruned_loss=0.01449, audio_tagging_loss=0.009384, over 3058378.95 frames. ], batch size: 62, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:46:03,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2147780.0, ans=0.0
2023-11-22 23:46:07,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2147780.0, ans=0.125
2023-11-22 23:46:14,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2147846.6666666665, ans=0.125
2023-11-22 23:46:17,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2147846.6666666665, ans=0.035
2023-11-22 23:46:26,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2147913.3333333335, ans=0.125
2023-11-22 23:46:36,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2147980.0, ans=0.125
2023-11-22 23:46:37,732 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322200
2023-11-22 23:46:42,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2147980.0, ans=0.0
2023-11-22 23:47:02,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2148113.3333333335, ans=0.1
2023-11-22 23:47:03,436 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9600, loss[loss=0.1065, simple_loss=0.1529, pruned_loss=0.02522, audio_tagging_loss=0.004838, over 15699.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09336, pruned_loss=0.01448, audio_tagging_loss=0.009488, over 3055018.62 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:47:05,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2148113.3333333335, ans=0.2
2023-11-22 23:47:13,487 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0
2023-11-22 23:47:22,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2148180.0, ans=0.2
2023-11-22 23:47:37,671 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.41 vs. limit=15.0
2023-11-22 23:47:43,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322250
2023-11-22 23:47:57,176 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.150e+01 8.727e+01 9.513e+01 1.269e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-22 23:47:59,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2148380.0, ans=0.1
2023-11-22 23:48:07,662 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9650, loss[loss=0.06521, simple_loss=0.08207, pruned_loss=0.01214, audio_tagging_loss=0.01203, over 14744.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09323, pruned_loss=0.01444, audio_tagging_loss=0.00946, over 3049735.02 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:48:12,886 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:48:47,552 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322300
2023-11-22 23:48:47,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2148646.6666666665, ans=0.125
2023-11-22 23:48:59,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2148713.3333333335, ans=0.125
2023-11-22 23:49:08,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2148713.3333333335, ans=0.0
2023-11-22 23:49:12,267 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9700, loss[loss=0.08515, simple_loss=0.1158, pruned_loss=0.01934, audio_tagging_loss=0.0079, over 14804.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.09384, pruned_loss=0.01454, audio_tagging_loss=0.009341, over 3051976.33 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:49:30,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2148846.6666666665, ans=0.125
2023-11-22 23:49:36,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2148913.3333333335, ans=0.125
2023-11-22 23:49:40,914 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.51 vs. limit=15.0
2023-11-22 23:49:52,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322350
2023-11-22 23:50:02,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2149046.6666666665, ans=0.0
2023-11-22 23:50:07,996 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.743e+01 8.299e+01 9.052e+01 9.780e+01 1.137e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-22 23:50:16,569 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9750, loss[loss=0.08177, simple_loss=0.1089, pruned_loss=0.01939, audio_tagging_loss=0.007938, over 15332.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09344, pruned_loss=0.01456, audio_tagging_loss=0.009264, over 3055212.44 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:50:18,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2149113.3333333335, ans=0.04949747468305833
2023-11-22 23:50:37,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2149180.0, ans=0.125
2023-11-22 23:50:56,619 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322400
2023-11-22 23:51:01,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.02 vs. limit=22.5
2023-11-22 23:51:10,111 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.38 vs. limit=22.5
2023-11-22 23:51:15,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2149380.0, ans=0.0
2023-11-22 23:51:17,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2149380.0, ans=0.125
2023-11-22 23:51:18,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2149380.0, ans=10.0
2023-11-22 23:51:20,650 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9800, loss[loss=0.0649, simple_loss=0.08385, pruned_loss=0.01351, audio_tagging_loss=0.009464, over 15417.00 frames. ], tot_loss[loss=0.07095, simple_loss=0.09401, pruned_loss=0.01472, audio_tagging_loss=0.00923, over 3052034.83 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:51:26,801 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0
2023-11-22 23:51:38,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2149513.3333333335, ans=0.125
2023-11-22 23:52:01,159 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322450
2023-11-22 23:52:01,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2149646.6666666665, ans=0.125
2023-11-22 23:52:16,548 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.583e+01 9.205e+01 9.862e+01 1.172e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-22 23:52:17,934 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:52:20,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2149713.3333333335, ans=0.125
2023-11-22 23:52:24,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2149780.0, ans=0.1
2023-11-22 23:52:25,272 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9850, loss[loss=0.06608, simple_loss=0.08777, pruned_loss=0.01269, audio_tagging_loss=0.009512, over 15650.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09377, pruned_loss=0.01465, audio_tagging_loss=0.00919, over 3057628.09 frames. ], batch size: 60, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:52:30,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2149780.0, ans=0.0
2023-11-22 23:52:47,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2149846.6666666665, ans=0.125
2023-11-22 23:52:54,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2149913.3333333335, ans=0.1
2023-11-22 23:52:56,099 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.26 vs. limit=6.0
2023-11-22 23:53:06,101 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322500
2023-11-22 23:53:31,654 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9900, loss[loss=0.07271, simple_loss=0.09378, pruned_loss=0.0151, audio_tagging_loss=0.01072, over 14766.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09397, pruned_loss=0.01473, audio_tagging_loss=0.009172, over 3059109.51 frames. ], batch size: 57, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:53:43,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.40 vs. limit=10.0
2023-11-22 23:53:49,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2150180.0, ans=0.125
2023-11-22 23:54:08,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2150246.6666666665, ans=0.1
2023-11-22 23:54:12,791 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322550
2023-11-22 23:54:26,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2150380.0, ans=0.0
2023-11-22 23:54:27,621 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.729e+01 8.162e+01 8.931e+01 9.625e+01 1.127e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-22 23:54:36,340 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 9950, loss[loss=0.09144, simple_loss=0.1274, pruned_loss=0.01958, audio_tagging_loss=0.008169, over 15423.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.0939, pruned_loss=0.01458, audio_tagging_loss=0.009211, over 3056205.11 frames. ], batch size: 54, lr: 2.51e-03, grad_scale: 16.0
2023-11-22 23:54:47,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2150446.6666666665, ans=0.125
2023-11-22 23:54:58,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2150513.3333333335, ans=0.1
2023-11-22 23:55:16,701 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322600
2023-11-22 23:55:32,042 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-22 23:55:41,023 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10000, loss[loss=0.05183, simple_loss=0.06979, pruned_loss=0.007076, audio_tagging_loss=0.009861, over 16214.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09406, pruned_loss=0.01452, audio_tagging_loss=0.009101, over 3056290.91 frames. ], batch size: 62, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:55:50,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2150780.0, ans=0.1
2023-11-22 23:55:50,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2150780.0, ans=0.2
2023-11-22 23:56:21,736 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322650
2023-11-22 23:56:27,661 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.60 vs. limit=10.0
2023-11-22 23:56:38,550 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.648e+01 8.099e+01 8.716e+01 9.532e+01 1.465e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-22 23:56:47,901 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10050, loss[loss=0.07165, simple_loss=0.09074, pruned_loss=0.01859, audio_tagging_loss=0.007684, over 15308.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.0933, pruned_loss=0.01439, audio_tagging_loss=0.009145, over 3051008.64 frames. ], batch size: 56, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:57:16,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2151246.6666666665, ans=0.125
2023-11-22 23:57:27,985 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322700
2023-11-22 23:57:28,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2151313.3333333335, ans=0.025
2023-11-22 23:57:36,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2151313.3333333335, ans=0.125
2023-11-22 23:57:44,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2151380.0, ans=0.125
2023-11-22 23:57:44,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2151380.0, ans=0.125
2023-11-22 23:57:52,570 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10100, loss[loss=0.07616, simple_loss=0.1065, pruned_loss=0.01391, audio_tagging_loss=0.009007, over 16274.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09246, pruned_loss=0.01424, audio_tagging_loss=0.009236, over 3054873.66 frames. ], batch size: 59, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:57:58,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.55 vs. limit=15.0
2023-11-22 23:58:00,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2151446.6666666665, ans=0.1
2023-11-22 23:58:02,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2151446.6666666665, ans=0.125
2023-11-22 23:58:09,310 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.48 vs. limit=15.0
2023-11-22 23:58:33,898 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322750
2023-11-22 23:58:36,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2151646.6666666665, ans=0.125
2023-11-22 23:58:41,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2151646.6666666665, ans=0.1
2023-11-22 23:58:44,960 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:58:48,678 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.658e+01 8.081e+01 8.819e+01 9.576e+01 1.489e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-22 23:58:57,826 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10150, loss[loss=0.07883, simple_loss=0.102, pruned_loss=0.01593, audio_tagging_loss=0.01188, over 14870.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09249, pruned_loss=0.01422, audio_tagging_loss=0.009233, over 3056459.84 frames. ], batch size: 55, lr: 2.51e-03, grad_scale: 32.0
2023-11-22 23:59:01,040 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.76 vs. limit=6.0
2023-11-22 23:59:09,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2151780.0, ans=0.0
2023-11-22 23:59:25,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2151913.3333333335, ans=0.125
2023-11-22 23:59:28,824 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-22 23:59:36,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2151980.0, ans=0.1
2023-11-22 23:59:38,957 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322800
2023-11-22 23:59:45,264 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.43 vs. limit=15.0
2023-11-22 23:59:52,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2152046.6666666665, ans=0.0
2023-11-22 23:59:56,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2152046.6666666665, ans=0.125
2023-11-23 00:00:04,832 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10200, loss[loss=0.05514, simple_loss=0.06894, pruned_loss=0.01317, audio_tagging_loss=0.007505, over 15750.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09272, pruned_loss=0.01431, audio_tagging_loss=0.009336, over 3054736.44 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:00:08,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.41 vs. limit=5.0
2023-11-23 00:00:12,670 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.23 vs. limit=12.0
2023-11-23 00:00:27,271 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 00:00:32,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2152246.6666666665, ans=0.0
2023-11-23 00:00:43,330 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322850
2023-11-23 00:00:55,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2152380.0, ans=0.125
2023-11-23 00:00:59,981 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.282e+01 8.917e+01 9.345e+01 1.171e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 00:01:01,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2152380.0, ans=0.0
2023-11-23 00:01:02,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2152380.0, ans=0.125
2023-11-23 00:01:06,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2152380.0, ans=0.1
2023-11-23 00:01:08,803 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10250, loss[loss=0.07537, simple_loss=0.09856, pruned_loss=0.01687, audio_tagging_loss=0.009214, over 15130.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09318, pruned_loss=0.01445, audio_tagging_loss=0.009411, over 3056955.39 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:01:14,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2152446.6666666665, ans=0.0
2023-11-23 00:01:35,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2152580.0, ans=0.125
2023-11-23 00:01:43,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2152580.0, ans=0.0
2023-11-23 00:01:49,921 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322900
2023-11-23 00:01:52,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2152646.6666666665, ans=0.0
2023-11-23 00:02:13,745 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10300, loss[loss=0.06882, simple_loss=0.09929, pruned_loss=0.0102, audio_tagging_loss=0.008967, over 15206.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09311, pruned_loss=0.01449, audio_tagging_loss=0.009402, over 3057708.94 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:02:26,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2152846.6666666665, ans=0.0
2023-11-23 00:02:31,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2152846.6666666665, ans=0.125
2023-11-23 00:02:39,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2152913.3333333335, ans=0.125
2023-11-23 00:02:48,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2152913.3333333335, ans=0.125
2023-11-23 00:02:53,991 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 322950
2023-11-23 00:02:57,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2152980.0, ans=0.125
2023-11-23 00:03:01,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2152980.0, ans=10.0
2023-11-23 00:03:02,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2152980.0, ans=0.1
2023-11-23 00:03:02,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=2152980.0, ans=0.1
2023-11-23 00:03:08,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2153046.6666666665, ans=0.125
2023-11-23 00:03:10,241 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.589e+01 8.288e+01 8.993e+01 9.603e+01 1.175e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 00:03:19,116 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10350, loss[loss=0.0752, simple_loss=0.1067, pruned_loss=0.01205, audio_tagging_loss=0.009827, over 15222.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09336, pruned_loss=0.01437, audio_tagging_loss=0.009428, over 3057246.24 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:03:23,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2153113.3333333335, ans=0.0
2023-11-23 00:03:37,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2153180.0, ans=10.0
2023-11-23 00:03:46,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.38 vs. limit=15.0
2023-11-23 00:03:48,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2153246.6666666665, ans=0.0
2023-11-23 00:03:58,566 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323000
2023-11-23 00:04:02,174 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.12 vs. limit=15.0
2023-11-23 00:04:17,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2153380.0, ans=0.1
2023-11-23 00:04:19,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2153380.0, ans=0.0
2023-11-23 00:04:24,475 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10400, loss[loss=0.0898, simple_loss=0.1097, pruned_loss=0.02638, audio_tagging_loss=0.00858, over 14889.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09274, pruned_loss=0.01421, audio_tagging_loss=0.009502, over 3060059.58 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:04:32,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2153446.6666666665, ans=0.0
2023-11-23 00:04:42,753 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.10 vs. limit=6.0
2023-11-23 00:04:47,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2153513.3333333335, ans=0.2
2023-11-23 00:05:05,672 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323050
2023-11-23 00:05:05,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2153646.6666666665, ans=0.125
2023-11-23 00:05:10,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2153646.6666666665, ans=0.1
2023-11-23 00:05:13,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2153646.6666666665, ans=0.125
2023-11-23 00:05:21,771 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.737e+01 8.434e+01 9.021e+01 9.797e+01 1.156e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 00:05:29,011 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10450, loss[loss=0.06968, simple_loss=0.09017, pruned_loss=0.01707, audio_tagging_loss=0.00752, over 14482.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09327, pruned_loss=0.01428, audio_tagging_loss=0.009492, over 3051556.06 frames. ], batch size: 54, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:05:39,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2153780.0, ans=0.0
2023-11-23 00:05:40,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2153846.6666666665, ans=0.125
2023-11-23 00:05:44,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2153846.6666666665, ans=0.125
2023-11-23 00:06:09,877 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323100
2023-11-23 00:06:11,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2153980.0, ans=0.125
2023-11-23 00:06:11,363 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:06:18,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2153980.0, ans=0.0
2023-11-23 00:06:21,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2154046.6666666665, ans=0.125
2023-11-23 00:06:21,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2154046.6666666665, ans=0.125
2023-11-23 00:06:34,673 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10500, loss[loss=0.07398, simple_loss=0.1005, pruned_loss=0.0146, audio_tagging_loss=0.009142, over 15420.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09331, pruned_loss=0.01439, audio_tagging_loss=0.009361, over 3051744.79 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:07:04,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2154246.6666666665, ans=0.1
2023-11-23 00:07:14,265 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323150
2023-11-23 00:07:31,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0
2023-11-23 00:07:32,103 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.17 vs. limit=6.0
2023-11-23 00:07:32,622 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.185e+01 8.836e+01 9.458e+01 1.353e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 00:07:39,226 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10550, loss[loss=0.06217, simple_loss=0.08778, pruned_loss=0.009976, audio_tagging_loss=0.008306, over 14143.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09295, pruned_loss=0.01437, audio_tagging_loss=0.009405, over 3054137.54 frames. ], batch size: 54, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:07:50,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2154513.3333333335, ans=0.125
2023-11-23 00:07:57,317 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.64 vs. limit=5.0
2023-11-23 00:08:19,446 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323200
2023-11-23 00:08:28,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2154646.6666666665, ans=0.125
2023-11-23 00:08:33,057 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:08:43,892 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10600, loss[loss=0.06221, simple_loss=0.09386, pruned_loss=0.008732, audio_tagging_loss=0.006553, over 14796.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09276, pruned_loss=0.01434, audio_tagging_loss=0.009186, over 3051005.56 frames. ], batch size: 52, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:08:45,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2154780.0, ans=0.035
2023-11-23 00:08:49,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2154780.0, ans=0.1
2023-11-23 00:08:49,611 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.94 vs. limit=15.0
2023-11-23 00:09:05,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2154846.6666666665, ans=0.125
2023-11-23 00:09:24,556 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323250
2023-11-23 00:09:33,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.58 vs. limit=15.0
2023-11-23 00:09:41,928 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.748e+01 7.982e+01 8.705e+01 9.514e+01 1.291e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-23 00:09:46,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2155046.6666666665, ans=0.0
2023-11-23 00:09:48,204 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10650, loss[loss=0.0662, simple_loss=0.08406, pruned_loss=0.0143, audio_tagging_loss=0.009868, over 14785.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09227, pruned_loss=0.01445, audio_tagging_loss=0.009244, over 3043855.11 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:10:28,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323300
2023-11-23 00:10:33,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2155313.3333333335, ans=0.125
2023-11-23 00:10:34,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2155313.3333333335, ans=0.2
2023-11-23 00:10:45,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2155380.0, ans=0.125
2023-11-23 00:10:48,773 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.38 vs. limit=15.0
2023-11-23 00:10:52,987 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10700, loss[loss=0.06703, simple_loss=0.08578, pruned_loss=0.01203, audio_tagging_loss=0.01211, over 15197.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09294, pruned_loss=0.01456, audio_tagging_loss=0.009142, over 3048193.02 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:10:57,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2155446.6666666665, ans=0.0
2023-11-23 00:11:31,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2155646.6666666665, ans=0.0
2023-11-23 00:11:33,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323350
2023-11-23 00:11:48,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2155713.3333333335, ans=0.125
2023-11-23 00:11:51,218 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.154e+01 8.901e+01 9.766e+01 1.298e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 00:11:51,612 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:11:55,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2155713.3333333335, ans=0.0
2023-11-23 00:11:57,404 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10750, loss[loss=0.06766, simple_loss=0.0972, pruned_loss=0.01002, audio_tagging_loss=0.009043, over 14910.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.0932, pruned_loss=0.0145, audio_tagging_loss=0.00912, over 3047479.52 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:12:02,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2155780.0, ans=0.1
2023-11-23 00:12:26,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.17 vs. limit=10.0
2023-11-23 00:12:32,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2155913.3333333335, ans=0.125
2023-11-23 00:12:34,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2155980.0, ans=0.125
2023-11-23 00:12:37,277 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323400
2023-11-23 00:12:42,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2155980.0, ans=0.125
2023-11-23 00:12:58,459 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.34 vs. limit=10.0
2023-11-23 00:12:58,646 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.14 vs. limit=15.0
2023-11-23 00:13:01,571 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10800, loss[loss=0.08667, simple_loss=0.1206, pruned_loss=0.01895, audio_tagging_loss=0.007413, over 14426.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09322, pruned_loss=0.01442, audio_tagging_loss=0.009136, over 3050472.71 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:13:07,440 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:13:11,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2156113.3333333335, ans=0.2
2023-11-23 00:13:22,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2156180.0, ans=0.125
2023-11-23 00:13:25,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2156180.0, ans=0.125
2023-11-23 00:13:25,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2156180.0, ans=0.125
2023-11-23 00:13:37,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2156246.6666666665, ans=0.125
2023-11-23 00:13:37,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2156246.6666666665, ans=0.125
2023-11-23 00:13:41,995 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323450
2023-11-23 00:13:42,762 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.88 vs. limit=15.0
2023-11-23 00:13:49,859 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:13:52,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2156380.0, ans=0.125
2023-11-23 00:13:57,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2156380.0, ans=0.125
2023-11-23 00:14:00,558 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.240e+01 8.825e+01 9.425e+01 1.162e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 00:14:06,751 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10850, loss[loss=0.06773, simple_loss=0.08748, pruned_loss=0.01321, audio_tagging_loss=0.01078, over 14993.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09339, pruned_loss=0.01453, audio_tagging_loss=0.009077, over 3049122.72 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:14:09,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2156446.6666666665, ans=0.2
2023-11-23 00:14:28,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.out_whiten.whitening_limit, batch_count=2156513.3333333335, ans=8.0
2023-11-23 00:14:40,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2156580.0, ans=0.0
2023-11-23 00:14:46,712 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323500
2023-11-23 00:14:54,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2156646.6666666665, ans=0.05
2023-11-23 00:15:03,905 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.27 vs. limit=15.0
2023-11-23 00:15:06,855 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 00:15:07,063 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:15:10,534 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10900, loss[loss=0.08053, simple_loss=0.1043, pruned_loss=0.02067, audio_tagging_loss=0.007698, over 15183.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09258, pruned_loss=0.01444, audio_tagging_loss=0.009131, over 3050068.58 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:15:15,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2156780.0, ans=0.125
2023-11-23 00:15:25,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2156846.6666666665, ans=0.0
2023-11-23 00:15:45,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2156913.3333333335, ans=0.125
2023-11-23 00:15:51,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323550
2023-11-23 00:15:52,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2156980.0, ans=0.125
2023-11-23 00:15:56,609 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:16:01,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2157046.6666666665, ans=0.0
2023-11-23 00:16:10,513 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.707e+01 8.063e+01 8.887e+01 9.492e+01 1.128e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 00:16:15,464 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 10950, loss[loss=0.07959, simple_loss=0.1015, pruned_loss=0.01865, audio_tagging_loss=0.01019, over 14730.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09129, pruned_loss=0.01429, audio_tagging_loss=0.009199, over 3045100.81 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:16:17,365 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.65 vs. limit=22.5
2023-11-23 00:16:19,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2157113.3333333335, ans=0.125
2023-11-23 00:16:26,565 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.81 vs. limit=15.0
2023-11-23 00:16:36,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2157180.0, ans=0.125
2023-11-23 00:16:46,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2157246.6666666665, ans=0.1
2023-11-23 00:16:47,707 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:16:54,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2157313.3333333335, ans=0.04949747468305833
2023-11-23 00:16:55,448 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323600
2023-11-23 00:16:58,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2157313.3333333335, ans=0.2
2023-11-23 00:17:01,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2157313.3333333335, ans=0.04949747468305833
2023-11-23 00:17:04,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2157313.3333333335, ans=0.125
2023-11-23 00:17:13,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2157380.0, ans=0.125
2023-11-23 00:17:14,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2157380.0, ans=0.125
2023-11-23 00:17:19,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2157446.6666666665, ans=0.125
2023-11-23 00:17:20,239 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11000, loss[loss=0.07398, simple_loss=0.1037, pruned_loss=0.01283, audio_tagging_loss=0.009286, over 16043.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.0918, pruned_loss=0.01425, audio_tagging_loss=0.009294, over 3048381.12 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:17:30,175 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 00:17:34,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2157513.3333333335, ans=0.0
2023-11-23 00:17:37,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2157513.3333333335, ans=0.0
2023-11-23 00:17:39,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2157513.3333333335, ans=0.2
2023-11-23 00:17:59,834 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323650
2023-11-23 00:18:18,918 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.419e+01 9.027e+01 1.014e+02 1.259e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 00:18:24,031 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11050, loss[loss=0.07182, simple_loss=0.09156, pruned_loss=0.01844, audio_tagging_loss=0.007602, over 14034.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09166, pruned_loss=0.01433, audio_tagging_loss=0.009408, over 3050466.47 frames. ], batch size: 53, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:18:42,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=2157846.6666666665, ans=0.02
2023-11-23 00:18:43,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2157846.6666666665, ans=0.1
2023-11-23 00:18:57,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2157913.3333333335, ans=0.2
2023-11-23 00:19:04,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323700
2023-11-23 00:19:28,773 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11100, loss[loss=0.08496, simple_loss=0.1151, pruned_loss=0.02091, audio_tagging_loss=0.006518, over 15814.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09292, pruned_loss=0.0147, audio_tagging_loss=0.009407, over 3047971.05 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:19:29,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2158113.3333333335, ans=0.125
2023-11-23 00:19:31,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2158113.3333333335, ans=0.125
2023-11-23 00:19:39,745 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.05 vs. limit=22.5
2023-11-23 00:19:43,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2158180.0, ans=0.0
2023-11-23 00:19:47,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2158180.0, ans=0.0
2023-11-23 00:19:48,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2158180.0, ans=0.125
2023-11-23 00:19:53,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2158246.6666666665, ans=0.125
2023-11-23 00:20:01,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2158246.6666666665, ans=0.1
2023-11-23 00:20:05,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2158246.6666666665, ans=0.2
2023-11-23 00:20:08,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323750
2023-11-23 00:20:15,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2158313.3333333335, ans=0.0
2023-11-23 00:20:21,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2158380.0, ans=0.05
2023-11-23 00:20:28,956 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.251e+01 8.340e+01 9.049e+01 9.831e+01 1.206e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 00:20:34,614 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11150, loss[loss=0.07085, simple_loss=0.1021, pruned_loss=0.01179, audio_tagging_loss=0.008029, over 16019.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09351, pruned_loss=0.01473, audio_tagging_loss=0.009419, over 3051721.72 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:20:36,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2158446.6666666665, ans=0.125
2023-11-23 00:21:01,542 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.04 vs. limit=15.0
2023-11-23 00:21:14,164 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323800
2023-11-23 00:21:39,505 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11200, loss[loss=0.05313, simple_loss=0.07438, pruned_loss=0.007765, audio_tagging_loss=0.008178, over 15731.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09268, pruned_loss=0.01427, audio_tagging_loss=0.009412, over 3052320.59 frames. ], batch size: 60, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:21:46,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2158780.0, ans=0.0
2023-11-23 00:21:53,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2158846.6666666665, ans=0.125
2023-11-23 00:22:17,673 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.44 vs. limit=15.0
2023-11-23 00:22:20,913 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323850
2023-11-23 00:22:28,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2158980.0, ans=0.125
2023-11-23 00:22:37,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2159046.6666666665, ans=0.125
2023-11-23 00:22:39,531 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.459e+01 8.191e+01 8.566e+01 9.520e+01 1.333e+02, threshold=1.713e+02, percent-clipped=0.0
2023-11-23 00:22:43,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2159113.3333333335, ans=0.035
2023-11-23 00:22:44,553 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11250, loss[loss=0.08203, simple_loss=0.1103, pruned_loss=0.02031, audio_tagging_loss=0.00654, over 15020.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09211, pruned_loss=0.01421, audio_tagging_loss=0.00948, over 3052585.27 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:23:08,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2159180.0, ans=0.125
2023-11-23 00:23:25,496 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323900
2023-11-23 00:23:50,755 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11300, loss[loss=0.064, simple_loss=0.07804, pruned_loss=0.0131, audio_tagging_loss=0.01188, over 15303.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09112, pruned_loss=0.01415, audio_tagging_loss=0.009406, over 3060207.03 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:24:04,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2159513.3333333335, ans=0.125
2023-11-23 00:24:07,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2159513.3333333335, ans=0.125
2023-11-23 00:24:29,413 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 323950
2023-11-23 00:24:40,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2159646.6666666665, ans=0.125
2023-11-23 00:24:45,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2159713.3333333335, ans=0.0
2023-11-23 00:24:51,670 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.888e+01 8.358e+01 9.001e+01 9.797e+01 1.218e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 00:24:55,601 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11350, loss[loss=0.0609, simple_loss=0.07545, pruned_loss=0.01104, audio_tagging_loss=0.01214, over 14787.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09079, pruned_loss=0.0142, audio_tagging_loss=0.009332, over 3047687.68 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:25:08,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2159846.6666666665, ans=0.125
2023-11-23 00:25:15,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=2159846.6666666665, ans=15.0
2023-11-23 00:25:27,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2159913.3333333335, ans=0.125
2023-11-23 00:25:36,770 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324000
2023-11-23 00:25:38,297 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-324000.pt
2023-11-23 00:25:50,426 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.25 vs. limit=12.0
2023-11-23 00:25:51,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2160046.6666666665, ans=0.2
2023-11-23 00:26:03,554 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11400, loss[loss=0.06186, simple_loss=0.0723, pruned_loss=0.01372, audio_tagging_loss=0.01199, over 14276.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09079, pruned_loss=0.01418, audio_tagging_loss=0.00926, over 3041877.24 frames. ], batch size: 54, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:26:03,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2160113.3333333335, ans=0.1
2023-11-23 00:26:17,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2160180.0, ans=0.0
2023-11-23 00:26:34,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2160246.6666666665, ans=0.2
2023-11-23 00:26:44,792 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324050
2023-11-23 00:27:05,050 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.089e+01 8.744e+01 9.515e+01 1.376e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-23 00:27:09,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2160446.6666666665, ans=0.125
2023-11-23 00:27:10,222 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11450, loss[loss=0.06685, simple_loss=0.08202, pruned_loss=0.01394, audio_tagging_loss=0.01189, over 14910.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09091, pruned_loss=0.01419, audio_tagging_loss=0.009217, over 3038283.47 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:27:20,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2160446.6666666665, ans=0.1
2023-11-23 00:27:36,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2160580.0, ans=0.0
2023-11-23 00:27:45,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2160580.0, ans=0.125
2023-11-23 00:27:50,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324100
2023-11-23 00:27:55,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2160646.6666666665, ans=0.1
2023-11-23 00:28:05,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2160713.3333333335, ans=0.1
2023-11-23 00:28:16,251 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11500, loss[loss=0.08258, simple_loss=0.1181, pruned_loss=0.01528, audio_tagging_loss=0.008233, over 16147.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.0919, pruned_loss=0.01423, audio_tagging_loss=0.009163, over 3041968.29 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:28:31,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2160846.6666666665, ans=0.0
2023-11-23 00:28:35,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2160846.6666666665, ans=0.125
2023-11-23 00:28:41,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2160913.3333333335, ans=0.2
2023-11-23 00:28:50,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2160913.3333333335, ans=0.0
2023-11-23 00:28:57,810 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324150
2023-11-23 00:29:03,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2160980.0, ans=0.1
2023-11-23 00:29:18,299 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 7.992e+01 8.677e+01 9.342e+01 1.172e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-23 00:29:22,162 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11550, loss[loss=0.08341, simple_loss=0.1077, pruned_loss=0.02202, audio_tagging_loss=0.00753, over 14907.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09187, pruned_loss=0.01422, audio_tagging_loss=0.009154, over 3046395.39 frames. ], batch size: 58, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:29:32,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2161113.3333333335, ans=0.125
2023-11-23 00:29:43,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2161180.0, ans=0.2
2023-11-23 00:29:53,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2161246.6666666665, ans=0.125
2023-11-23 00:29:58,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2161246.6666666665, ans=0.07
2023-11-23 00:30:03,380 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 00:30:04,653 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324200
2023-11-23 00:30:04,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2161313.3333333335, ans=0.125
2023-11-23 00:30:12,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2161313.3333333335, ans=0.125
2023-11-23 00:30:29,748 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11600, loss[loss=0.04697, simple_loss=0.05921, pruned_loss=0.005033, audio_tagging_loss=0.01233, over 14703.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09271, pruned_loss=0.01424, audio_tagging_loss=0.00909, over 3054883.16 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:30:42,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2161446.6666666665, ans=0.0
2023-11-23 00:31:05,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2161580.0, ans=0.125
2023-11-23 00:31:07,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2161580.0, ans=0.125
2023-11-23 00:31:10,701 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324250
2023-11-23 00:31:11,289 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.07 vs. limit=15.0
2023-11-23 00:31:26,234 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.92 vs. limit=12.0
2023-11-23 00:31:32,554 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.566e+01 8.394e+01 8.884e+01 9.610e+01 1.253e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 00:31:36,391 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11650, loss[loss=0.06882, simple_loss=0.08999, pruned_loss=0.01439, audio_tagging_loss=0.009437, over 16445.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.0934, pruned_loss=0.01455, audio_tagging_loss=0.009151, over 3053785.09 frames. ], batch size: 64, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:31:45,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2161780.0, ans=0.125
2023-11-23 00:31:50,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2161846.6666666665, ans=0.1
2023-11-23 00:31:55,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2161846.6666666665, ans=0.125
2023-11-23 00:32:17,581 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324300
2023-11-23 00:32:42,006 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11700, loss[loss=0.05985, simple_loss=0.07705, pruned_loss=0.01026, audio_tagging_loss=0.01107, over 14643.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09283, pruned_loss=0.01456, audio_tagging_loss=0.009228, over 3049420.77 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:32:54,099 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=22.5
2023-11-23 00:32:57,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2162180.0, ans=0.1
2023-11-23 00:33:13,214 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.09 vs. limit=15.0
2023-11-23 00:33:23,482 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324350
2023-11-23 00:33:43,248 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.852e+01 8.270e+01 8.836e+01 9.259e+01 1.366e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 00:33:45,297 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.82 vs. limit=15.0
2023-11-23 00:33:46,986 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11750, loss[loss=0.06046, simple_loss=0.06993, pruned_loss=0.01252, audio_tagging_loss=0.01298, over 15368.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09202, pruned_loss=0.0145, audio_tagging_loss=0.009334, over 3044535.81 frames. ], batch size: 59, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:34:05,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2162513.3333333335, ans=0.125
2023-11-23 00:34:17,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2162580.0, ans=0.125
2023-11-23 00:34:25,353 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.37 vs. limit=12.0
2023-11-23 00:34:28,648 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324400
2023-11-23 00:34:54,194 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11800, loss[loss=0.09793, simple_loss=0.1374, pruned_loss=0.02287, audio_tagging_loss=0.006372, over 15417.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09284, pruned_loss=0.01464, audio_tagging_loss=0.009327, over 3042114.92 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:34:56,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2162780.0, ans=0.1
2023-11-23 00:35:13,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2162846.6666666665, ans=0.1
2023-11-23 00:35:23,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2162913.3333333335, ans=0.07
2023-11-23 00:35:34,986 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324450
2023-11-23 00:35:41,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2162980.0, ans=0.0
2023-11-23 00:35:41,985 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:35:46,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2163046.6666666665, ans=0.125
2023-11-23 00:35:49,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2163046.6666666665, ans=0.125
2023-11-23 00:35:57,200 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.148e+01 8.225e+01 9.023e+01 9.525e+01 1.272e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 00:35:59,720 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11850, loss[loss=0.07748, simple_loss=0.09642, pruned_loss=0.0206, audio_tagging_loss=0.008673, over 15518.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.0931, pruned_loss=0.0146, audio_tagging_loss=0.009474, over 3034945.54 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:36:05,561 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.87 vs. limit=15.0
2023-11-23 00:36:12,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2163180.0, ans=0.1
2023-11-23 00:36:14,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2163180.0, ans=0.0
2023-11-23 00:36:18,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2163180.0, ans=0.1
2023-11-23 00:36:20,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2163180.0, ans=0.05
2023-11-23 00:36:21,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2163180.0, ans=0.1
2023-11-23 00:36:35,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2163246.6666666665, ans=0.95
2023-11-23 00:36:40,357 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324500
2023-11-23 00:36:52,744 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.65 vs. limit=15.0
2023-11-23 00:36:59,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2163380.0, ans=0.0
2023-11-23 00:37:04,527 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11900, loss[loss=0.06088, simple_loss=0.08645, pruned_loss=0.01004, audio_tagging_loss=0.00762, over 14960.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09352, pruned_loss=0.01463, audio_tagging_loss=0.009465, over 3037318.24 frames. ], batch size: 56, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:37:23,130 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.43 vs. limit=10.0
2023-11-23 00:37:45,545 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324550
2023-11-23 00:37:55,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2163713.3333333335, ans=0.0
2023-11-23 00:38:02,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2163713.3333333335, ans=0.0
2023-11-23 00:38:08,411 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 8.374e+01 8.930e+01 9.704e+01 2.155e+02, threshold=1.786e+02, percent-clipped=1.0
2023-11-23 00:38:10,888 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 11950, loss[loss=0.06931, simple_loss=0.0879, pruned_loss=0.0164, audio_tagging_loss=0.008962, over 15291.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09386, pruned_loss=0.01475, audio_tagging_loss=0.009498, over 3035156.14 frames. ], batch size: 57, lr: 2.50e-03, grad_scale: 16.0
2023-11-23 00:38:45,792 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.61 vs. limit=22.5
2023-11-23 00:38:49,916 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324600
2023-11-23 00:38:54,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2163980.0, ans=0.125
2023-11-23 00:39:14,363 INFO [train_asr.py:1221] (0/4) Epoch 27, batch 12000, loss[loss=0.06878, simple_loss=0.09992, pruned_loss=0.01051, audio_tagging_loss=0.008309, over 14763.00 frames. ], tot_loss[loss=0.07144, simple_loss=0.0945, pruned_loss=0.01473, audio_tagging_loss=0.00946, over 3032835.44 frames. ], batch size: 55, lr: 2.50e-03, grad_scale: 32.0
2023-11-23 00:39:14,367 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 00:39:34,416 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.7250, 3.3791, 3.5382, 2.9359, 3.7256, 3.7980, 3.7799, 3.7191],
       device='cuda:0')
2023-11-23 00:39:41,164 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4928, 3.2235, 3.7283, 3.3104], device='cuda:0')
2023-11-23 00:39:57,347 INFO [train_asr.py:1253] (0/4) Epoch 27, validation: loss=0.05869, simple_loss=0.05138, pruned_loss=0.005099, audio_tagging_loss=0.0279, over 4681554.00 frames. 
2023-11-23 00:39:57,348 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 00:40:27,231 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-27.pt
2023-11-23 00:41:02,703 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 0, loss[loss=0.08737, simple_loss=0.1117, pruned_loss=0.01276, audio_tagging_loss=0.01878, over 14374.00 frames. ], tot_loss[loss=0.08737, simple_loss=0.1117, pruned_loss=0.01276, audio_tagging_loss=0.01878, over 14374.00 frames. ], batch size: 53, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:41:02,707 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 00:41:39,763 INFO [train_asr.py:1253] (0/4) Epoch 28, validation: loss=0.0583, simple_loss=0.05139, pruned_loss=0.005161, audio_tagging_loss=0.02744, over 4681554.00 frames. 
2023-11-23 00:41:39,764 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 00:41:47,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324650
2023-11-23 00:42:03,339 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.90 vs. limit=15.0
2023-11-23 00:42:04,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2164413.3333333335, ans=0.125
2023-11-23 00:42:09,901 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.350e+01 8.610e+01 9.593e+01 1.048e+02 1.343e+02, threshold=1.919e+02, percent-clipped=0.0
2023-11-23 00:42:32,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.12 vs. limit=15.0
2023-11-23 00:42:35,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2164546.6666666665, ans=0.1
2023-11-23 00:42:43,124 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 50, loss[loss=0.08646, simple_loss=0.1057, pruned_loss=0.0184, audio_tagging_loss=0.01522, over 14954.00 frames. ], tot_loss[loss=0.0778, simple_loss=0.09243, pruned_loss=0.01394, audio_tagging_loss=0.01765, over 688091.03 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:42:43,710 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.91 vs. limit=22.5
2023-11-23 00:42:50,593 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324700
2023-11-23 00:42:56,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2164680.0, ans=0.2
2023-11-23 00:42:58,023 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.53 vs. limit=15.0
2023-11-23 00:43:29,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2164813.3333333335, ans=0.125
2023-11-23 00:43:30,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2164813.3333333335, ans=0.2
2023-11-23 00:43:40,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2164880.0, ans=0.125
2023-11-23 00:43:46,722 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 100, loss[loss=0.09307, simple_loss=0.1262, pruned_loss=0.02006, audio_tagging_loss=0.009922, over 16110.00 frames. ], tot_loss[loss=0.07762, simple_loss=0.09263, pruned_loss=0.0144, audio_tagging_loss=0.01691, over 1210048.97 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:43:54,151 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324750
2023-11-23 00:44:01,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2165013.3333333335, ans=0.0
2023-11-23 00:44:17,226 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.407e+01 9.043e+01 9.747e+01 1.056e+02 1.211e+02, threshold=1.949e+02, percent-clipped=0.0
2023-11-23 00:44:19,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2165080.0, ans=0.05
2023-11-23 00:44:32,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2165146.6666666665, ans=0.125
2023-11-23 00:44:39,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2165213.3333333335, ans=0.0
2023-11-23 00:44:46,925 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.29 vs. limit=15.0
2023-11-23 00:44:49,971 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 150, loss[loss=0.07689, simple_loss=0.09869, pruned_loss=0.01588, audio_tagging_loss=0.01167, over 14558.00 frames. ], tot_loss[loss=0.07652, simple_loss=0.09351, pruned_loss=0.01454, audio_tagging_loss=0.01522, over 1617747.80 frames. ], batch size: 53, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:44:53,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2165280.0, ans=0.0
2023-11-23 00:44:58,158 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324800
2023-11-23 00:45:17,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2165413.3333333335, ans=0.0
2023-11-23 00:45:55,028 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 200, loss[loss=0.0834, simple_loss=0.1086, pruned_loss=0.01872, audio_tagging_loss=0.01037, over 15683.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09388, pruned_loss=0.01478, audio_tagging_loss=0.01352, over 1929500.58 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:46:02,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324850
2023-11-23 00:46:16,843 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.34 vs. limit=15.0
2023-11-23 00:46:25,072 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.466e+01 9.149e+01 9.933e+01 1.935e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 00:46:50,676 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.77 vs. limit=6.0
2023-11-23 00:46:52,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.25 vs. limit=15.0
2023-11-23 00:46:58,571 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 250, loss[loss=0.06758, simple_loss=0.08917, pruned_loss=0.01232, audio_tagging_loss=0.01068, over 14710.00 frames. ], tot_loss[loss=0.07345, simple_loss=0.09345, pruned_loss=0.01439, audio_tagging_loss=0.01234, over 2178154.78 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:47:06,494 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324900
2023-11-23 00:47:20,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2166013.3333333335, ans=6.0
2023-11-23 00:47:29,350 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0
2023-11-23 00:47:34,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2166080.0, ans=0.125
2023-11-23 00:47:34,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2166080.0, ans=0.1
2023-11-23 00:47:43,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2166146.6666666665, ans=0.125
2023-11-23 00:47:47,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2166146.6666666665, ans=0.125
2023-11-23 00:47:59,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2166213.3333333335, ans=0.0
2023-11-23 00:48:03,094 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 300, loss[loss=0.05331, simple_loss=0.07512, pruned_loss=0.006108, audio_tagging_loss=0.009641, over 15421.00 frames. ], tot_loss[loss=0.0732, simple_loss=0.09477, pruned_loss=0.01455, audio_tagging_loss=0.01126, over 2378449.23 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:48:11,810 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 324950
2023-11-23 00:48:34,186 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.439e+01 8.323e+01 9.001e+01 1.002e+02 1.826e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 00:48:37,463 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.53 vs. limit=22.5
2023-11-23 00:48:39,817 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.86 vs. limit=22.5
2023-11-23 00:49:03,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2166546.6666666665, ans=0.125
2023-11-23 00:49:07,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.44 vs. limit=15.0
2023-11-23 00:49:09,306 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 350, loss[loss=0.08575, simple_loss=0.1114, pruned_loss=0.02092, audio_tagging_loss=0.009104, over 14697.00 frames. ], tot_loss[loss=0.07223, simple_loss=0.09425, pruned_loss=0.01448, audio_tagging_loss=0.01063, over 2526450.87 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 00:49:13,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2166613.3333333335, ans=0.0
2023-11-23 00:49:16,930 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325000
2023-11-23 00:49:21,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2166680.0, ans=0.0
2023-11-23 00:49:26,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2166680.0, ans=0.125
2023-11-23 00:49:56,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.41 vs. limit=15.0
2023-11-23 00:50:05,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2166880.0, ans=0.125
2023-11-23 00:50:12,991 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 400, loss[loss=0.08971, simple_loss=0.1124, pruned_loss=0.02218, audio_tagging_loss=0.01133, over 15682.00 frames. ], tot_loss[loss=0.07209, simple_loss=0.09424, pruned_loss=0.01468, audio_tagging_loss=0.01029, over 2652188.88 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:50:20,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325050
2023-11-23 00:50:27,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2167013.3333333335, ans=0.125
2023-11-23 00:50:30,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2167013.3333333335, ans=0.2
2023-11-23 00:50:41,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2167080.0, ans=0.0
2023-11-23 00:50:42,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2167080.0, ans=0.0
2023-11-23 00:50:45,771 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.175e+01 8.785e+01 9.638e+01 1.202e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-23 00:50:47,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2167080.0, ans=0.2
2023-11-23 00:50:53,889 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.57 vs. limit=10.0
2023-11-23 00:51:04,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2167213.3333333335, ans=0.125
2023-11-23 00:51:13,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.91 vs. limit=22.5
2023-11-23 00:51:17,328 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 450, loss[loss=0.07127, simple_loss=0.09073, pruned_loss=0.01641, audio_tagging_loss=0.009489, over 14214.00 frames. ], tot_loss[loss=0.0718, simple_loss=0.09415, pruned_loss=0.01475, audio_tagging_loss=0.009975, over 2737674.50 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:51:25,983 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325100
2023-11-23 00:51:29,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2167280.0, ans=0.2
2023-11-23 00:51:32,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.18 vs. limit=15.0
2023-11-23 00:51:42,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2167346.6666666665, ans=0.125
2023-11-23 00:52:07,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2167480.0, ans=0.125
2023-11-23 00:52:13,843 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 00:52:23,260 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 500, loss[loss=0.09198, simple_loss=0.1196, pruned_loss=0.02466, audio_tagging_loss=0.0075, over 14560.00 frames. ], tot_loss[loss=0.07176, simple_loss=0.09414, pruned_loss=0.01489, audio_tagging_loss=0.009805, over 2810137.85 frames. ], batch size: 53, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:52:30,664 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325150
2023-11-23 00:52:35,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2167680.0, ans=0.1
2023-11-23 00:52:42,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2167680.0, ans=0.125
2023-11-23 00:52:54,605 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.145e+01 8.686e+01 9.449e+01 1.268e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-23 00:53:27,170 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 550, loss[loss=0.08229, simple_loss=0.1086, pruned_loss=0.01796, audio_tagging_loss=0.01003, over 15460.00 frames. ], tot_loss[loss=0.07137, simple_loss=0.09386, pruned_loss=0.01472, audio_tagging_loss=0.009719, over 2863729.96 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:53:34,585 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325200
2023-11-23 00:53:48,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2168013.3333333335, ans=0.125
2023-11-23 00:54:22,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=2168213.3333333335, ans=15.0
2023-11-23 00:54:32,709 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 600, loss[loss=0.09588, simple_loss=0.1263, pruned_loss=0.02437, audio_tagging_loss=0.008342, over 15300.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09286, pruned_loss=0.01443, audio_tagging_loss=0.009675, over 2901878.05 frames. ], batch size: 60, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:54:40,363 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325250
2023-11-23 00:54:42,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2168280.0, ans=0.125
2023-11-23 00:54:55,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2168346.6666666665, ans=0.0
2023-11-23 00:55:01,162 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.03 vs. limit=6.0
2023-11-23 00:55:03,810 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.09 vs. limit=22.5
2023-11-23 00:55:04,340 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.706e+01 8.119e+01 8.698e+01 9.405e+01 1.451e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-23 00:55:04,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2168413.3333333335, ans=0.1
2023-11-23 00:55:11,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2168480.0, ans=0.125
2023-11-23 00:55:12,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2168480.0, ans=0.0
2023-11-23 00:55:21,601 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.22 vs. limit=15.0
2023-11-23 00:55:36,590 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 650, loss[loss=0.06889, simple_loss=0.1059, pruned_loss=0.009294, audio_tagging_loss=0.00664, over 16389.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09304, pruned_loss=0.01458, audio_tagging_loss=0.009636, over 2934703.11 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:55:44,118 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325300
2023-11-23 00:55:44,592 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.66 vs. limit=22.5
2023-11-23 00:55:55,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2168680.0, ans=0.04949747468305833
2023-11-23 00:56:09,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2168746.6666666665, ans=0.125
2023-11-23 00:56:11,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2168746.6666666665, ans=0.125
2023-11-23 00:56:38,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2168880.0, ans=0.1
2023-11-23 00:56:40,466 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 700, loss[loss=0.08328, simple_loss=0.1146, pruned_loss=0.01532, audio_tagging_loss=0.01066, over 15059.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09346, pruned_loss=0.01456, audio_tagging_loss=0.009596, over 2958090.57 frames. ], batch size: 55, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:56:47,967 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325350
2023-11-23 00:57:13,023 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.373e+01 9.027e+01 9.909e+01 1.159e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 00:57:16,891 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.97 vs. limit=15.0
2023-11-23 00:57:21,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2169146.6666666665, ans=0.125
2023-11-23 00:57:26,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2169146.6666666665, ans=0.125
2023-11-23 00:57:44,761 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 750, loss[loss=0.08163, simple_loss=0.1198, pruned_loss=0.014, audio_tagging_loss=0.00772, over 14782.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09396, pruned_loss=0.01459, audio_tagging_loss=0.009545, over 2977856.06 frames. ], batch size: 53, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:57:50,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2169280.0, ans=0.5
2023-11-23 00:57:52,878 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325400
2023-11-23 00:57:56,275 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.29 vs. limit=15.0
2023-11-23 00:58:01,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2169346.6666666665, ans=0.125
2023-11-23 00:58:14,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2169413.3333333335, ans=0.125
2023-11-23 00:58:28,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2169480.0, ans=0.2
2023-11-23 00:58:49,694 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 800, loss[loss=0.06218, simple_loss=0.08486, pruned_loss=0.0091, audio_tagging_loss=0.01066, over 15823.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09403, pruned_loss=0.01468, audio_tagging_loss=0.009515, over 2992149.02 frames. ], batch size: 60, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 00:58:57,649 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325450
2023-11-23 00:59:03,309 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.37 vs. limit=6.0
2023-11-23 00:59:04,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2169680.0, ans=0.09899494936611666
2023-11-23 00:59:05,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2169680.0, ans=0.2
2023-11-23 00:59:11,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2169680.0, ans=0.1
2023-11-23 00:59:20,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2169746.6666666665, ans=0.125
2023-11-23 00:59:21,175 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.947e+01 8.109e+01 8.929e+01 9.816e+01 1.280e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 00:59:33,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2169813.3333333335, ans=0.125
2023-11-23 00:59:48,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2169880.0, ans=0.04949747468305833
2023-11-23 00:59:54,380 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 850, loss[loss=0.08378, simple_loss=0.1148, pruned_loss=0.01843, audio_tagging_loss=0.007951, over 15009.00 frames. ], tot_loss[loss=0.07112, simple_loss=0.09383, pruned_loss=0.01459, audio_tagging_loss=0.009623, over 3001681.03 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 00:59:56,275 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.90 vs. limit=22.5
2023-11-23 00:59:57,078 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:00:01,735 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325500
2023-11-23 01:00:09,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2170013.3333333335, ans=0.125
2023-11-23 01:00:11,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2170013.3333333335, ans=0.125
2023-11-23 01:00:27,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2170080.0, ans=0.1
2023-11-23 01:00:40,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2170146.6666666665, ans=0.0
2023-11-23 01:00:49,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2170213.3333333335, ans=0.0
2023-11-23 01:00:57,846 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 900, loss[loss=0.0732, simple_loss=0.09027, pruned_loss=0.01946, audio_tagging_loss=0.008605, over 15231.00 frames. ], tot_loss[loss=0.0709, simple_loss=0.09366, pruned_loss=0.01442, audio_tagging_loss=0.009644, over 3010520.22 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:01:06,198 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325550
2023-11-23 01:01:07,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2170280.0, ans=0.125
2023-11-23 01:01:22,650 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.60 vs. limit=12.0
2023-11-23 01:01:24,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2170413.3333333335, ans=0.04949747468305833
2023-11-23 01:01:30,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2170413.3333333335, ans=0.2
2023-11-23 01:01:32,256 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.700e+01 8.072e+01 8.678e+01 9.441e+01 1.146e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-23 01:01:47,907 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.37 vs. limit=15.0
2023-11-23 01:01:53,343 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.73 vs. limit=15.0
2023-11-23 01:01:55,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2170546.6666666665, ans=0.1
2023-11-23 01:02:02,706 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 950, loss[loss=0.07299, simple_loss=0.1011, pruned_loss=0.01562, audio_tagging_loss=0.006846, over 15216.00 frames. ], tot_loss[loss=0.07131, simple_loss=0.0946, pruned_loss=0.01456, audio_tagging_loss=0.009448, over 3022647.17 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:02:04,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2170613.3333333335, ans=0.125
2023-11-23 01:02:06,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2170613.3333333335, ans=0.125
2023-11-23 01:02:11,577 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325600
2023-11-23 01:02:11,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2170613.3333333335, ans=0.125
2023-11-23 01:02:22,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2170680.0, ans=0.125
2023-11-23 01:02:41,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2170813.3333333335, ans=0.125
2023-11-23 01:02:49,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2170813.3333333335, ans=0.125
2023-11-23 01:02:53,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2170813.3333333335, ans=0.0
2023-11-23 01:02:53,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2170813.3333333335, ans=0.1
2023-11-23 01:02:53,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2170813.3333333335, ans=0.2
2023-11-23 01:03:08,283 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1000, loss[loss=0.06738, simple_loss=0.08361, pruned_loss=0.01357, audio_tagging_loss=0.012, over 15252.00 frames. ], tot_loss[loss=0.07139, simple_loss=0.09481, pruned_loss=0.01469, audio_tagging_loss=0.009293, over 3031740.56 frames. ], batch size: 60, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:03:16,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325650
2023-11-23 01:03:24,049 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.01 vs. limit=6.0
2023-11-23 01:03:28,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2171013.3333333335, ans=0.125
2023-11-23 01:03:34,968 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:03:35,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2171080.0, ans=0.0
2023-11-23 01:03:41,020 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.239e+01 8.945e+01 9.781e+01 1.255e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 01:03:47,393 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.41 vs. limit=22.5
2023-11-23 01:04:03,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2171213.3333333335, ans=0.0
2023-11-23 01:04:11,929 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1050, loss[loss=0.08428, simple_loss=0.1205, pruned_loss=0.01742, audio_tagging_loss=0.006585, over 15589.00 frames. ], tot_loss[loss=0.07136, simple_loss=0.09481, pruned_loss=0.01473, audio_tagging_loss=0.009224, over 3042610.64 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:04:13,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2171280.0, ans=0.125
2023-11-23 01:04:19,837 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325700
2023-11-23 01:04:26,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2171346.6666666665, ans=0.0
2023-11-23 01:04:40,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2171413.3333333335, ans=0.125
2023-11-23 01:05:13,692 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.45 vs. limit=15.0
2023-11-23 01:05:15,414 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1100, loss[loss=0.08135, simple_loss=0.1056, pruned_loss=0.01852, audio_tagging_loss=0.01002, over 15884.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09458, pruned_loss=0.01456, audio_tagging_loss=0.009091, over 3044161.73 frames. ], batch size: 60, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:05:19,163 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:05:23,640 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325750
2023-11-23 01:05:29,254 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.33 vs. limit=12.0
2023-11-23 01:05:33,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2171680.0, ans=0.2
2023-11-23 01:05:48,722 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.874e+01 8.331e+01 9.113e+01 9.844e+01 1.667e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-23 01:05:53,034 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.94 vs. limit=15.0
2023-11-23 01:06:05,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.50 vs. limit=15.0
2023-11-23 01:06:07,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2171880.0, ans=0.125
2023-11-23 01:06:11,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2171880.0, ans=0.5
2023-11-23 01:06:20,029 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1150, loss[loss=0.06134, simple_loss=0.07765, pruned_loss=0.01058, audio_tagging_loss=0.01194, over 15235.00 frames. ], tot_loss[loss=0.07171, simple_loss=0.09556, pruned_loss=0.01487, audio_tagging_loss=0.009061, over 3047520.21 frames. ], batch size: 61, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:06:27,769 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325800
2023-11-23 01:06:47,259 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.15 vs. limit=15.0
2023-11-23 01:07:05,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.07 vs. limit=22.5
2023-11-23 01:07:18,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2172213.3333333335, ans=0.0
2023-11-23 01:07:19,666 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.45 vs. limit=15.0
2023-11-23 01:07:24,604 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1200, loss[loss=0.08009, simple_loss=0.1089, pruned_loss=0.01492, audio_tagging_loss=0.01071, over 15203.00 frames. ], tot_loss[loss=0.07107, simple_loss=0.09465, pruned_loss=0.01471, audio_tagging_loss=0.00903, over 3049920.58 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 01:07:32,299 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325850
2023-11-23 01:07:57,978 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.204e+01 8.430e+01 8.952e+01 9.650e+01 2.402e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 01:07:59,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2172413.3333333335, ans=0.125
2023-11-23 01:08:29,033 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1250, loss[loss=0.05794, simple_loss=0.08019, pruned_loss=0.01056, audio_tagging_loss=0.007288, over 14546.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09318, pruned_loss=0.01455, audio_tagging_loss=0.009108, over 3042668.94 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 01:08:33,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2172613.3333333335, ans=0.125
2023-11-23 01:08:37,159 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325900
2023-11-23 01:08:50,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2172680.0, ans=0.1
2023-11-23 01:09:02,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2172746.6666666665, ans=0.0
2023-11-23 01:09:20,145 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.46 vs. limit=15.0
2023-11-23 01:09:23,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2172880.0, ans=0.125
2023-11-23 01:09:34,068 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1300, loss[loss=0.066, simple_loss=0.08554, pruned_loss=0.01425, audio_tagging_loss=0.008982, over 15210.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09294, pruned_loss=0.0146, audio_tagging_loss=0.009091, over 3034891.54 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:09:35,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2172946.6666666665, ans=0.125
2023-11-23 01:09:41,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 325950
2023-11-23 01:09:50,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2173013.3333333335, ans=0.125
2023-11-23 01:10:08,886 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.443e+01 8.136e+01 8.697e+01 9.484e+01 1.452e+02, threshold=1.739e+02, percent-clipped=1.0
2023-11-23 01:10:09,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2173080.0, ans=0.2
2023-11-23 01:10:16,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2173146.6666666665, ans=0.1
2023-11-23 01:10:27,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2173213.3333333335, ans=0.125
2023-11-23 01:10:32,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2173213.3333333335, ans=0.125
2023-11-23 01:10:33,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2173213.3333333335, ans=0.1
2023-11-23 01:10:38,104 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1350, loss[loss=0.05731, simple_loss=0.07858, pruned_loss=0.01119, audio_tagging_loss=0.006836, over 15442.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09374, pruned_loss=0.01461, audio_tagging_loss=0.009011, over 3036009.95 frames. ], batch size: 59, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:10:46,033 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326000
2023-11-23 01:11:06,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2173413.3333333335, ans=0.125
2023-11-23 01:11:11,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2173413.3333333335, ans=0.0
2023-11-23 01:11:11,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.39 vs. limit=12.0
2023-11-23 01:11:17,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2173480.0, ans=0.07
2023-11-23 01:11:18,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2173480.0, ans=0.1
2023-11-23 01:11:25,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.44 vs. limit=15.0
2023-11-23 01:11:25,648 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:11:37,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2173546.6666666665, ans=0.2
2023-11-23 01:11:42,201 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.93 vs. limit=15.0
2023-11-23 01:11:42,779 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1400, loss[loss=0.06622, simple_loss=0.08807, pruned_loss=0.01368, audio_tagging_loss=0.00851, over 14820.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09295, pruned_loss=0.01443, audio_tagging_loss=0.009086, over 3043587.42 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:11:44,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2173613.3333333335, ans=0.1
2023-11-23 01:11:50,193 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326050
2023-11-23 01:12:08,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2173746.6666666665, ans=0.0
2023-11-23 01:12:17,025 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 8.070e+01 8.559e+01 9.321e+01 1.215e+02, threshold=1.712e+02, percent-clipped=0.0
2023-11-23 01:12:19,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2173813.3333333335, ans=0.125
2023-11-23 01:12:25,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2173813.3333333335, ans=0.0
2023-11-23 01:12:34,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2173880.0, ans=0.1
2023-11-23 01:12:47,049 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1450, loss[loss=0.07566, simple_loss=0.1029, pruned_loss=0.01844, audio_tagging_loss=0.005772, over 15584.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09311, pruned_loss=0.01459, audio_tagging_loss=0.009102, over 3043531.74 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:12:53,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2173946.6666666665, ans=0.1
2023-11-23 01:12:54,561 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326100
2023-11-23 01:13:09,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2174013.3333333335, ans=0.125
2023-11-23 01:13:27,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2174146.6666666665, ans=0.0
2023-11-23 01:13:32,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2174146.6666666665, ans=0.05
2023-11-23 01:13:50,359 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1500, loss[loss=0.08575, simple_loss=0.1094, pruned_loss=0.01913, audio_tagging_loss=0.01191, over 15279.00 frames. ], tot_loss[loss=0.07088, simple_loss=0.09381, pruned_loss=0.01476, audio_tagging_loss=0.009212, over 3042925.22 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:13:57,994 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326150
2023-11-23 01:14:22,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2174413.3333333335, ans=0.125
2023-11-23 01:14:24,428 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.79 vs. limit=15.0
2023-11-23 01:14:26,199 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.046e+01 8.933e+01 9.681e+01 1.186e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 01:14:27,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2174413.3333333335, ans=0.125
2023-11-23 01:14:43,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2174546.6666666665, ans=0.125
2023-11-23 01:14:46,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2174546.6666666665, ans=0.125
2023-11-23 01:14:55,307 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1550, loss[loss=0.07274, simple_loss=0.09958, pruned_loss=0.01339, audio_tagging_loss=0.009557, over 15129.00 frames. ], tot_loss[loss=0.0712, simple_loss=0.09424, pruned_loss=0.01476, audio_tagging_loss=0.009319, over 3043818.10 frames. ], batch size: 56, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:15:00,992 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.17 vs. limit=10.0
2023-11-23 01:15:03,650 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.54 vs. limit=15.0
2023-11-23 01:15:04,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326200
2023-11-23 01:15:17,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2174680.0, ans=0.125
2023-11-23 01:15:21,533 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.42 vs. limit=15.0
2023-11-23 01:15:35,039 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:15:35,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.88 vs. limit=6.0
2023-11-23 01:15:51,439 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.47 vs. limit=6.0
2023-11-23 01:15:55,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2174880.0, ans=0.125
2023-11-23 01:16:03,404 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1600, loss[loss=0.07363, simple_loss=0.09603, pruned_loss=0.01611, audio_tagging_loss=0.009502, over 14571.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09347, pruned_loss=0.01465, audio_tagging_loss=0.009319, over 3038726.54 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 01:16:04,084 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.35 vs. limit=15.0
2023-11-23 01:16:11,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326250
2023-11-23 01:16:26,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2175013.3333333335, ans=0.125
2023-11-23 01:16:28,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2175080.0, ans=0.1
2023-11-23 01:16:31,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.47 vs. limit=15.0
2023-11-23 01:16:37,598 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.240e+01 8.854e+01 9.452e+01 1.276e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 01:17:07,135 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1650, loss[loss=0.07182, simple_loss=0.08823, pruned_loss=0.01655, audio_tagging_loss=0.01115, over 15296.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09345, pruned_loss=0.01464, audio_tagging_loss=0.00933, over 3044451.42 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 32.0
2023-11-23 01:17:14,767 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326300
2023-11-23 01:17:15,603 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.85 vs. limit=15.0
2023-11-23 01:17:49,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2175480.0, ans=0.2
2023-11-23 01:17:59,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2175546.6666666665, ans=0.1
2023-11-23 01:18:01,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2175546.6666666665, ans=0.125
2023-11-23 01:18:11,461 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1700, loss[loss=0.08085, simple_loss=0.1115, pruned_loss=0.01837, audio_tagging_loss=0.006721, over 14877.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09295, pruned_loss=0.01442, audio_tagging_loss=0.009362, over 3047425.29 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:18:11,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2175613.3333333335, ans=0.125
2023-11-23 01:18:19,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326350
2023-11-23 01:18:47,668 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.185e+01 8.813e+01 9.746e+01 1.184e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 01:18:48,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2175746.6666666665, ans=0.125
2023-11-23 01:19:08,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2175880.0, ans=0.125
2023-11-23 01:19:09,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2175880.0, ans=0.0
2023-11-23 01:19:15,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2175946.6666666665, ans=0.1
2023-11-23 01:19:16,140 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1750, loss[loss=0.07094, simple_loss=0.09741, pruned_loss=0.01509, audio_tagging_loss=0.007147, over 14371.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09205, pruned_loss=0.01418, audio_tagging_loss=0.009304, over 3042738.19 frames. ], batch size: 54, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:19:23,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326400
2023-11-23 01:19:34,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1.whitening_limit, batch_count=2176013.3333333335, ans=10.0
2023-11-23 01:19:39,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2176080.0, ans=0.0
2023-11-23 01:19:44,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2176080.0, ans=0.0
2023-11-23 01:19:50,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2176080.0, ans=0.125
2023-11-23 01:20:20,570 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1800, loss[loss=0.05901, simple_loss=0.07928, pruned_loss=0.008551, audio_tagging_loss=0.01082, over 15130.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09257, pruned_loss=0.01431, audio_tagging_loss=0.009231, over 3042397.93 frames. ], batch size: 57, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:20:27,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2176280.0, ans=10.0
2023-11-23 01:20:28,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326450
2023-11-23 01:20:33,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2176346.6666666665, ans=0.1
2023-11-23 01:20:57,617 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.825e+01 8.178e+01 8.671e+01 9.411e+01 1.178e+02, threshold=1.734e+02, percent-clipped=0.0
2023-11-23 01:21:07,264 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.50 vs. limit=6.0
2023-11-23 01:21:09,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2176480.0, ans=0.0
2023-11-23 01:21:20,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2176546.6666666665, ans=0.125
2023-11-23 01:21:23,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2176546.6666666665, ans=0.125
2023-11-23 01:21:25,130 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1850, loss[loss=0.06862, simple_loss=0.09112, pruned_loss=0.01414, audio_tagging_loss=0.008915, over 15689.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09348, pruned_loss=0.01443, audio_tagging_loss=0.009201, over 3044602.19 frames. ], batch size: 58, lr: 2.45e-03, grad_scale: 16.0
2023-11-23 01:21:26,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2176613.3333333335, ans=0.125
2023-11-23 01:21:30,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.05 vs. limit=15.0
2023-11-23 01:21:33,516 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326500
2023-11-23 01:22:08,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2176813.3333333335, ans=0.0
2023-11-23 01:22:31,055 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1900, loss[loss=0.07484, simple_loss=0.09952, pruned_loss=0.01678, audio_tagging_loss=0.008293, over 16610.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09321, pruned_loss=0.01443, audio_tagging_loss=0.009146, over 3055297.92 frames. ], batch size: 61, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:22:39,631 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326550
2023-11-23 01:22:54,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2177013.3333333335, ans=0.125
2023-11-23 01:22:57,224 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.26 vs. limit=22.5
2023-11-23 01:23:06,400 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.801e+01 8.128e+01 8.928e+01 9.740e+01 2.315e+02, threshold=1.786e+02, percent-clipped=1.0
2023-11-23 01:23:11,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2177146.6666666665, ans=0.0
2023-11-23 01:23:26,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2177213.3333333335, ans=0.07
2023-11-23 01:23:32,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2177213.3333333335, ans=0.125
2023-11-23 01:23:36,012 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 1950, loss[loss=0.07122, simple_loss=0.08869, pruned_loss=0.01345, audio_tagging_loss=0.01343, over 16284.00 frames. ], tot_loss[loss=0.07031, simple_loss=0.09349, pruned_loss=0.01446, audio_tagging_loss=0.009103, over 3057535.71 frames. ], batch size: 62, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:23:36,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.80 vs. limit=15.0
2023-11-23 01:23:43,560 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326600
2023-11-23 01:23:58,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2177346.6666666665, ans=0.125
2023-11-23 01:24:18,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2177480.0, ans=0.125
2023-11-23 01:24:28,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2177546.6666666665, ans=0.04949747468305833
2023-11-23 01:24:37,877 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.51 vs. limit=22.5
2023-11-23 01:24:40,692 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2000, loss[loss=0.0864, simple_loss=0.111, pruned_loss=0.01942, audio_tagging_loss=0.01147, over 15657.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09266, pruned_loss=0.01431, audio_tagging_loss=0.009199, over 3053851.30 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:24:41,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2177613.3333333335, ans=0.0
2023-11-23 01:24:43,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2177613.3333333335, ans=0.125
2023-11-23 01:24:45,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2177613.3333333335, ans=0.0
2023-11-23 01:24:46,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.07 vs. limit=12.0
2023-11-23 01:24:48,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326650
2023-11-23 01:24:53,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2177680.0, ans=0.1
2023-11-23 01:25:06,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2177746.6666666665, ans=0.125
2023-11-23 01:25:15,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2177746.6666666665, ans=0.0
2023-11-23 01:25:17,649 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.624e+01 8.041e+01 8.683e+01 9.382e+01 1.367e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-23 01:25:28,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2177813.3333333335, ans=0.1
2023-11-23 01:25:40,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2177880.0, ans=0.125
2023-11-23 01:25:41,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2177880.0, ans=0.1
2023-11-23 01:25:43,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2177880.0, ans=0.1
2023-11-23 01:25:45,380 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2050, loss[loss=0.07613, simple_loss=0.1088, pruned_loss=0.01559, audio_tagging_loss=0.006128, over 15333.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09349, pruned_loss=0.01442, audio_tagging_loss=0.009129, over 3050714.75 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:25:46,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2177946.6666666665, ans=0.125
2023-11-23 01:25:53,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326700
2023-11-23 01:26:06,984 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.44 vs. limit=15.0
2023-11-23 01:26:07,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2178013.3333333335, ans=0.0
2023-11-23 01:26:10,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2178080.0, ans=0.0
2023-11-23 01:26:13,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2178080.0, ans=0.2
2023-11-23 01:26:16,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2178080.0, ans=0.2
2023-11-23 01:26:25,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2178146.6666666665, ans=0.125
2023-11-23 01:26:28,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2178146.6666666665, ans=10.0
2023-11-23 01:26:31,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2178146.6666666665, ans=0.125
2023-11-23 01:26:34,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2178146.6666666665, ans=0.05
2023-11-23 01:26:36,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.39 vs. limit=15.0
2023-11-23 01:26:46,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2178213.3333333335, ans=0.125
2023-11-23 01:26:49,584 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2100, loss[loss=0.04814, simple_loss=0.05499, pruned_loss=0.00782, audio_tagging_loss=0.01283, over 14877.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09249, pruned_loss=0.01427, audio_tagging_loss=0.009111, over 3054422.48 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:26:57,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326750
2023-11-23 01:27:03,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.43 vs. limit=12.0
2023-11-23 01:27:25,011 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.865e+01 8.249e+01 8.824e+01 9.661e+01 1.236e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 01:27:53,297 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2150, loss[loss=0.0544, simple_loss=0.07332, pruned_loss=0.009662, audio_tagging_loss=0.008078, over 15070.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09364, pruned_loss=0.01442, audio_tagging_loss=0.009031, over 3044707.28 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:28:00,763 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326800
2023-11-23 01:28:03,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2178613.3333333335, ans=0.0
2023-11-23 01:28:24,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2178746.6666666665, ans=0.2
2023-11-23 01:28:31,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2178813.3333333335, ans=0.125
2023-11-23 01:28:33,815 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:28:37,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2178813.3333333335, ans=0.125
2023-11-23 01:28:58,347 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2200, loss[loss=0.0698, simple_loss=0.09774, pruned_loss=0.01276, audio_tagging_loss=0.008175, over 15360.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09345, pruned_loss=0.01449, audio_tagging_loss=0.009083, over 3041754.30 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:29:07,017 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326850
2023-11-23 01:29:09,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2178946.6666666665, ans=0.125
2023-11-23 01:29:14,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2179013.3333333335, ans=0.2
2023-11-23 01:29:15,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2179013.3333333335, ans=0.0
2023-11-23 01:29:21,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2179013.3333333335, ans=0.1
2023-11-23 01:29:27,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2179080.0, ans=0.125
2023-11-23 01:29:34,175 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.469e+01 8.071e+01 8.694e+01 9.748e+01 1.270e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-23 01:30:03,056 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2250, loss[loss=0.08364, simple_loss=0.1128, pruned_loss=0.01738, audio_tagging_loss=0.009877, over 15653.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09397, pruned_loss=0.01446, audio_tagging_loss=0.009107, over 3047607.25 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:30:08,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2179280.0, ans=0.0
2023-11-23 01:30:10,491 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326900
2023-11-23 01:30:25,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2179346.6666666665, ans=0.0
2023-11-23 01:30:35,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2179413.3333333335, ans=0.2
2023-11-23 01:30:48,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2179480.0, ans=0.0
2023-11-23 01:30:54,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.87 vs. limit=22.5
2023-11-23 01:31:07,423 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2300, loss[loss=0.06708, simple_loss=0.09186, pruned_loss=0.01159, audio_tagging_loss=0.009567, over 16706.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09433, pruned_loss=0.01441, audio_tagging_loss=0.009164, over 3050065.29 frames. ], batch size: 64, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:31:14,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 326950
2023-11-23 01:31:19,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2179680.0, ans=0.2
2023-11-23 01:31:44,824 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.554e+01 8.250e+01 8.860e+01 9.394e+01 1.286e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 01:32:05,578 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:32:12,387 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2350, loss[loss=0.09492, simple_loss=0.1372, pruned_loss=0.02097, audio_tagging_loss=0.005358, over 15557.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09347, pruned_loss=0.01426, audio_tagging_loss=0.009214, over 3057071.46 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:32:15,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=2179946.6666666665, ans=0.5
2023-11-23 01:32:19,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327000
2023-11-23 01:32:20,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2179946.6666666665, ans=0.2
2023-11-23 01:32:23,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2179946.6666666665, ans=0.0
2023-11-23 01:32:37,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2180080.0, ans=0.0
2023-11-23 01:33:07,281 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.50 vs. limit=15.0
2023-11-23 01:33:17,545 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2400, loss[loss=0.08613, simple_loss=0.1172, pruned_loss=0.02027, audio_tagging_loss=0.007255, over 15348.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09381, pruned_loss=0.01433, audio_tagging_loss=0.009349, over 3056310.96 frames. ], batch size: 53, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:33:18,357 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.82 vs. limit=15.0
2023-11-23 01:33:19,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2180280.0, ans=0.1
2023-11-23 01:33:25,028 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327050
2023-11-23 01:33:35,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2180346.6666666665, ans=0.125
2023-11-23 01:33:54,375 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.524e+01 9.128e+01 9.621e+01 1.327e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-23 01:33:55,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2180480.0, ans=0.1
2023-11-23 01:33:57,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2180480.0, ans=0.125
2023-11-23 01:34:21,273 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2450, loss[loss=0.04919, simple_loss=0.04833, pruned_loss=0.01125, audio_tagging_loss=0.01378, over 13336.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09337, pruned_loss=0.01424, audio_tagging_loss=0.00936, over 3053582.17 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:34:28,765 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327100
2023-11-23 01:34:35,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2180680.0, ans=0.125
2023-11-23 01:34:36,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2180680.0, ans=0.125
2023-11-23 01:34:40,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2180680.0, ans=0.1
2023-11-23 01:35:03,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2180813.3333333335, ans=0.125
2023-11-23 01:35:04,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2180813.3333333335, ans=0.125
2023-11-23 01:35:05,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2180813.3333333335, ans=0.0
2023-11-23 01:35:08,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.15 vs. limit=10.0
2023-11-23 01:35:11,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2180880.0, ans=0.125
2023-11-23 01:35:11,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2180880.0, ans=0.07
2023-11-23 01:35:16,515 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:35:17,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2180880.0, ans=0.1
2023-11-23 01:35:20,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2180880.0, ans=0.1
2023-11-23 01:35:25,349 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2500, loss[loss=0.06766, simple_loss=0.09107, pruned_loss=0.01351, audio_tagging_loss=0.008611, over 14422.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09292, pruned_loss=0.01411, audio_tagging_loss=0.009434, over 3054816.67 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:35:27,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2180946.6666666665, ans=0.0
2023-11-23 01:35:29,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2180946.6666666665, ans=0.09899494936611666
2023-11-23 01:35:32,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2180946.6666666665, ans=10.0
2023-11-23 01:35:33,334 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327150
2023-11-23 01:35:34,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2180946.6666666665, ans=0.125
2023-11-23 01:35:34,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2180946.6666666665, ans=0.2
2023-11-23 01:35:35,065 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.04 vs. limit=15.0
2023-11-23 01:36:02,403 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.294e+01 8.910e+01 9.572e+01 1.200e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 01:36:03,080 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.90 vs. limit=15.0
2023-11-23 01:36:23,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2181213.3333333335, ans=0.125
2023-11-23 01:36:25,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2181213.3333333335, ans=0.1
2023-11-23 01:36:30,874 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2550, loss[loss=0.04593, simple_loss=0.06651, pruned_loss=0.006414, audio_tagging_loss=0.006258, over 15005.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09288, pruned_loss=0.01414, audio_tagging_loss=0.009309, over 3049140.29 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:36:33,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2181280.0, ans=0.07
2023-11-23 01:36:38,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327200
2023-11-23 01:36:38,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2181280.0, ans=0.2
2023-11-23 01:36:41,672 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.16 vs. limit=22.5
2023-11-23 01:36:54,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2181413.3333333335, ans=0.125
2023-11-23 01:37:03,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2181413.3333333335, ans=0.1
2023-11-23 01:37:23,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2181546.6666666665, ans=0.95
2023-11-23 01:37:25,534 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:37:27,188 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.99 vs. limit=22.5
2023-11-23 01:37:35,101 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2600, loss[loss=0.07415, simple_loss=0.1002, pruned_loss=0.01584, audio_tagging_loss=0.008224, over 14904.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09291, pruned_loss=0.01402, audio_tagging_loss=0.009206, over 3043664.67 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:37:39,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2181613.3333333335, ans=0.125
2023-11-23 01:37:42,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327250
2023-11-23 01:38:02,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2181746.6666666665, ans=0.2
2023-11-23 01:38:13,517 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.815e+01 8.227e+01 8.835e+01 9.553e+01 1.278e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 01:38:39,224 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2650, loss[loss=0.07049, simple_loss=0.09668, pruned_loss=0.01358, audio_tagging_loss=0.008571, over 14843.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09234, pruned_loss=0.01403, audio_tagging_loss=0.009126, over 3045108.66 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:38:47,257 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327300
2023-11-23 01:39:06,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2182080.0, ans=0.5
2023-11-23 01:39:12,610 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.48 vs. limit=22.5
2023-11-23 01:39:19,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2182146.6666666665, ans=0.0
2023-11-23 01:39:32,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2182213.3333333335, ans=0.0
2023-11-23 01:39:37,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2182213.3333333335, ans=0.125
2023-11-23 01:39:44,315 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2700, loss[loss=0.08069, simple_loss=0.1019, pruned_loss=0.02115, audio_tagging_loss=0.008568, over 14495.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.0923, pruned_loss=0.01424, audio_tagging_loss=0.0091, over 3050223.25 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:39:52,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327350
2023-11-23 01:40:03,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2182346.6666666665, ans=0.2
2023-11-23 01:40:21,076 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.147e+01 8.099e+01 8.608e+01 9.208e+01 1.380e+02, threshold=1.722e+02, percent-clipped=0.0
2023-11-23 01:40:29,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2182480.0, ans=0.125
2023-11-23 01:40:40,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2182546.6666666665, ans=0.0
2023-11-23 01:40:44,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2182546.6666666665, ans=0.125
2023-11-23 01:40:48,630 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2750, loss[loss=0.06242, simple_loss=0.08865, pruned_loss=0.008894, audio_tagging_loss=0.009201, over 14274.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.0928, pruned_loss=0.01433, audio_tagging_loss=0.009054, over 3048437.82 frames. ], batch size: 53, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:40:56,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327400
2023-11-23 01:41:15,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2182746.6666666665, ans=0.09899494936611666
2023-11-23 01:41:21,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2182746.6666666665, ans=0.125
2023-11-23 01:41:44,323 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:41:52,833 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2800, loss[loss=0.06464, simple_loss=0.07908, pruned_loss=0.01217, audio_tagging_loss=0.01293, over 15266.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09227, pruned_loss=0.01413, audio_tagging_loss=0.009049, over 3041329.44 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:41:54,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2182946.6666666665, ans=0.2
2023-11-23 01:42:00,769 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327450
2023-11-23 01:42:13,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2183013.3333333335, ans=0.5
2023-11-23 01:42:16,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2183013.3333333335, ans=0.2
2023-11-23 01:42:21,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2183080.0, ans=0.125
2023-11-23 01:42:31,831 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.504e+01 8.188e+01 8.710e+01 9.337e+01 1.091e+02, threshold=1.742e+02, percent-clipped=0.0
2023-11-23 01:42:48,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2183213.3333333335, ans=0.1
2023-11-23 01:42:58,034 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2850, loss[loss=0.05322, simple_loss=0.06819, pruned_loss=0.008433, audio_tagging_loss=0.01069, over 15469.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09193, pruned_loss=0.01412, audio_tagging_loss=0.009115, over 3034570.21 frames. ], batch size: 61, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:43:06,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327500
2023-11-23 01:43:06,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.44 vs. limit=15.0
2023-11-23 01:43:07,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2183280.0, ans=0.0
2023-11-23 01:43:29,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2183413.3333333335, ans=0.0
2023-11-23 01:43:30,949 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:43:48,947 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.78 vs. limit=15.0
2023-11-23 01:43:56,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2183546.6666666665, ans=0.125
2023-11-23 01:44:01,868 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2900, loss[loss=0.07984, simple_loss=0.0969, pruned_loss=0.01867, audio_tagging_loss=0.01272, over 15161.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09236, pruned_loss=0.01427, audio_tagging_loss=0.009177, over 3034598.63 frames. ], batch size: 58, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:44:04,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2183613.3333333335, ans=0.0
2023-11-23 01:44:09,349 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327550
2023-11-23 01:44:37,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2183746.6666666665, ans=0.125
2023-11-23 01:44:37,697 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.15 vs. limit=15.0
2023-11-23 01:44:41,226 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 8.459e+01 9.104e+01 9.833e+01 1.241e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-23 01:44:50,466 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.81 vs. limit=15.0
2023-11-23 01:44:57,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2183880.0, ans=0.0
2023-11-23 01:45:03,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2183880.0, ans=0.1
2023-11-23 01:45:05,875 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 2950, loss[loss=0.082, simple_loss=0.1159, pruned_loss=0.01636, audio_tagging_loss=0.007685, over 15745.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09337, pruned_loss=0.01439, audio_tagging_loss=0.009096, over 3038865.43 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:45:13,325 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327600
2023-11-23 01:45:13,876 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0
2023-11-23 01:45:51,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2184146.6666666665, ans=0.1
2023-11-23 01:46:09,982 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3000, loss[loss=0.06605, simple_loss=0.09515, pruned_loss=0.01006, audio_tagging_loss=0.008414, over 15606.00 frames. ], tot_loss[loss=0.07016, simple_loss=0.09308, pruned_loss=0.01442, audio_tagging_loss=0.009203, over 3042610.12 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:46:09,984 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 01:46:30,823 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([6.5004, 6.1487, 6.4398, 5.8544], device='cuda:0')
2023-11-23 01:46:53,258 INFO [train_asr.py:1253] (0/4) Epoch 28, validation: loss=0.05807, simple_loss=0.05122, pruned_loss=0.005026, audio_tagging_loss=0.02744, over 4681554.00 frames. 
2023-11-23 01:46:53,259 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 01:46:58,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2184280.0, ans=0.05
2023-11-23 01:47:00,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327650
2023-11-23 01:47:32,173 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.903e+01 8.143e+01 8.821e+01 9.621e+01 1.257e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 01:47:46,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2184546.6666666665, ans=0.2
2023-11-23 01:47:54,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2184546.6666666665, ans=0.0
2023-11-23 01:47:56,822 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3050, loss[loss=0.05738, simple_loss=0.07772, pruned_loss=0.009124, audio_tagging_loss=0.009399, over 14652.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09327, pruned_loss=0.01466, audio_tagging_loss=0.009326, over 3037331.42 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:47:58,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2184613.3333333335, ans=0.0
2023-11-23 01:48:04,749 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327700
2023-11-23 01:48:07,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2184613.3333333335, ans=0.0
2023-11-23 01:48:17,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.71 vs. limit=12.0
2023-11-23 01:48:28,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2184746.6666666665, ans=0.1
2023-11-23 01:48:35,717 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:48:40,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2184813.3333333335, ans=10.0
2023-11-23 01:48:44,816 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.32 vs. limit=15.0
2023-11-23 01:49:01,426 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3100, loss[loss=0.08416, simple_loss=0.1186, pruned_loss=0.01617, audio_tagging_loss=0.008709, over 14265.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.0941, pruned_loss=0.01485, audio_tagging_loss=0.009302, over 3037111.42 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:49:02,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=2184946.6666666665, ans=15.0
2023-11-23 01:49:02,273 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.99 vs. limit=15.0
2023-11-23 01:49:09,526 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327750
2023-11-23 01:49:18,859 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:49:39,092 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.214e+01 8.129e+01 8.806e+01 9.628e+01 1.604e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-23 01:49:53,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2185213.3333333335, ans=0.0
2023-11-23 01:50:05,784 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3150, loss[loss=0.06513, simple_loss=0.08931, pruned_loss=0.01358, audio_tagging_loss=0.006893, over 15220.00 frames. ], tot_loss[loss=0.07147, simple_loss=0.09472, pruned_loss=0.01484, audio_tagging_loss=0.009277, over 3041841.86 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 01:50:11,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2185280.0, ans=0.125
2023-11-23 01:50:13,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327800
2023-11-23 01:50:24,092 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.76 vs. limit=15.0
2023-11-23 01:50:26,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2185346.6666666665, ans=0.125
2023-11-23 01:50:42,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2185413.3333333335, ans=0.2
2023-11-23 01:51:07,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2185546.6666666665, ans=0.125
2023-11-23 01:51:09,981 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3200, loss[loss=0.06627, simple_loss=0.07755, pruned_loss=0.01244, audio_tagging_loss=0.01505, over 15643.00 frames. ], tot_loss[loss=0.0714, simple_loss=0.09448, pruned_loss=0.01479, audio_tagging_loss=0.009375, over 3051368.78 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:51:15,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2185613.3333333335, ans=0.0
2023-11-23 01:51:17,674 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327850
2023-11-23 01:51:20,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2185613.3333333335, ans=0.1
2023-11-23 01:51:33,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2185680.0, ans=0.2
2023-11-23 01:51:34,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2185680.0, ans=0.0
2023-11-23 01:51:43,217 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 01:51:49,573 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.239e+01 8.167e+01 8.814e+01 9.617e+01 1.193e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 01:52:02,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2185880.0, ans=0.125
2023-11-23 01:52:14,661 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3250, loss[loss=0.04784, simple_loss=0.05268, pruned_loss=0.01008, audio_tagging_loss=0.01142, over 18690.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09405, pruned_loss=0.01483, audio_tagging_loss=0.009404, over 3050465.12 frames. ], batch size: 75, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:52:23,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327900
2023-11-23 01:52:32,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.31 vs. limit=6.0
2023-11-23 01:52:32,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2186013.3333333335, ans=0.0
2023-11-23 01:52:51,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2186146.6666666665, ans=0.125
2023-11-23 01:52:58,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2186146.6666666665, ans=10.0
2023-11-23 01:53:12,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2186213.3333333335, ans=0.1
2023-11-23 01:53:18,592 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3300, loss[loss=0.07715, simple_loss=0.09929, pruned_loss=0.01745, audio_tagging_loss=0.01006, over 14385.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09325, pruned_loss=0.0145, audio_tagging_loss=0.0095, over 3051552.18 frames. ], batch size: 53, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:53:26,606 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 327950
2023-11-23 01:53:33,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2186346.6666666665, ans=0.1
2023-11-23 01:53:34,512 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.32 vs. limit=22.5
2023-11-23 01:53:40,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2186346.6666666665, ans=10.0
2023-11-23 01:53:48,421 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.73 vs. limit=15.0
2023-11-23 01:53:57,275 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.779e+01 8.400e+01 9.018e+01 9.669e+01 1.285e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 01:54:08,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2186546.6666666665, ans=0.125
2023-11-23 01:54:09,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2186546.6666666665, ans=0.0
2023-11-23 01:54:22,727 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3350, loss[loss=0.0644, simple_loss=0.08268, pruned_loss=0.01658, audio_tagging_loss=0.006485, over 15951.00 frames. ], tot_loss[loss=0.07065, simple_loss=0.09329, pruned_loss=0.01456, audio_tagging_loss=0.009438, over 3051464.56 frames. ], batch size: 61, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:54:27,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2186613.3333333335, ans=0.0
2023-11-23 01:54:30,349 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328000
2023-11-23 01:54:31,924 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-328000.pt
2023-11-23 01:55:13,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2186813.3333333335, ans=15.0
2023-11-23 01:55:18,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2186880.0, ans=0.1
2023-11-23 01:55:24,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2186880.0, ans=0.125
2023-11-23 01:55:30,831 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3400, loss[loss=0.0573, simple_loss=0.0754, pruned_loss=0.01108, audio_tagging_loss=0.008523, over 16116.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.09341, pruned_loss=0.01444, audio_tagging_loss=0.009351, over 3055835.44 frames. ], batch size: 61, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:55:39,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328050
2023-11-23 01:55:42,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2186946.6666666665, ans=0.0
2023-11-23 01:56:04,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2187080.0, ans=0.0
2023-11-23 01:56:09,475 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.501e+01 8.298e+01 8.922e+01 9.490e+01 1.312e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 01:56:11,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2187146.6666666665, ans=0.125
2023-11-23 01:56:21,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.21 vs. limit=6.0
2023-11-23 01:56:35,890 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3450, loss[loss=0.08624, simple_loss=0.1077, pruned_loss=0.02287, audio_tagging_loss=0.009501, over 13531.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09399, pruned_loss=0.01451, audio_tagging_loss=0.009311, over 3052016.09 frames. ], batch size: 53, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:56:43,045 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328100
2023-11-23 01:56:43,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2187280.0, ans=0.05
2023-11-23 01:56:48,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2187346.6666666665, ans=0.0
2023-11-23 01:57:15,047 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.48 vs. limit=22.5
2023-11-23 01:57:39,737 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3500, loss[loss=0.05832, simple_loss=0.0808, pruned_loss=0.008223, audio_tagging_loss=0.009697, over 16580.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09351, pruned_loss=0.01437, audio_tagging_loss=0.009174, over 3044823.32 frames. ], batch size: 63, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:57:47,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328150
2023-11-23 01:57:59,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2187680.0, ans=0.125
2023-11-23 01:58:00,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2187680.0, ans=0.125
2023-11-23 01:58:13,423 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 01:58:18,286 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.297e+01 8.906e+01 9.538e+01 1.458e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 01:58:40,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2187880.0, ans=0.0
2023-11-23 01:58:44,548 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3550, loss[loss=0.06228, simple_loss=0.08842, pruned_loss=0.009225, audio_tagging_loss=0.008845, over 15026.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09327, pruned_loss=0.0143, audio_tagging_loss=0.009108, over 3042385.95 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:58:53,247 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328200
2023-11-23 01:59:29,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2188146.6666666665, ans=0.125
2023-11-23 01:59:44,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2188213.3333333335, ans=0.2
2023-11-23 01:59:50,116 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3600, loss[loss=0.07108, simple_loss=0.1018, pruned_loss=0.01298, audio_tagging_loss=0.007178, over 14502.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09287, pruned_loss=0.01426, audio_tagging_loss=0.009086, over 3046883.71 frames. ], batch size: 53, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 01:59:54,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2188280.0, ans=0.125
2023-11-23 01:59:57,454 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328250
2023-11-23 02:00:26,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2188413.3333333335, ans=0.0
2023-11-23 02:00:27,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2188480.0, ans=0.1
2023-11-23 02:00:28,192 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.210e+01 8.927e+01 9.741e+01 1.268e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 02:00:34,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2188480.0, ans=0.0
2023-11-23 02:00:52,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2188613.3333333335, ans=0.0
2023-11-23 02:00:53,679 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3650, loss[loss=0.07253, simple_loss=0.08788, pruned_loss=0.01778, audio_tagging_loss=0.01081, over 15157.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09295, pruned_loss=0.01427, audio_tagging_loss=0.009058, over 3048534.38 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:00:55,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2188613.3333333335, ans=0.09899494936611666
2023-11-23 02:00:55,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2188613.3333333335, ans=0.125
2023-11-23 02:00:59,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2188613.3333333335, ans=0.1
2023-11-23 02:01:00,886 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328300
2023-11-23 02:01:27,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2188746.6666666665, ans=0.125
2023-11-23 02:01:38,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2188813.3333333335, ans=0.0
2023-11-23 02:01:42,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2188813.3333333335, ans=0.125
2023-11-23 02:01:51,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2188880.0, ans=0.0
2023-11-23 02:01:57,290 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3700, loss[loss=0.05926, simple_loss=0.08106, pruned_loss=0.01178, audio_tagging_loss=0.00695, over 14076.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.0931, pruned_loss=0.01429, audio_tagging_loss=0.008943, over 3052331.02 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:02:04,632 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.61 vs. limit=22.5
2023-11-23 02:02:05,399 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328350
2023-11-23 02:02:11,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2189013.3333333335, ans=0.125
2023-11-23 02:02:11,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2189013.3333333335, ans=0.0
2023-11-23 02:02:36,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2189146.6666666665, ans=0.125
2023-11-23 02:02:37,560 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.324e+01 8.915e+01 9.653e+01 1.223e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 02:02:43,651 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.67 vs. limit=10.0
2023-11-23 02:02:51,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2189213.3333333335, ans=0.2
2023-11-23 02:03:03,364 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3750, loss[loss=0.08234, simple_loss=0.114, pruned_loss=0.01662, audio_tagging_loss=0.008736, over 14965.00 frames. ], tot_loss[loss=0.0706, simple_loss=0.09408, pruned_loss=0.01455, audio_tagging_loss=0.009012, over 3053371.70 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:03:10,867 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328400
2023-11-23 02:03:13,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2189280.0, ans=0.07
2023-11-23 02:03:19,874 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:03:20,580 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.09 vs. limit=6.0
2023-11-23 02:03:48,577 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:03:48,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2189480.0, ans=0.0
2023-11-23 02:03:52,783 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.89 vs. limit=15.0
2023-11-23 02:04:04,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2189546.6666666665, ans=0.2
2023-11-23 02:04:06,827 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3800, loss[loss=0.07968, simple_loss=0.1044, pruned_loss=0.01678, audio_tagging_loss=0.01072, over 16009.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09384, pruned_loss=0.0145, audio_tagging_loss=0.009144, over 3053605.20 frames. ], batch size: 59, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:04:08,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2189613.3333333335, ans=0.1
2023-11-23 02:04:14,413 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328450
2023-11-23 02:04:20,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2189680.0, ans=0.05
2023-11-23 02:04:26,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2189680.0, ans=0.125
2023-11-23 02:04:37,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2189746.6666666665, ans=0.125
2023-11-23 02:04:47,089 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 8.455e+01 8.996e+01 9.813e+01 1.168e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 02:04:59,390 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:05:06,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2189880.0, ans=0.0
2023-11-23 02:05:06,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2189880.0, ans=0.07
2023-11-23 02:05:06,974 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:05:09,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2189946.6666666665, ans=0.0
2023-11-23 02:05:10,389 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3850, loss[loss=0.05826, simple_loss=0.07751, pruned_loss=0.01049, audio_tagging_loss=0.009009, over 15177.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09338, pruned_loss=0.01444, audio_tagging_loss=0.00928, over 3056465.01 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:05:19,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328500
2023-11-23 02:06:05,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2190213.3333333335, ans=0.5
2023-11-23 02:06:14,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2190280.0, ans=0.125
2023-11-23 02:06:15,819 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3900, loss[loss=0.06883, simple_loss=0.09142, pruned_loss=0.01335, audio_tagging_loss=0.009768, over 14969.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09397, pruned_loss=0.01469, audio_tagging_loss=0.009256, over 3051049.52 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:06:19,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2190280.0, ans=0.2
2023-11-23 02:06:23,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328550
2023-11-23 02:06:23,850 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.76 vs. limit=15.0
2023-11-23 02:06:26,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2190280.0, ans=0.0
2023-11-23 02:06:29,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2190346.6666666665, ans=0.5
2023-11-23 02:06:33,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2190346.6666666665, ans=0.0
2023-11-23 02:06:46,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2190413.3333333335, ans=0.125
2023-11-23 02:06:53,554 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.052e+01 8.828e+01 9.660e+01 1.313e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 02:07:18,501 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 3950, loss[loss=0.06593, simple_loss=0.09179, pruned_loss=0.01373, audio_tagging_loss=0.006297, over 14880.00 frames. ], tot_loss[loss=0.0705, simple_loss=0.0934, pruned_loss=0.01444, audio_tagging_loss=0.009363, over 3053060.09 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:07:19,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2190613.3333333335, ans=0.125
2023-11-23 02:07:25,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328600
2023-11-23 02:07:30,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2190680.0, ans=0.0
2023-11-23 02:07:34,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2190680.0, ans=0.0
2023-11-23 02:07:40,413 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.69 vs. limit=15.0
2023-11-23 02:07:59,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2190813.3333333335, ans=0.09899494936611666
2023-11-23 02:08:00,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2190813.3333333335, ans=0.0
2023-11-23 02:08:03,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2190813.3333333335, ans=0.125
2023-11-23 02:08:14,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2190880.0, ans=0.125
2023-11-23 02:08:14,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2190880.0, ans=0.1
2023-11-23 02:08:16,088 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.51 vs. limit=22.5
2023-11-23 02:08:19,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2190880.0, ans=0.125
2023-11-23 02:08:22,784 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4000, loss[loss=0.07298, simple_loss=0.1057, pruned_loss=0.01129, audio_tagging_loss=0.008848, over 15272.00 frames. ], tot_loss[loss=0.07077, simple_loss=0.09382, pruned_loss=0.01444, audio_tagging_loss=0.009423, over 3049411.54 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:08:22,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2190946.6666666665, ans=0.125
2023-11-23 02:08:24,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2190946.6666666665, ans=0.125
2023-11-23 02:08:30,465 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328650
2023-11-23 02:08:32,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2190946.6666666665, ans=0.2
2023-11-23 02:08:44,569 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.18 vs. limit=10.0
2023-11-23 02:08:46,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2191013.3333333335, ans=0.125
2023-11-23 02:08:48,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2191080.0, ans=0.125
2023-11-23 02:09:03,451 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.297e+01 8.975e+01 9.665e+01 1.273e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-23 02:09:06,684 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.43 vs. limit=15.0
2023-11-23 02:09:28,252 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4050, loss[loss=0.06988, simple_loss=0.1019, pruned_loss=0.01271, audio_tagging_loss=0.006236, over 14824.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09348, pruned_loss=0.01436, audio_tagging_loss=0.009464, over 3042900.48 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:09:29,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2191280.0, ans=0.1
2023-11-23 02:09:32,518 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:09:36,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328700
2023-11-23 02:09:40,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2191346.6666666665, ans=0.025
2023-11-23 02:10:32,646 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4100, loss[loss=0.06208, simple_loss=0.0829, pruned_loss=0.01131, audio_tagging_loss=0.00932, over 15267.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09352, pruned_loss=0.01438, audio_tagging_loss=0.009483, over 3043071.74 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:10:37,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2191613.3333333335, ans=0.125
2023-11-23 02:10:40,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328750
2023-11-23 02:10:48,096 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.49 vs. limit=15.0
2023-11-23 02:11:13,908 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 8.198e+01 8.662e+01 9.320e+01 1.292e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-23 02:11:14,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2191813.3333333335, ans=0.125
2023-11-23 02:11:16,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2191813.3333333335, ans=0.5
2023-11-23 02:11:36,023 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4150, loss[loss=0.06317, simple_loss=0.08035, pruned_loss=0.01142, audio_tagging_loss=0.01158, over 15333.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09239, pruned_loss=0.01416, audio_tagging_loss=0.009428, over 3039869.97 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:11:36,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2191946.6666666665, ans=0.0
2023-11-23 02:11:40,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2191946.6666666665, ans=0.125
2023-11-23 02:11:43,576 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328800
2023-11-23 02:11:53,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2192013.3333333335, ans=0.125
2023-11-23 02:11:55,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2192013.3333333335, ans=0.125
2023-11-23 02:11:56,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.66 vs. limit=10.0
2023-11-23 02:11:58,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2192013.3333333335, ans=0.125
2023-11-23 02:12:09,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2192080.0, ans=0.0
2023-11-23 02:12:17,406 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.79 vs. limit=15.0
2023-11-23 02:12:18,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2192146.6666666665, ans=0.2
2023-11-23 02:12:18,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2192146.6666666665, ans=0.125
2023-11-23 02:12:22,723 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:12:29,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2192213.3333333335, ans=0.125
2023-11-23 02:12:31,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2192213.3333333335, ans=0.0
2023-11-23 02:12:40,185 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4200, loss[loss=0.0665, simple_loss=0.08119, pruned_loss=0.01537, audio_tagging_loss=0.01054, over 15366.00 frames. ], tot_loss[loss=0.07041, simple_loss=0.09331, pruned_loss=0.01441, audio_tagging_loss=0.009349, over 3050244.95 frames. ], batch size: 60, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:12:48,746 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328850
2023-11-23 02:12:53,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2192346.6666666665, ans=0.0
2023-11-23 02:13:01,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2192346.6666666665, ans=0.0
2023-11-23 02:13:06,725 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.85 vs. limit=15.0
2023-11-23 02:13:16,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2192413.3333333335, ans=0.125
2023-11-23 02:13:20,634 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.473e+01 9.086e+01 9.924e+01 1.385e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 02:13:44,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2192613.3333333335, ans=0.125
2023-11-23 02:13:44,954 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4250, loss[loss=0.06042, simple_loss=0.07265, pruned_loss=0.014, audio_tagging_loss=0.01009, over 15003.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09332, pruned_loss=0.01441, audio_tagging_loss=0.009248, over 3049860.84 frames. ], batch size: 57, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:13:52,415 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328900
2023-11-23 02:14:38,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2192880.0, ans=0.125
2023-11-23 02:14:44,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2192880.0, ans=0.0
2023-11-23 02:14:49,007 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4300, loss[loss=0.06599, simple_loss=0.0886, pruned_loss=0.01457, audio_tagging_loss=0.007124, over 14527.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09362, pruned_loss=0.01421, audio_tagging_loss=0.009112, over 3063944.37 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:14:55,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2192946.6666666665, ans=0.125
2023-11-23 02:14:56,490 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 328950
2023-11-23 02:15:01,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2193013.3333333335, ans=0.125
2023-11-23 02:15:03,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2193013.3333333335, ans=0.125
2023-11-23 02:15:06,340 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.85 vs. limit=15.0
2023-11-23 02:15:14,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2193080.0, ans=0.0
2023-11-23 02:15:27,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2193146.6666666665, ans=0.2
2023-11-23 02:15:27,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2193146.6666666665, ans=0.125
2023-11-23 02:15:30,846 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.649e+01 8.452e+01 8.978e+01 9.927e+01 1.263e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 02:15:33,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2193146.6666666665, ans=0.0
2023-11-23 02:15:53,674 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4350, loss[loss=0.08254, simple_loss=0.1104, pruned_loss=0.01855, audio_tagging_loss=0.008805, over 14680.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09379, pruned_loss=0.01429, audio_tagging_loss=0.009064, over 3057579.06 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 16.0
2023-11-23 02:16:02,568 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329000
2023-11-23 02:16:06,162 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=12.47 vs. limit=15.0
2023-11-23 02:16:16,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.60 vs. limit=6.0
2023-11-23 02:16:31,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2193480.0, ans=0.125
2023-11-23 02:16:46,221 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.21 vs. limit=15.0
2023-11-23 02:16:58,965 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4400, loss[loss=0.04201, simple_loss=0.05481, pruned_loss=0.006438, audio_tagging_loss=0.008172, over 14844.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09354, pruned_loss=0.01419, audio_tagging_loss=0.009114, over 3053637.37 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:17:07,022 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329050
2023-11-23 02:17:09,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2193613.3333333335, ans=0.95
2023-11-23 02:17:09,862 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.60 vs. limit=10.0
2023-11-23 02:17:10,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2193680.0, ans=0.1
2023-11-23 02:17:12,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2193680.0, ans=0.125
2023-11-23 02:17:17,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2193680.0, ans=0.0
2023-11-23 02:17:23,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2193746.6666666665, ans=0.125
2023-11-23 02:17:30,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2193746.6666666665, ans=0.0
2023-11-23 02:17:40,399 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.171e+01 8.943e+01 9.862e+01 1.169e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 02:17:43,532 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.09 vs. limit=15.0
2023-11-23 02:18:00,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2193880.0, ans=0.125
2023-11-23 02:18:03,068 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4450, loss[loss=0.06872, simple_loss=0.09373, pruned_loss=0.01315, audio_tagging_loss=0.008708, over 13942.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.0938, pruned_loss=0.0142, audio_tagging_loss=0.0091, over 3048137.30 frames. ], batch size: 54, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:18:09,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2193946.6666666665, ans=0.05
2023-11-23 02:18:10,650 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329100
2023-11-23 02:19:04,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2194213.3333333335, ans=15.0
2023-11-23 02:19:06,931 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.10 vs. limit=10.0
2023-11-23 02:19:07,656 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4500, loss[loss=0.05633, simple_loss=0.07591, pruned_loss=0.01084, audio_tagging_loss=0.007534, over 14697.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09325, pruned_loss=0.01417, audio_tagging_loss=0.009093, over 3044062.47 frames. ], batch size: 55, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:19:13,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2194280.0, ans=0.125
2023-11-23 02:19:16,364 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329150
2023-11-23 02:19:17,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2194280.0, ans=0.125
2023-11-23 02:19:17,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2194280.0, ans=0.1
2023-11-23 02:19:22,604 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.32 vs. limit=15.0
2023-11-23 02:19:48,966 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.124e+01 8.364e+01 9.070e+01 9.690e+01 1.146e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-23 02:19:55,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2194480.0, ans=0.125
2023-11-23 02:20:08,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2194546.6666666665, ans=0.125
2023-11-23 02:20:13,055 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4550, loss[loss=0.06397, simple_loss=0.08374, pruned_loss=0.01232, audio_tagging_loss=0.009773, over 14579.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.0928, pruned_loss=0.01402, audio_tagging_loss=0.009045, over 3033545.15 frames. ], batch size: 56, lr: 2.44e-03, grad_scale: 32.0
2023-11-23 02:20:14,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2194613.3333333335, ans=0.125
2023-11-23 02:20:20,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329200
2023-11-23 02:21:01,809 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:21:16,488 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4600, loss[loss=0.07713, simple_loss=0.09774, pruned_loss=0.01701, audio_tagging_loss=0.01125, over 15831.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09251, pruned_loss=0.01409, audio_tagging_loss=0.009107, over 3035207.65 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:21:24,530 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329250
2023-11-23 02:21:26,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2194946.6666666665, ans=0.0
2023-11-23 02:21:27,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2194946.6666666665, ans=0.2
2023-11-23 02:21:58,553 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.216e+01 8.878e+01 9.478e+01 1.349e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-23 02:22:00,743 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0
2023-11-23 02:22:15,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2195213.3333333335, ans=0.125
2023-11-23 02:22:21,461 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4650, loss[loss=0.06866, simple_loss=0.08796, pruned_loss=0.01634, audio_tagging_loss=0.008332, over 14149.00 frames. ], tot_loss[loss=0.07032, simple_loss=0.09345, pruned_loss=0.01445, audio_tagging_loss=0.009145, over 3040206.13 frames. ], batch size: 52, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:22:24,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2195280.0, ans=0.125
2023-11-23 02:22:29,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329300
2023-11-23 02:22:30,974 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.47 vs. limit=22.5
2023-11-23 02:22:43,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2195346.6666666665, ans=0.125
2023-11-23 02:23:04,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2195480.0, ans=0.2
2023-11-23 02:23:27,303 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4700, loss[loss=0.0648, simple_loss=0.07985, pruned_loss=0.01501, audio_tagging_loss=0.009862, over 15881.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09281, pruned_loss=0.01436, audio_tagging_loss=0.009243, over 3043617.55 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:23:34,817 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329350
2023-11-23 02:23:55,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2195746.6666666665, ans=0.125
2023-11-23 02:23:56,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2195746.6666666665, ans=0.0
2023-11-23 02:24:09,207 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.732e+01 8.291e+01 9.016e+01 9.707e+01 1.177e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 02:24:15,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.04 vs. limit=15.0
2023-11-23 02:24:31,726 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4750, loss[loss=0.07975, simple_loss=0.1059, pruned_loss=0.01797, audio_tagging_loss=0.008837, over 14634.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09319, pruned_loss=0.01459, audio_tagging_loss=0.009342, over 3041398.83 frames. ], batch size: 52, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:24:37,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2195946.6666666665, ans=0.125
2023-11-23 02:24:39,053 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329400
2023-11-23 02:24:49,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2196013.3333333335, ans=0.1
2023-11-23 02:25:09,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2196080.0, ans=0.0
2023-11-23 02:25:28,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2196213.3333333335, ans=0.0
2023-11-23 02:25:33,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2196213.3333333335, ans=0.2
2023-11-23 02:25:37,126 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4800, loss[loss=0.06672, simple_loss=0.08017, pruned_loss=0.01693, audio_tagging_loss=0.009705, over 13825.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09322, pruned_loss=0.01458, audio_tagging_loss=0.009371, over 3043494.69 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:25:44,687 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329450
2023-11-23 02:25:54,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2196346.6666666665, ans=0.1
2023-11-23 02:25:54,669 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.36 vs. limit=15.0
2023-11-23 02:26:13,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2196413.3333333335, ans=0.125
2023-11-23 02:26:19,713 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.219e+01 8.831e+01 9.548e+01 1.229e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 02:26:30,004 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.20 vs. limit=8.0
2023-11-23 02:26:38,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2196546.6666666665, ans=0.2
2023-11-23 02:26:39,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.41 vs. limit=22.5
2023-11-23 02:26:41,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2196613.3333333335, ans=0.2
2023-11-23 02:26:43,275 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4850, loss[loss=0.0772, simple_loss=0.09573, pruned_loss=0.01645, audio_tagging_loss=0.01288, over 15138.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09318, pruned_loss=0.01445, audio_tagging_loss=0.009503, over 3046083.41 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:26:50,761 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329500
2023-11-23 02:26:54,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2196680.0, ans=10.0
2023-11-23 02:26:58,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2196680.0, ans=0.125
2023-11-23 02:27:03,109 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:27:03,439 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.47 vs. limit=15.0
2023-11-23 02:27:21,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2196813.3333333335, ans=0.0
2023-11-23 02:27:21,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2196813.3333333335, ans=0.5
2023-11-23 02:27:33,523 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.44 vs. limit=10.0
2023-11-23 02:27:42,330 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.00 vs. limit=15.0
2023-11-23 02:27:47,949 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4900, loss[loss=0.08125, simple_loss=0.1138, pruned_loss=0.01778, audio_tagging_loss=0.006585, over 15087.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09239, pruned_loss=0.0142, audio_tagging_loss=0.009463, over 3041500.74 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:27:51,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2196946.6666666665, ans=0.1
2023-11-23 02:27:55,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329550
2023-11-23 02:28:18,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2197080.0, ans=0.0
2023-11-23 02:28:29,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2197146.6666666665, ans=0.0
2023-11-23 02:28:31,634 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.423e+01 8.170e+01 8.747e+01 9.254e+01 1.120e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-23 02:28:42,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2197213.3333333335, ans=0.125
2023-11-23 02:28:52,724 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 4950, loss[loss=0.06749, simple_loss=0.09094, pruned_loss=0.01268, audio_tagging_loss=0.009337, over 15684.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09319, pruned_loss=0.01431, audio_tagging_loss=0.009265, over 3047135.48 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:29:00,748 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329600
2023-11-23 02:29:02,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2197280.0, ans=0.125
2023-11-23 02:29:15,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2197346.6666666665, ans=0.0
2023-11-23 02:29:17,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2197346.6666666665, ans=0.0
2023-11-23 02:29:27,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2197413.3333333335, ans=0.05
2023-11-23 02:29:33,534 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.94 vs. limit=15.0
2023-11-23 02:29:36,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2197480.0, ans=0.0
2023-11-23 02:29:51,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2197546.6666666665, ans=0.125
2023-11-23 02:29:52,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2197546.6666666665, ans=0.125
2023-11-23 02:29:58,257 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.30 vs. limit=10.0
2023-11-23 02:29:59,506 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5000, loss[loss=0.05573, simple_loss=0.07445, pruned_loss=0.008311, audio_tagging_loss=0.01019, over 14379.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.092, pruned_loss=0.01412, audio_tagging_loss=0.009128, over 3041454.50 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:30:07,558 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329650
2023-11-23 02:30:15,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2197680.0, ans=0.125
2023-11-23 02:30:15,207 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=1.018e-02
2023-11-23 02:30:41,660 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.477e+01 9.069e+01 9.699e+01 1.131e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-23 02:30:50,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2197880.0, ans=0.125
2023-11-23 02:30:57,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2197880.0, ans=0.125
2023-11-23 02:31:01,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2197880.0, ans=0.125
2023-11-23 02:31:02,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2197880.0, ans=0.125
2023-11-23 02:31:04,435 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5050, loss[loss=0.05152, simple_loss=0.07224, pruned_loss=0.007294, audio_tagging_loss=0.008109, over 15429.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09135, pruned_loss=0.01398, audio_tagging_loss=0.009123, over 3040030.14 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:31:07,794 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.42 vs. limit=22.5
2023-11-23 02:31:11,523 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.50 vs. limit=22.5
2023-11-23 02:31:11,928 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329700
2023-11-23 02:31:12,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2197946.6666666665, ans=0.125
2023-11-23 02:31:17,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2198013.3333333335, ans=0.1
2023-11-23 02:31:29,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2198080.0, ans=0.125
2023-11-23 02:31:43,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2198146.6666666665, ans=0.125
2023-11-23 02:32:08,959 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5100, loss[loss=0.05395, simple_loss=0.06967, pruned_loss=0.008368, audio_tagging_loss=0.01075, over 14039.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09096, pruned_loss=0.01406, audio_tagging_loss=0.009083, over 3038269.94 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:32:16,335 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329750
2023-11-23 02:32:27,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2198346.6666666665, ans=0.125
2023-11-23 02:32:37,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2198413.3333333335, ans=0.125
2023-11-23 02:32:38,870 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.08 vs. limit=15.0
2023-11-23 02:32:51,748 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.231e+01 8.134e+01 8.796e+01 9.528e+01 1.100e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 02:32:52,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2198480.0, ans=0.5
2023-11-23 02:33:07,375 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.35 vs. limit=12.0
2023-11-23 02:33:13,789 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5150, loss[loss=0.08453, simple_loss=0.1207, pruned_loss=0.01902, audio_tagging_loss=0.005166, over 15522.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09193, pruned_loss=0.0143, audio_tagging_loss=0.009062, over 3044262.34 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:33:23,177 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329800
2023-11-23 02:34:03,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2198813.3333333335, ans=0.0
2023-11-23 02:34:08,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2198880.0, ans=0.125
2023-11-23 02:34:20,438 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5200, loss[loss=0.07986, simple_loss=0.107, pruned_loss=0.0176, audio_tagging_loss=0.008768, over 16263.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09306, pruned_loss=0.01447, audio_tagging_loss=0.008996, over 3043717.63 frames. ], batch size: 63, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:34:20,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2198946.6666666665, ans=0.1
2023-11-23 02:34:27,865 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329850
2023-11-23 02:34:39,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2199013.3333333335, ans=0.125
2023-11-23 02:34:42,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2199013.3333333335, ans=0.1
2023-11-23 02:35:05,350 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.387e+01 9.079e+01 9.668e+01 1.776e+02, threshold=1.816e+02, percent-clipped=1.0
2023-11-23 02:35:25,328 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5250, loss[loss=0.06145, simple_loss=0.07919, pruned_loss=0.01165, audio_tagging_loss=0.01021, over 15545.00 frames. ], tot_loss[loss=0.07025, simple_loss=0.09365, pruned_loss=0.01447, audio_tagging_loss=0.008949, over 3043800.08 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:35:32,882 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329900
2023-11-23 02:35:33,666 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.44 vs. limit=15.0
2023-11-23 02:35:55,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2199413.3333333335, ans=0.125
2023-11-23 02:36:11,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2199480.0, ans=0.025
2023-11-23 02:36:30,303 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5300, loss[loss=0.06229, simple_loss=0.08, pruned_loss=0.01341, audio_tagging_loss=0.00888, over 15109.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09373, pruned_loss=0.01465, audio_tagging_loss=0.008968, over 3041673.52 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:36:35,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2199613.3333333335, ans=0.04949747468305833
2023-11-23 02:36:39,698 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 329950
2023-11-23 02:37:14,806 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.403e+01 8.848e+01 9.438e+01 1.127e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 02:37:17,791 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:37:22,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2199880.0, ans=0.025
2023-11-23 02:37:37,488 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5350, loss[loss=0.0647, simple_loss=0.08473, pruned_loss=0.01564, audio_tagging_loss=0.006687, over 14907.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09318, pruned_loss=0.01466, audio_tagging_loss=0.008996, over 3035562.14 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:37:44,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330000
2023-11-23 02:37:49,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2200013.3333333335, ans=0.2
2023-11-23 02:37:57,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2200013.3333333335, ans=0.1
2023-11-23 02:38:04,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2200080.0, ans=0.125
2023-11-23 02:38:19,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2200146.6666666665, ans=0.1
2023-11-23 02:38:20,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2200146.6666666665, ans=0.125
2023-11-23 02:38:26,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2200146.6666666665, ans=0.1
2023-11-23 02:38:27,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2200146.6666666665, ans=0.0
2023-11-23 02:38:42,202 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5400, loss[loss=0.08179, simple_loss=0.1058, pruned_loss=0.01867, audio_tagging_loss=0.01023, over 16377.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09274, pruned_loss=0.01457, audio_tagging_loss=0.009141, over 3043473.43 frames. ], batch size: 60, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:38:46,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2200280.0, ans=0.125
2023-11-23 02:38:49,849 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330050
2023-11-23 02:38:50,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2200280.0, ans=0.125
2023-11-23 02:38:56,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2200346.6666666665, ans=0.125
2023-11-23 02:38:59,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2200346.6666666665, ans=0.05
2023-11-23 02:39:15,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2200413.3333333335, ans=0.0
2023-11-23 02:39:18,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2200413.3333333335, ans=0.04949747468305833
2023-11-23 02:39:25,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2200480.0, ans=0.125
2023-11-23 02:39:27,025 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.068e+01 8.843e+01 9.502e+01 1.253e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 02:39:36,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2200546.6666666665, ans=0.0
2023-11-23 02:39:47,573 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5450, loss[loss=0.07437, simple_loss=0.1012, pruned_loss=0.01516, audio_tagging_loss=0.008624, over 16536.00 frames. ], tot_loss[loss=0.07089, simple_loss=0.09377, pruned_loss=0.01492, audio_tagging_loss=0.009078, over 3045182.65 frames. ], batch size: 60, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:39:53,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2200613.3333333335, ans=0.1
2023-11-23 02:39:54,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2200613.3333333335, ans=0.125
2023-11-23 02:39:56,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330100
2023-11-23 02:40:00,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2200680.0, ans=0.125
2023-11-23 02:40:17,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2200746.6666666665, ans=0.0
2023-11-23 02:40:28,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2200813.3333333335, ans=0.5
2023-11-23 02:40:38,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2200880.0, ans=0.125
2023-11-23 02:40:45,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2200880.0, ans=0.1
2023-11-23 02:40:52,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2200946.6666666665, ans=0.1
2023-11-23 02:40:53,891 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5500, loss[loss=0.05698, simple_loss=0.06732, pruned_loss=0.0113, audio_tagging_loss=0.01202, over 13993.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.0932, pruned_loss=0.01456, audio_tagging_loss=0.009175, over 3055048.57 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:41:01,958 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330150
2023-11-23 02:41:25,889 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.63 vs. limit=22.5
2023-11-23 02:41:33,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2201146.6666666665, ans=0.0
2023-11-23 02:41:37,664 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.752e+01 8.324e+01 8.941e+01 9.835e+01 1.232e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 02:41:39,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2201146.6666666665, ans=0.125
2023-11-23 02:41:43,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2201146.6666666665, ans=0.09899494936611666
2023-11-23 02:41:45,718 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.46 vs. limit=15.0
2023-11-23 02:41:49,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2201213.3333333335, ans=0.2
2023-11-23 02:41:50,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2201213.3333333335, ans=0.0
2023-11-23 02:41:57,986 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5550, loss[loss=0.07807, simple_loss=0.09874, pruned_loss=0.01872, audio_tagging_loss=0.009975, over 14411.00 frames. ], tot_loss[loss=0.07072, simple_loss=0.09371, pruned_loss=0.01462, audio_tagging_loss=0.009238, over 3047122.93 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:42:05,549 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330200
2023-11-23 02:42:21,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2201346.6666666665, ans=0.04949747468305833
2023-11-23 02:42:44,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2201480.0, ans=0.0
2023-11-23 02:42:54,957 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:42:54,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2201546.6666666665, ans=0.05
2023-11-23 02:43:02,323 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5600, loss[loss=0.06115, simple_loss=0.07538, pruned_loss=0.01312, audio_tagging_loss=0.01034, over 15368.00 frames. ], tot_loss[loss=0.07162, simple_loss=0.0949, pruned_loss=0.01492, audio_tagging_loss=0.009244, over 3048794.45 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:43:06,717 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.99 vs. limit=22.5
2023-11-23 02:43:10,569 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330250
2023-11-23 02:43:12,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2201613.3333333335, ans=0.1
2023-11-23 02:43:21,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2201680.0, ans=0.1
2023-11-23 02:43:39,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2201746.6666666665, ans=0.09899494936611666
2023-11-23 02:43:47,710 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.200e+01 8.820e+01 9.406e+01 1.283e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 02:43:48,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2201813.3333333335, ans=0.0
2023-11-23 02:43:49,103 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:43:53,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2201880.0, ans=0.0
2023-11-23 02:43:59,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2201880.0, ans=0.2
2023-11-23 02:43:59,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2201880.0, ans=0.1
2023-11-23 02:44:08,235 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5650, loss[loss=0.07834, simple_loss=0.09595, pruned_loss=0.02014, audio_tagging_loss=0.01022, over 14442.00 frames. ], tot_loss[loss=0.07154, simple_loss=0.09451, pruned_loss=0.01487, audio_tagging_loss=0.009413, over 3056603.94 frames. ], batch size: 54, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:44:15,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330300
2023-11-23 02:44:17,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2201946.6666666665, ans=0.0
2023-11-23 02:44:21,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2202013.3333333335, ans=0.125
2023-11-23 02:44:26,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2202013.3333333335, ans=0.1
2023-11-23 02:44:39,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2202080.0, ans=0.125
2023-11-23 02:44:40,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2202080.0, ans=0.0
2023-11-23 02:44:40,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2202080.0, ans=0.125
2023-11-23 02:45:11,978 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5700, loss[loss=0.08378, simple_loss=0.1121, pruned_loss=0.01725, audio_tagging_loss=0.01047, over 14657.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09368, pruned_loss=0.01469, audio_tagging_loss=0.009475, over 3054677.68 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:45:19,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2202280.0, ans=0.0
2023-11-23 02:45:19,576 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=15.0
2023-11-23 02:45:20,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330350
2023-11-23 02:45:57,497 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.129e+01 8.046e+01 8.620e+01 9.394e+01 1.276e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-23 02:46:08,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.86 vs. limit=10.0
2023-11-23 02:46:16,715 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5750, loss[loss=0.05812, simple_loss=0.07925, pruned_loss=0.007983, audio_tagging_loss=0.01051, over 15885.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09355, pruned_loss=0.01446, audio_tagging_loss=0.009314, over 3055546.68 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:46:17,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2202613.3333333335, ans=0.2
2023-11-23 02:46:23,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2202613.3333333335, ans=0.125
2023-11-23 02:46:24,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330400
2023-11-23 02:46:45,986 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.93 vs. limit=12.0
2023-11-23 02:46:46,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2202746.6666666665, ans=0.0
2023-11-23 02:46:57,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2202813.3333333335, ans=0.2
2023-11-23 02:47:04,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2202813.3333333335, ans=0.0
2023-11-23 02:47:05,275 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0
2023-11-23 02:47:09,279 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.19 vs. limit=15.0
2023-11-23 02:47:22,024 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5800, loss[loss=0.04375, simple_loss=0.05727, pruned_loss=0.005261, audio_tagging_loss=0.009854, over 14047.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09318, pruned_loss=0.01438, audio_tagging_loss=0.009225, over 3052123.31 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:47:29,591 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330450
2023-11-23 02:47:50,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2203080.0, ans=0.125
2023-11-23 02:48:06,943 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.335e+01 8.418e+01 9.019e+01 9.630e+01 1.229e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 02:48:14,614 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0
2023-11-23 02:48:22,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2203213.3333333335, ans=0.2
2023-11-23 02:48:26,336 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5850, loss[loss=0.06015, simple_loss=0.08812, pruned_loss=0.009605, audio_tagging_loss=0.006483, over 15002.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.0937, pruned_loss=0.01452, audio_tagging_loss=0.009062, over 3058072.22 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:48:33,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330500
2023-11-23 02:49:00,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2203413.3333333335, ans=0.1
2023-11-23 02:49:05,973 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.14 vs. limit=15.0
2023-11-23 02:49:23,519 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=14.01 vs. limit=15.0
2023-11-23 02:49:30,673 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5900, loss[loss=0.08237, simple_loss=0.1145, pruned_loss=0.01911, audio_tagging_loss=0.006002, over 16236.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09309, pruned_loss=0.0144, audio_tagging_loss=0.009018, over 3058452.87 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:49:38,204 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330550
2023-11-23 02:49:47,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2203680.0, ans=0.1
2023-11-23 02:50:02,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.98 vs. limit=15.0
2023-11-23 02:50:07,057 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=12.0
2023-11-23 02:50:13,389 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.09 vs. limit=15.0
2023-11-23 02:50:15,292 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.273e+01 9.005e+01 9.559e+01 1.170e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 02:50:35,448 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 5950, loss[loss=0.07454, simple_loss=0.09821, pruned_loss=0.01532, audio_tagging_loss=0.01012, over 14582.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09343, pruned_loss=0.0145, audio_tagging_loss=0.009085, over 3057745.99 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 02:50:36,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2203946.6666666665, ans=0.125
2023-11-23 02:50:43,464 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330600
2023-11-23 02:50:51,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2204013.3333333335, ans=0.125
2023-11-23 02:50:53,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2204013.3333333335, ans=0.125
2023-11-23 02:51:12,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2204080.0, ans=0.1
2023-11-23 02:51:24,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2204146.6666666665, ans=0.1
2023-11-23 02:51:40,722 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6000, loss[loss=0.07687, simple_loss=0.1067, pruned_loss=0.01451, audio_tagging_loss=0.00902, over 15437.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09267, pruned_loss=0.0144, audio_tagging_loss=0.009108, over 3046217.44 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:51:40,726 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 02:52:24,766 INFO [train_asr.py:1253] (0/4) Epoch 28, validation: loss=0.05863, simple_loss=0.05128, pruned_loss=0.0051, audio_tagging_loss=0.02789, over 4681554.00 frames. 
2023-11-23 02:52:24,767 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 02:52:26,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2204280.0, ans=0.035
2023-11-23 02:52:32,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2204280.0, ans=0.07
2023-11-23 02:52:33,046 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330650
2023-11-23 02:52:51,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2204413.3333333335, ans=0.0
2023-11-23 02:53:10,016 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.180e+01 8.677e+01 9.605e+01 1.280e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-23 02:53:10,980 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.39 vs. limit=15.0
2023-11-23 02:53:11,342 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 02:53:12,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2204480.0, ans=0.125
2023-11-23 02:53:25,757 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.58 vs. limit=12.0
2023-11-23 02:53:30,870 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6050, loss[loss=0.07811, simple_loss=0.09872, pruned_loss=0.01868, audio_tagging_loss=0.01007, over 14004.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09301, pruned_loss=0.01439, audio_tagging_loss=0.009109, over 3046313.31 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:53:31,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.73 vs. limit=22.5
2023-11-23 02:53:37,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2204613.3333333335, ans=0.125
2023-11-23 02:53:38,449 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330700
2023-11-23 02:54:05,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2204746.6666666665, ans=0.0
2023-11-23 02:54:19,315 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.08 vs. limit=10.0
2023-11-23 02:54:34,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2204946.6666666665, ans=0.125
2023-11-23 02:54:35,072 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6100, loss[loss=0.07929, simple_loss=0.1012, pruned_loss=0.01851, audio_tagging_loss=0.0102, over 14325.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09165, pruned_loss=0.0142, audio_tagging_loss=0.00914, over 3040455.78 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:54:39,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2204946.6666666665, ans=0.2
2023-11-23 02:54:42,621 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330750
2023-11-23 02:55:00,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2205080.0, ans=0.0
2023-11-23 02:55:01,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.73 vs. limit=6.0
2023-11-23 02:55:18,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2205146.6666666665, ans=0.0
2023-11-23 02:55:20,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2205146.6666666665, ans=0.125
2023-11-23 02:55:21,025 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.895e+01 7.997e+01 8.605e+01 9.181e+01 1.192e+02, threshold=1.721e+02, percent-clipped=0.0
2023-11-23 02:55:26,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2205213.3333333335, ans=0.0
2023-11-23 02:55:27,973 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.08 vs. limit=22.5
2023-11-23 02:55:28,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2205213.3333333335, ans=0.125
2023-11-23 02:55:32,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2205213.3333333335, ans=0.125
2023-11-23 02:55:38,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2205280.0, ans=0.2
2023-11-23 02:55:39,778 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6150, loss[loss=0.05395, simple_loss=0.06862, pruned_loss=0.008976, audio_tagging_loss=0.01066, over 14895.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09209, pruned_loss=0.01423, audio_tagging_loss=0.009169, over 3043118.93 frames. ], batch size: 59, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:55:48,269 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330800
2023-11-23 02:55:52,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2205280.0, ans=0.2
2023-11-23 02:56:03,162 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:56:27,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2205480.0, ans=0.0
2023-11-23 02:56:31,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2205546.6666666665, ans=0.125
2023-11-23 02:56:33,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2205546.6666666665, ans=0.2
2023-11-23 02:56:45,229 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6200, loss[loss=0.06277, simple_loss=0.08988, pruned_loss=0.01125, audio_tagging_loss=0.006579, over 15805.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09219, pruned_loss=0.01435, audio_tagging_loss=0.009145, over 3047448.36 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:56:50,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2205613.3333333335, ans=0.025
2023-11-23 02:56:53,715 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330850
2023-11-23 02:57:25,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.78 vs. limit=22.5
2023-11-23 02:57:28,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2205813.3333333335, ans=0.0
2023-11-23 02:57:30,722 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.609e+01 8.249e+01 8.951e+01 9.781e+01 1.178e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 02:57:41,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2205880.0, ans=0.0
2023-11-23 02:57:49,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2205946.6666666665, ans=0.125
2023-11-23 02:57:50,216 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6250, loss[loss=0.0654, simple_loss=0.07923, pruned_loss=0.01473, audio_tagging_loss=0.01105, over 13828.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09227, pruned_loss=0.01434, audio_tagging_loss=0.009311, over 3045394.70 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:57:56,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2205946.6666666665, ans=0.125
2023-11-23 02:57:57,664 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330900
2023-11-23 02:57:59,435 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.89 vs. limit=12.0
2023-11-23 02:58:00,202 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 02:58:06,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2206013.3333333335, ans=0.125
2023-11-23 02:58:25,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2206080.0, ans=0.125
2023-11-23 02:58:33,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2206146.6666666665, ans=0.125
2023-11-23 02:58:49,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2206213.3333333335, ans=0.125
2023-11-23 02:58:54,312 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6300, loss[loss=0.06489, simple_loss=0.09141, pruned_loss=0.01188, audio_tagging_loss=0.00731, over 14833.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.0927, pruned_loss=0.01448, audio_tagging_loss=0.009424, over 3043723.00 frames. ], batch size: 53, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 02:59:01,958 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 330950
2023-11-23 02:59:20,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2206413.3333333335, ans=0.125
2023-11-23 02:59:40,240 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.187e+01 8.713e+01 9.379e+01 1.255e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-23 02:59:46,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2206546.6666666665, ans=0.125
2023-11-23 02:59:59,642 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6350, loss[loss=0.07302, simple_loss=0.09288, pruned_loss=0.01697, audio_tagging_loss=0.009611, over 14495.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.09293, pruned_loss=0.01432, audio_tagging_loss=0.009512, over 3046301.28 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:00:08,877 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331000
2023-11-23 03:00:14,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2206680.0, ans=0.1
2023-11-23 03:00:17,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2206680.0, ans=0.125
2023-11-23 03:00:18,429 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.64 vs. limit=22.5
2023-11-23 03:00:40,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2206813.3333333335, ans=0.125
2023-11-23 03:00:41,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2206813.3333333335, ans=0.0
2023-11-23 03:01:05,840 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6400, loss[loss=0.09045, simple_loss=0.1215, pruned_loss=0.02189, audio_tagging_loss=0.007801, over 16159.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09372, pruned_loss=0.01457, audio_tagging_loss=0.00948, over 3040935.28 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:01:07,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2206946.6666666665, ans=0.125
2023-11-23 03:01:11,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2206946.6666666665, ans=10.0
2023-11-23 03:01:13,435 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331050
2023-11-23 03:01:14,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2206946.6666666665, ans=0.05
2023-11-23 03:01:29,144 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.59 vs. limit=12.0
2023-11-23 03:01:52,586 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.579e+01 8.045e+01 8.704e+01 9.425e+01 1.351e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-23 03:01:53,194 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.92 vs. limit=6.0
2023-11-23 03:02:09,985 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6450, loss[loss=0.06853, simple_loss=0.08804, pruned_loss=0.01132, audio_tagging_loss=0.0132, over 15140.00 frames. ], tot_loss[loss=0.0707, simple_loss=0.09345, pruned_loss=0.01445, audio_tagging_loss=0.009525, over 3037945.75 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:02:16,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2207280.0, ans=0.0
2023-11-23 03:02:17,473 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331100
2023-11-23 03:03:05,364 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=23.75 vs. limit=22.5
2023-11-23 03:03:08,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2207546.6666666665, ans=0.0
2023-11-23 03:03:15,327 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6500, loss[loss=0.08744, simple_loss=0.1152, pruned_loss=0.02145, audio_tagging_loss=0.008407, over 15449.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09397, pruned_loss=0.01457, audio_tagging_loss=0.009519, over 3042456.23 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:03:23,457 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331150
2023-11-23 03:03:28,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2207680.0, ans=0.0
2023-11-23 03:03:43,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2207746.6666666665, ans=0.125
2023-11-23 03:03:54,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2207813.3333333335, ans=0.2
2023-11-23 03:04:01,371 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.379e+01 8.162e+01 8.865e+01 9.552e+01 1.149e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 03:04:01,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2207813.3333333335, ans=0.1
2023-11-23 03:04:10,010 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.60 vs. limit=12.0
2023-11-23 03:04:21,233 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6550, loss[loss=0.06705, simple_loss=0.08956, pruned_loss=0.01192, audio_tagging_loss=0.01035, over 14830.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09387, pruned_loss=0.0146, audio_tagging_loss=0.009375, over 3044746.94 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:04:26,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2207946.6666666665, ans=0.0
2023-11-23 03:04:27,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2207946.6666666665, ans=0.125
2023-11-23 03:04:28,963 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331200
2023-11-23 03:04:40,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2208013.3333333335, ans=0.09899494936611666
2023-11-23 03:05:20,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2208213.3333333335, ans=0.2
2023-11-23 03:05:24,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2208280.0, ans=0.125
2023-11-23 03:05:25,300 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6600, loss[loss=0.06534, simple_loss=0.08568, pruned_loss=0.01272, audio_tagging_loss=0.009777, over 14962.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09292, pruned_loss=0.01429, audio_tagging_loss=0.009328, over 3048178.48 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:05:25,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2208280.0, ans=0.0
2023-11-23 03:05:29,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2208280.0, ans=0.125
2023-11-23 03:05:32,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2208280.0, ans=15.0
2023-11-23 03:05:32,794 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331250
2023-11-23 03:05:42,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2208346.6666666665, ans=0.0
2023-11-23 03:05:52,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2208413.3333333335, ans=0.125
2023-11-23 03:06:12,267 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.516e+01 8.179e+01 8.913e+01 9.440e+01 1.641e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 03:06:29,576 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6650, loss[loss=0.0877, simple_loss=0.1187, pruned_loss=0.02002, audio_tagging_loss=0.008321, over 16652.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09164, pruned_loss=0.01413, audio_tagging_loss=0.009324, over 3045002.17 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:06:34,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2208613.3333333335, ans=0.125
2023-11-23 03:06:36,859 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:06:37,758 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331300
2023-11-23 03:06:39,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2208613.3333333335, ans=0.125
2023-11-23 03:07:26,874 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.42 vs. limit=12.0
2023-11-23 03:07:29,003 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.81 vs. limit=6.0
2023-11-23 03:07:33,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2208946.6666666665, ans=0.125
2023-11-23 03:07:34,622 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6700, loss[loss=0.08772, simple_loss=0.1179, pruned_loss=0.02116, audio_tagging_loss=0.007621, over 15866.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09169, pruned_loss=0.01415, audio_tagging_loss=0.009238, over 3047521.44 frames. ], batch size: 55, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:07:43,173 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331350
2023-11-23 03:08:11,789 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.37 vs. limit=22.5
2023-11-23 03:08:22,315 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.251e+01 8.844e+01 9.540e+01 1.359e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 03:08:40,195 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6750, loss[loss=0.0944, simple_loss=0.1296, pruned_loss=0.02248, audio_tagging_loss=0.007102, over 16103.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09282, pruned_loss=0.0144, audio_tagging_loss=0.009154, over 3044610.97 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:08:47,696 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331400
2023-11-23 03:09:14,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2209413.3333333335, ans=0.0
2023-11-23 03:09:18,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2209480.0, ans=0.0
2023-11-23 03:09:29,115 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:09:35,914 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.02 vs. limit=12.0
2023-11-23 03:09:45,063 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6800, loss[loss=0.06878, simple_loss=0.09293, pruned_loss=0.01255, audio_tagging_loss=0.009764, over 16358.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09272, pruned_loss=0.01442, audio_tagging_loss=0.009119, over 3042659.23 frames. ], batch size: 62, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:09:53,307 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331450
2023-11-23 03:10:03,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2209680.0, ans=0.2
2023-11-23 03:10:20,221 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.84 vs. limit=15.0
2023-11-23 03:10:31,998 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.723e+01 8.090e+01 8.741e+01 9.523e+01 1.182e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 03:10:37,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2209880.0, ans=0.125
2023-11-23 03:10:51,113 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6850, loss[loss=0.07663, simple_loss=0.09545, pruned_loss=0.01966, audio_tagging_loss=0.009243, over 15586.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09285, pruned_loss=0.01435, audio_tagging_loss=0.009089, over 3044736.05 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:10:58,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331500
2023-11-23 03:11:21,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2210080.0, ans=0.2
2023-11-23 03:11:24,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2210080.0, ans=0.125
2023-11-23 03:11:30,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2210146.6666666665, ans=0.125
2023-11-23 03:11:56,053 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6900, loss[loss=0.06657, simple_loss=0.08774, pruned_loss=0.01457, audio_tagging_loss=0.008127, over 16081.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09379, pruned_loss=0.01457, audio_tagging_loss=0.009071, over 3045326.33 frames. ], batch size: 61, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:11:56,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2210280.0, ans=0.0
2023-11-23 03:12:01,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2210280.0, ans=0.125
2023-11-23 03:12:03,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331550
2023-11-23 03:12:43,537 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.094e+01 8.690e+01 9.402e+01 1.184e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-23 03:12:44,898 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 03:12:47,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2210546.6666666665, ans=0.1
2023-11-23 03:12:56,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2210546.6666666665, ans=0.04949747468305833
2023-11-23 03:13:00,464 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 6950, loss[loss=0.06566, simple_loss=0.09318, pruned_loss=0.00987, audio_tagging_loss=0.009199, over 14352.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.09398, pruned_loss=0.01442, audio_tagging_loss=0.009172, over 3049660.25 frames. ], batch size: 52, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:13:08,062 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331600
2023-11-23 03:13:12,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2210680.0, ans=0.125
2023-11-23 03:13:21,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2210680.0, ans=0.125
2023-11-23 03:13:27,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2210746.6666666665, ans=0.125
2023-11-23 03:13:31,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2210746.6666666665, ans=0.2
2023-11-23 03:13:38,544 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.88 vs. limit=22.5
2023-11-23 03:13:41,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2210813.3333333335, ans=0.125
2023-11-23 03:13:51,114 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.12 vs. limit=6.0
2023-11-23 03:13:56,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2210880.0, ans=0.125
2023-11-23 03:14:06,053 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7000, loss[loss=0.06025, simple_loss=0.0824, pruned_loss=0.01023, audio_tagging_loss=0.008813, over 15530.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09207, pruned_loss=0.01399, audio_tagging_loss=0.009333, over 3051375.07 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:14:14,228 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331650
2023-11-23 03:14:23,536 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.35 vs. limit=10.0
2023-11-23 03:14:36,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2211080.0, ans=0.125
2023-11-23 03:14:52,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2211146.6666666665, ans=0.0
2023-11-23 03:14:53,598 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.328e+01 8.294e+01 8.963e+01 9.673e+01 1.557e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 03:15:01,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2211213.3333333335, ans=0.2
2023-11-23 03:15:06,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2211213.3333333335, ans=0.125
2023-11-23 03:15:10,767 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7050, loss[loss=0.06856, simple_loss=0.08492, pruned_loss=0.01525, audio_tagging_loss=0.01085, over 14824.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09152, pruned_loss=0.01402, audio_tagging_loss=0.009369, over 3044922.97 frames. ], batch size: 58, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:15:11,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.87 vs. limit=22.5
2023-11-23 03:15:18,697 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331700
2023-11-23 03:15:20,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2211280.0, ans=0.125
2023-11-23 03:15:20,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2211280.0, ans=0.125
2023-11-23 03:15:41,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2211413.3333333335, ans=0.035
2023-11-23 03:15:41,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2211413.3333333335, ans=0.0
2023-11-23 03:15:53,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2211480.0, ans=0.2
2023-11-23 03:16:11,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2211546.6666666665, ans=0.125
2023-11-23 03:16:14,854 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7100, loss[loss=0.07628, simple_loss=0.1065, pruned_loss=0.01279, audio_tagging_loss=0.01026, over 14757.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09176, pruned_loss=0.01397, audio_tagging_loss=0.009397, over 3054194.80 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:16:22,314 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331750
2023-11-23 03:16:32,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2211680.0, ans=0.125
2023-11-23 03:17:00,994 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=16.32 vs. limit=22.5
2023-11-23 03:17:02,522 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.317e+01 8.441e+01 9.062e+01 9.663e+01 1.297e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-23 03:17:06,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2211880.0, ans=0.125
2023-11-23 03:17:09,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2211880.0, ans=0.2
2023-11-23 03:17:19,054 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7150, loss[loss=0.06409, simple_loss=0.09186, pruned_loss=0.01166, audio_tagging_loss=0.006504, over 15182.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.0917, pruned_loss=0.01417, audio_tagging_loss=0.009377, over 3046293.14 frames. ], batch size: 56, lr: 2.43e-03, grad_scale: 16.0
2023-11-23 03:17:26,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331800
2023-11-23 03:17:30,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2211946.6666666665, ans=0.0
2023-11-23 03:17:31,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2212013.3333333335, ans=0.1
2023-11-23 03:17:33,516 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:18:21,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2212280.0, ans=0.125
2023-11-23 03:18:22,450 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7200, loss[loss=0.04896, simple_loss=0.06492, pruned_loss=0.006512, audio_tagging_loss=0.009987, over 16156.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09132, pruned_loss=0.01385, audio_tagging_loss=0.009438, over 3054843.37 frames. ], batch size: 67, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:18:29,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331850
2023-11-23 03:18:50,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2212413.3333333335, ans=0.0
2023-11-23 03:19:08,961 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.715e+01 8.113e+01 8.957e+01 9.713e+01 1.213e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 03:19:14,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2212546.6666666665, ans=0.125
2023-11-23 03:19:24,864 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7250, loss[loss=0.05319, simple_loss=0.06678, pruned_loss=0.01013, audio_tagging_loss=0.009673, over 14488.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.0921, pruned_loss=0.01412, audio_tagging_loss=0.00943, over 3049594.64 frames. ], batch size: 57, lr: 2.43e-03, grad_scale: 32.0
2023-11-23 03:19:27,377 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.57 vs. limit=12.0
2023-11-23 03:19:32,955 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331900
2023-11-23 03:19:59,218 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.15 vs. limit=15.0
2023-11-23 03:20:14,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2212880.0, ans=0.125
2023-11-23 03:20:28,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2212946.6666666665, ans=0.09899494936611666
2023-11-23 03:20:29,145 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7300, loss[loss=0.06807, simple_loss=0.08427, pruned_loss=0.01415, audio_tagging_loss=0.01179, over 15930.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09241, pruned_loss=0.01425, audio_tagging_loss=0.009272, over 3049271.48 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:20:30,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2212946.6666666665, ans=0.2
2023-11-23 03:20:35,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2212946.6666666665, ans=0.0
2023-11-23 03:20:36,944 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 331950
2023-11-23 03:20:43,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2213013.3333333335, ans=0.0
2023-11-23 03:20:49,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2213013.3333333335, ans=0.0
2023-11-23 03:21:09,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2213146.6666666665, ans=0.125
2023-11-23 03:21:09,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2213146.6666666665, ans=0.0
2023-11-23 03:21:15,974 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.909e+01 8.147e+01 8.723e+01 9.402e+01 1.286e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-23 03:21:32,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2213280.0, ans=0.2
2023-11-23 03:21:33,055 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7350, loss[loss=0.08757, simple_loss=0.1159, pruned_loss=0.0226, audio_tagging_loss=0.007022, over 15600.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09303, pruned_loss=0.01449, audio_tagging_loss=0.009073, over 3051803.69 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:21:35,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2213280.0, ans=0.0
2023-11-23 03:21:40,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332000
2023-11-23 03:21:41,850 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-332000.pt
2023-11-23 03:21:46,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2213280.0, ans=0.125
2023-11-23 03:22:39,833 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7400, loss[loss=0.06084, simple_loss=0.08215, pruned_loss=0.01292, audio_tagging_loss=0.006852, over 15793.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.09341, pruned_loss=0.01433, audio_tagging_loss=0.008898, over 3047694.89 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:22:47,302 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332050
2023-11-23 03:23:28,622 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.451e+01 8.955e+01 9.673e+01 1.193e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 03:23:44,634 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7450, loss[loss=0.08175, simple_loss=0.1128, pruned_loss=0.01521, audio_tagging_loss=0.01016, over 14984.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09349, pruned_loss=0.01434, audio_tagging_loss=0.008932, over 3043651.79 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:23:48,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2213946.6666666665, ans=0.0
2023-11-23 03:23:52,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332100
2023-11-23 03:24:04,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2214013.3333333335, ans=0.025
2023-11-23 03:24:19,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2214080.0, ans=0.125
2023-11-23 03:24:22,443 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:24:48,991 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7500, loss[loss=0.06832, simple_loss=0.09368, pruned_loss=0.01395, audio_tagging_loss=0.007523, over 15030.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09331, pruned_loss=0.01432, audio_tagging_loss=0.00907, over 3041296.77 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:24:54,894 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.94 vs. limit=15.0
2023-11-23 03:24:56,663 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332150
2023-11-23 03:25:01,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2214346.6666666665, ans=0.035
2023-11-23 03:25:18,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2214413.3333333335, ans=0.0
2023-11-23 03:25:37,865 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.184e+01 8.938e+01 9.763e+01 1.268e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 03:25:38,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2214480.0, ans=0.125
2023-11-23 03:25:47,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2214546.6666666665, ans=0.125
2023-11-23 03:25:52,773 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7550, loss[loss=0.04856, simple_loss=0.06543, pruned_loss=0.007065, audio_tagging_loss=0.008786, over 15959.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09303, pruned_loss=0.01425, audio_tagging_loss=0.009073, over 3045066.40 frames. ], batch size: 61, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:26:00,065 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332200
2023-11-23 03:26:06,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2214680.0, ans=0.0
2023-11-23 03:26:30,806 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.00 vs. limit=22.5
2023-11-23 03:26:55,373 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.27 vs. limit=22.5
2023-11-23 03:26:55,773 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7600, loss[loss=0.0522, simple_loss=0.0638, pruned_loss=0.009561, audio_tagging_loss=0.01074, over 14280.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09215, pruned_loss=0.01411, audio_tagging_loss=0.009015, over 3042080.53 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:26:59,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2214946.6666666665, ans=0.0
2023-11-23 03:27:01,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2214946.6666666665, ans=0.1
2023-11-23 03:27:03,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2214946.6666666665, ans=0.125
2023-11-23 03:27:04,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332250
2023-11-23 03:27:05,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2214946.6666666665, ans=0.95
2023-11-23 03:27:15,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2215013.3333333335, ans=0.0
2023-11-23 03:27:21,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2215080.0, ans=0.0
2023-11-23 03:27:21,771 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.69 vs. limit=15.0
2023-11-23 03:27:22,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2215080.0, ans=0.1
2023-11-23 03:27:23,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2215080.0, ans=22.5
2023-11-23 03:27:25,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2215080.0, ans=0.125
2023-11-23 03:27:44,365 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.025e+01 8.532e+01 9.208e+01 1.132e+02, threshold=1.706e+02, percent-clipped=0.0
2023-11-23 03:28:01,622 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7650, loss[loss=0.07373, simple_loss=0.09771, pruned_loss=0.01553, audio_tagging_loss=0.00934, over 15503.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09191, pruned_loss=0.01398, audio_tagging_loss=0.009001, over 3050025.75 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:28:09,143 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332300
2023-11-23 03:28:50,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2215480.0, ans=0.04949747468305833
2023-11-23 03:29:00,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2215546.6666666665, ans=0.0
2023-11-23 03:29:05,108 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7700, loss[loss=0.07512, simple_loss=0.1072, pruned_loss=0.01356, audio_tagging_loss=0.007962, over 15932.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09231, pruned_loss=0.014, audio_tagging_loss=0.009014, over 3046861.29 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:29:05,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2215613.3333333335, ans=0.2
2023-11-23 03:29:12,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332350
2023-11-23 03:29:14,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2215613.3333333335, ans=0.09899494936611666
2023-11-23 03:29:23,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2215680.0, ans=0.125
2023-11-23 03:29:23,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2215680.0, ans=0.1
2023-11-23 03:29:55,234 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.558e+01 8.060e+01 8.592e+01 9.630e+01 1.129e+02, threshold=1.718e+02, percent-clipped=0.0
2023-11-23 03:30:02,797 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:30:04,549 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.69 vs. limit=22.5
2023-11-23 03:30:08,647 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7750, loss[loss=0.05818, simple_loss=0.07378, pruned_loss=0.01091, audio_tagging_loss=0.01038, over 15288.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09233, pruned_loss=0.01401, audio_tagging_loss=0.009075, over 3039095.15 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:30:16,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332400
2023-11-23 03:30:41,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2216080.0, ans=0.125
2023-11-23 03:31:04,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2216213.3333333335, ans=0.125
2023-11-23 03:31:07,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2216213.3333333335, ans=0.125
2023-11-23 03:31:14,414 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7800, loss[loss=0.0624, simple_loss=0.08531, pruned_loss=0.009656, audio_tagging_loss=0.01009, over 15894.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09268, pruned_loss=0.014, audio_tagging_loss=0.009165, over 3036775.02 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:31:20,003 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.02 vs. limit=15.0
2023-11-23 03:31:21,762 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332450
2023-11-23 03:31:22,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2216280.0, ans=0.125
2023-11-23 03:31:31,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2216346.6666666665, ans=0.0
2023-11-23 03:32:04,606 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.076e+01 8.334e+01 8.987e+01 9.736e+01 1.242e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 03:32:18,100 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7850, loss[loss=0.06219, simple_loss=0.07979, pruned_loss=0.01234, audio_tagging_loss=0.009952, over 15815.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09243, pruned_loss=0.01419, audio_tagging_loss=0.009189, over 3036388.56 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:32:25,690 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332500
2023-11-23 03:33:06,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2216813.3333333335, ans=0.125
2023-11-23 03:33:13,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2216880.0, ans=0.025
2023-11-23 03:33:21,820 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7900, loss[loss=0.05881, simple_loss=0.07331, pruned_loss=0.01131, audio_tagging_loss=0.01084, over 14156.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09244, pruned_loss=0.01429, audio_tagging_loss=0.009244, over 3039239.94 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:33:29,919 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332550
2023-11-23 03:33:36,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2217013.3333333335, ans=0.125
2023-11-23 03:33:55,526 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.37 vs. limit=6.0
2023-11-23 03:33:58,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2217080.0, ans=0.125
2023-11-23 03:34:11,385 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.317e+01 8.969e+01 9.609e+01 1.150e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-23 03:34:11,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2217213.3333333335, ans=0.2
2023-11-23 03:34:18,816 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.16 vs. limit=15.0
2023-11-23 03:34:26,053 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 7950, loss[loss=0.07686, simple_loss=0.09254, pruned_loss=0.01874, audio_tagging_loss=0.01185, over 15432.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09203, pruned_loss=0.01424, audio_tagging_loss=0.009438, over 3032172.30 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:34:26,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2217280.0, ans=0.0
2023-11-23 03:34:33,841 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332600
2023-11-23 03:34:40,840 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 03:34:51,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2217413.3333333335, ans=0.0
2023-11-23 03:35:04,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2217480.0, ans=0.0
2023-11-23 03:35:04,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2217480.0, ans=10.0
2023-11-23 03:35:05,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2217480.0, ans=0.1
2023-11-23 03:35:19,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2217546.6666666665, ans=0.1
2023-11-23 03:35:28,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2217546.6666666665, ans=0.0
2023-11-23 03:35:30,610 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8000, loss[loss=0.09569, simple_loss=0.1397, pruned_loss=0.02115, audio_tagging_loss=0.004683, over 16108.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09168, pruned_loss=0.01422, audio_tagging_loss=0.009449, over 3036108.01 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:35:37,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.59 vs. limit=10.0
2023-11-23 03:35:37,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332650
2023-11-23 03:35:44,052 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:35:58,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2217746.6666666665, ans=0.0
2023-11-23 03:36:13,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2217813.3333333335, ans=0.2
2023-11-23 03:36:20,159 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.081e+01 8.720e+01 9.413e+01 1.910e+02, threshold=1.744e+02, percent-clipped=1.0
2023-11-23 03:36:22,974 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:36:33,483 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8050, loss[loss=0.08538, simple_loss=0.1136, pruned_loss=0.02093, audio_tagging_loss=0.007675, over 14957.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09072, pruned_loss=0.01416, audio_tagging_loss=0.00949, over 3032874.52 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:36:33,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2217946.6666666665, ans=0.0
2023-11-23 03:36:36,245 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:36:40,725 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332700
2023-11-23 03:37:37,081 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8100, loss[loss=0.07363, simple_loss=0.08777, pruned_loss=0.02004, audio_tagging_loss=0.009708, over 14580.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09105, pruned_loss=0.01419, audio_tagging_loss=0.009363, over 3030885.24 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:37:45,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332750
2023-11-23 03:37:52,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2218346.6666666665, ans=0.125
2023-11-23 03:38:09,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2218413.3333333335, ans=0.125
2023-11-23 03:38:27,681 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.654e+01 8.342e+01 8.867e+01 9.602e+01 1.203e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 03:38:39,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2218613.3333333335, ans=0.125
2023-11-23 03:38:40,752 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8150, loss[loss=0.06533, simple_loss=0.08369, pruned_loss=0.01318, audio_tagging_loss=0.0103, over 13617.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09133, pruned_loss=0.01417, audio_tagging_loss=0.009241, over 3031239.05 frames. ], batch size: 52, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:38:42,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2218613.3333333335, ans=0.125
2023-11-23 03:38:48,861 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332800
2023-11-23 03:38:50,563 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:39:24,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2218813.3333333335, ans=0.125
2023-11-23 03:39:41,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=2218880.0, ans=0.5
2023-11-23 03:39:45,666 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8200, loss[loss=0.0559, simple_loss=0.06654, pruned_loss=0.0105, audio_tagging_loss=0.01213, over 15653.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09139, pruned_loss=0.01423, audio_tagging_loss=0.009153, over 3032108.52 frames. ], batch size: 60, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:39:45,706 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 03:39:53,167 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332850
2023-11-23 03:39:56,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2218946.6666666665, ans=0.125
2023-11-23 03:40:06,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2219013.3333333335, ans=0.125
2023-11-23 03:40:06,814 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.03 vs. limit=12.0
2023-11-23 03:40:28,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2219146.6666666665, ans=0.2
2023-11-23 03:40:37,039 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.334e+01 8.200e+01 8.804e+01 9.476e+01 1.277e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-23 03:40:49,907 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8250, loss[loss=0.06472, simple_loss=0.08985, pruned_loss=0.008006, audio_tagging_loss=0.01179, over 15108.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09154, pruned_loss=0.0143, audio_tagging_loss=0.009079, over 3033734.92 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:40:52,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2219280.0, ans=0.125
2023-11-23 03:40:58,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332900
2023-11-23 03:41:36,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2219480.0, ans=0.125
2023-11-23 03:41:42,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2219546.6666666665, ans=0.125
2023-11-23 03:41:54,269 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8300, loss[loss=0.06795, simple_loss=0.08869, pruned_loss=0.01402, audio_tagging_loss=0.009588, over 15652.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.0916, pruned_loss=0.01454, audio_tagging_loss=0.009019, over 3033729.28 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:41:59,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2219613.3333333335, ans=0.05
2023-11-23 03:42:00,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2219613.3333333335, ans=0.125
2023-11-23 03:42:01,601 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 332950
2023-11-23 03:42:03,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2219613.3333333335, ans=0.0
2023-11-23 03:42:18,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2219746.6666666665, ans=0.1
2023-11-23 03:42:34,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2219813.3333333335, ans=0.2
2023-11-23 03:42:46,812 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.213e+01 8.531e+01 9.088e+01 9.532e+01 2.273e+02, threshold=1.818e+02, percent-clipped=2.0
2023-11-23 03:42:53,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.75 vs. limit=10.0
2023-11-23 03:42:54,519 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:42:55,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0
2023-11-23 03:42:57,842 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8350, loss[loss=0.06766, simple_loss=0.08936, pruned_loss=0.0112, audio_tagging_loss=0.01178, over 15318.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09241, pruned_loss=0.01442, audio_tagging_loss=0.008959, over 3044566.80 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 8.0
2023-11-23 03:43:05,980 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333000
2023-11-23 03:43:06,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2219946.6666666665, ans=0.0
2023-11-23 03:43:10,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2220013.3333333335, ans=0.0
2023-11-23 03:43:14,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2220013.3333333335, ans=0.0
2023-11-23 03:43:23,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2220080.0, ans=0.2
2023-11-23 03:43:26,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2220080.0, ans=0.09899494936611666
2023-11-23 03:43:54,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2220213.3333333335, ans=0.0
2023-11-23 03:44:03,099 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8400, loss[loss=0.05871, simple_loss=0.07861, pruned_loss=0.008341, audio_tagging_loss=0.01106, over 13725.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09277, pruned_loss=0.01447, audio_tagging_loss=0.009066, over 3046030.04 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:44:03,800 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.61 vs. limit=12.0
2023-11-23 03:44:07,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2220280.0, ans=0.125
2023-11-23 03:44:10,540 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333050
2023-11-23 03:44:36,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2220413.3333333335, ans=0.025
2023-11-23 03:44:55,764 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 7.977e+01 8.728e+01 9.553e+01 1.214e+02, threshold=1.746e+02, percent-clipped=0.0
2023-11-23 03:45:07,859 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8450, loss[loss=0.06983, simple_loss=0.09431, pruned_loss=0.01657, audio_tagging_loss=0.006099, over 15882.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09275, pruned_loss=0.01441, audio_tagging_loss=0.009157, over 3050389.04 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:45:15,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333100
2023-11-23 03:45:23,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2220680.0, ans=0.125
2023-11-23 03:45:31,748 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.66 vs. limit=15.0
2023-11-23 03:45:33,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2220746.6666666665, ans=0.125
2023-11-23 03:46:00,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2220880.0, ans=0.125
2023-11-23 03:46:06,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2220880.0, ans=0.0
2023-11-23 03:46:12,227 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8500, loss[loss=0.06827, simple_loss=0.09042, pruned_loss=0.01398, audio_tagging_loss=0.009084, over 15886.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09284, pruned_loss=0.01419, audio_tagging_loss=0.009158, over 3048849.20 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:46:19,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333150
2023-11-23 03:46:42,743 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.22 vs. limit=6.0
2023-11-23 03:46:46,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=2221080.0, ans=0.2
2023-11-23 03:47:03,975 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.153e+01 8.814e+01 9.449e+01 1.976e+02, threshold=1.763e+02, percent-clipped=1.0
2023-11-23 03:47:15,500 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8550, loss[loss=0.08388, simple_loss=0.109, pruned_loss=0.02111, audio_tagging_loss=0.008262, over 14346.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09299, pruned_loss=0.01415, audio_tagging_loss=0.009249, over 3051269.63 frames. ], batch size: 52, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:47:19,522 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.77 vs. limit=22.5
2023-11-23 03:47:23,680 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333200
2023-11-23 03:47:37,749 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.34 vs. limit=10.0
2023-11-23 03:47:38,838 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.53 vs. limit=15.0
2023-11-23 03:47:59,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2221480.0, ans=0.04949747468305833
2023-11-23 03:48:09,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2221546.6666666665, ans=0.125
2023-11-23 03:48:18,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2221546.6666666665, ans=0.125
2023-11-23 03:48:18,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2221546.6666666665, ans=0.125
2023-11-23 03:48:20,348 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8600, loss[loss=0.07722, simple_loss=0.1023, pruned_loss=0.01713, audio_tagging_loss=0.008932, over 13611.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09293, pruned_loss=0.01421, audio_tagging_loss=0.009193, over 3050677.36 frames. ], batch size: 52, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:48:27,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333250
2023-11-23 03:48:35,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2221680.0, ans=0.125
2023-11-23 03:48:44,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2221746.6666666665, ans=0.0
2023-11-23 03:48:57,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2221813.3333333335, ans=0.125
2023-11-23 03:49:00,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2221813.3333333335, ans=0.125
2023-11-23 03:49:01,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2221813.3333333335, ans=0.125
2023-11-23 03:49:12,188 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.558e+01 9.096e+01 9.715e+01 2.705e+02, threshold=1.819e+02, percent-clipped=1.0
2023-11-23 03:49:17,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2221880.0, ans=0.0
2023-11-23 03:49:19,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2221880.0, ans=0.04949747468305833
2023-11-23 03:49:23,136 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8650, loss[loss=0.08902, simple_loss=0.1184, pruned_loss=0.02345, audio_tagging_loss=0.006373, over 15734.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09345, pruned_loss=0.01411, audio_tagging_loss=0.009213, over 3046682.66 frames. ], batch size: 61, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:49:30,634 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333300
2023-11-23 03:49:44,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2222013.3333333335, ans=0.0
2023-11-23 03:50:08,563 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.58 vs. limit=15.0
2023-11-23 03:50:11,930 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.23 vs. limit=6.0
2023-11-23 03:50:17,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2222213.3333333335, ans=0.125
2023-11-23 03:50:19,283 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.30 vs. limit=10.0
2023-11-23 03:50:25,110 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.25 vs. limit=15.0
2023-11-23 03:50:25,867 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8700, loss[loss=0.08417, simple_loss=0.1033, pruned_loss=0.02582, audio_tagging_loss=0.006721, over 14061.00 frames. ], tot_loss[loss=0.07037, simple_loss=0.09383, pruned_loss=0.01427, audio_tagging_loss=0.009187, over 3044488.33 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:50:34,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333350
2023-11-23 03:50:38,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2222346.6666666665, ans=0.0
2023-11-23 03:51:06,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2222480.0, ans=0.0
2023-11-23 03:51:17,214 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.257e+01 8.537e+01 9.054e+01 9.922e+01 1.181e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 03:51:30,119 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8750, loss[loss=0.07162, simple_loss=0.09063, pruned_loss=0.01695, audio_tagging_loss=0.009354, over 14583.00 frames. ], tot_loss[loss=0.07122, simple_loss=0.09502, pruned_loss=0.01451, audio_tagging_loss=0.009202, over 3047235.79 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:51:32,079 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.00 vs. limit=12.0
2023-11-23 03:51:37,729 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333400
2023-11-23 03:51:45,979 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.05 vs. limit=6.0
2023-11-23 03:51:47,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2222680.0, ans=0.0
2023-11-23 03:51:47,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2222680.0, ans=0.125
2023-11-23 03:51:55,747 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.08 vs. limit=15.0
2023-11-23 03:52:08,717 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.14 vs. limit=15.0
2023-11-23 03:52:09,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2222813.3333333335, ans=0.0
2023-11-23 03:52:14,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2222813.3333333335, ans=0.0
2023-11-23 03:52:19,296 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:52:29,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2222880.0, ans=15.0
2023-11-23 03:52:33,778 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8800, loss[loss=0.05793, simple_loss=0.06735, pruned_loss=0.01197, audio_tagging_loss=0.01229, over 15466.00 frames. ], tot_loss[loss=0.07114, simple_loss=0.09454, pruned_loss=0.01455, audio_tagging_loss=0.009312, over 3046390.12 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:52:35,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2222946.6666666665, ans=0.0
2023-11-23 03:52:41,210 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333450
2023-11-23 03:52:45,462 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.81 vs. limit=6.0
2023-11-23 03:52:46,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2223013.3333333335, ans=0.125
2023-11-23 03:52:56,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2223013.3333333335, ans=0.125
2023-11-23 03:53:02,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2223080.0, ans=0.07
2023-11-23 03:53:03,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2223080.0, ans=0.0
2023-11-23 03:53:26,099 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.173e+01 8.235e+01 8.890e+01 9.571e+01 2.057e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-23 03:53:28,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2223213.3333333335, ans=0.125
2023-11-23 03:53:37,223 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8850, loss[loss=0.05455, simple_loss=0.07557, pruned_loss=0.006375, audio_tagging_loss=0.01038, over 14265.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09403, pruned_loss=0.01454, audio_tagging_loss=0.00923, over 3054381.92 frames. ], batch size: 54, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:53:45,236 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333500
2023-11-23 03:53:48,770 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 03:54:08,250 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 03:54:11,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2223413.3333333335, ans=0.125
2023-11-23 03:54:26,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2223480.0, ans=0.1
2023-11-23 03:54:27,844 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.98 vs. limit=15.0
2023-11-23 03:54:35,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2223546.6666666665, ans=0.0
2023-11-23 03:54:36,674 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.02 vs. limit=12.0
2023-11-23 03:54:40,997 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8900, loss[loss=0.07309, simple_loss=0.09706, pruned_loss=0.01729, audio_tagging_loss=0.007265, over 15288.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09465, pruned_loss=0.01474, audio_tagging_loss=0.009193, over 3052611.62 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:54:45,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2223613.3333333335, ans=0.125
2023-11-23 03:54:50,401 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333550
2023-11-23 03:55:34,579 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.471e+01 9.014e+01 9.931e+01 1.268e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 03:55:46,448 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 8950, loss[loss=0.07727, simple_loss=0.1079, pruned_loss=0.01643, audio_tagging_loss=0.006902, over 16378.00 frames. ], tot_loss[loss=0.07118, simple_loss=0.09494, pruned_loss=0.01464, audio_tagging_loss=0.009063, over 3052897.12 frames. ], batch size: 61, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:55:53,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333600
2023-11-23 03:56:01,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2224013.3333333335, ans=0.0
2023-11-23 03:56:27,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2224146.6666666665, ans=0.125
2023-11-23 03:56:27,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2224146.6666666665, ans=0.0
2023-11-23 03:56:45,076 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.60 vs. limit=6.0
2023-11-23 03:56:50,590 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9000, loss[loss=0.06416, simple_loss=0.08329, pruned_loss=0.01316, audio_tagging_loss=0.009354, over 15201.00 frames. ], tot_loss[loss=0.07091, simple_loss=0.09448, pruned_loss=0.01462, audio_tagging_loss=0.009043, over 3055800.33 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 03:56:50,593 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 03:57:17,515 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9708, 3.2012, 2.9398, 3.1543, 3.3985, 2.8579, 3.3926, 2.6969],
       device='cuda:0')
2023-11-23 03:57:33,767 INFO [train_asr.py:1253] (0/4) Epoch 28, validation: loss=0.05919, simple_loss=0.05113, pruned_loss=0.004978, audio_tagging_loss=0.02865, over 4681554.00 frames. 
2023-11-23 03:57:33,768 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 03:57:41,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333650
2023-11-23 03:57:49,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2224346.6666666665, ans=0.0
2023-11-23 03:58:13,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2224480.0, ans=0.0
2023-11-23 03:58:15,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2224480.0, ans=0.2
2023-11-23 03:58:28,167 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.337e+01 8.839e+01 9.721e+01 1.226e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-23 03:58:37,988 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9050, loss[loss=0.06084, simple_loss=0.07905, pruned_loss=0.01174, audio_tagging_loss=0.009579, over 16109.00 frames. ], tot_loss[loss=0.07055, simple_loss=0.09425, pruned_loss=0.01444, audio_tagging_loss=0.008985, over 3056837.75 frames. ], batch size: 61, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:58:45,337 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333700
2023-11-23 03:59:08,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2224746.6666666665, ans=0.125
2023-11-23 03:59:14,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2224746.6666666665, ans=0.0
2023-11-23 03:59:18,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2224813.3333333335, ans=0.2
2023-11-23 03:59:21,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2224813.3333333335, ans=0.0
2023-11-23 03:59:22,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.87 vs. limit=10.0
2023-11-23 03:59:29,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2224880.0, ans=0.0
2023-11-23 03:59:39,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2224880.0, ans=0.1
2023-11-23 03:59:41,505 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9100, loss[loss=0.07469, simple_loss=0.1126, pruned_loss=0.01223, audio_tagging_loss=0.006175, over 15565.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09468, pruned_loss=0.01449, audio_tagging_loss=0.008992, over 3056065.52 frames. ], batch size: 58, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 03:59:49,521 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333750
2023-11-23 04:00:34,987 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.085e+01 8.276e+01 8.928e+01 9.613e+01 1.240e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 04:00:35,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2225213.3333333335, ans=0.1
2023-11-23 04:00:43,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2225213.3333333335, ans=0.125
2023-11-23 04:00:46,293 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9150, loss[loss=0.09073, simple_loss=0.1236, pruned_loss=0.02177, audio_tagging_loss=0.007143, over 15451.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09454, pruned_loss=0.01464, audio_tagging_loss=0.009031, over 3050780.00 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:00:49,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2225280.0, ans=0.0
2023-11-23 04:00:52,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2225280.0, ans=0.0
2023-11-23 04:00:54,092 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333800
2023-11-23 04:01:50,743 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9200, loss[loss=0.07092, simple_loss=0.1042, pruned_loss=0.01105, audio_tagging_loss=0.00775, over 16869.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09438, pruned_loss=0.01452, audio_tagging_loss=0.009039, over 3050145.92 frames. ], batch size: 62, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:01:57,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2225613.3333333335, ans=0.125
2023-11-23 04:01:58,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333850
2023-11-23 04:02:38,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2225813.3333333335, ans=0.0
2023-11-23 04:02:38,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2225813.3333333335, ans=0.0
2023-11-23 04:02:44,368 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.331e+01 8.795e+01 9.315e+01 1.180e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 04:02:49,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2225880.0, ans=0.125
2023-11-23 04:02:49,923 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.48 vs. limit=15.0
2023-11-23 04:02:53,995 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9250, loss[loss=0.06416, simple_loss=0.08468, pruned_loss=0.01356, audio_tagging_loss=0.008263, over 15293.00 frames. ], tot_loss[loss=0.07039, simple_loss=0.09396, pruned_loss=0.01446, audio_tagging_loss=0.008949, over 3054443.80 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:02:57,277 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.01 vs. limit=22.5
2023-11-23 04:02:57,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2225946.6666666665, ans=0.125
2023-11-23 04:02:59,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2225946.6666666665, ans=0.125
2023-11-23 04:03:01,475 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333900
2023-11-23 04:03:05,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2226013.3333333335, ans=0.125
2023-11-23 04:03:18,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2226013.3333333335, ans=0.025
2023-11-23 04:03:32,014 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:03:34,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2226146.6666666665, ans=0.125
2023-11-23 04:03:58,098 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9300, loss[loss=0.0787, simple_loss=0.1102, pruned_loss=0.01759, audio_tagging_loss=0.005993, over 14395.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09391, pruned_loss=0.01441, audio_tagging_loss=0.009013, over 3051363.01 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:04:06,705 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 333950
2023-11-23 04:04:34,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2226413.3333333335, ans=0.125
2023-11-23 04:04:40,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2226480.0, ans=0.0
2023-11-23 04:04:47,343 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.34 vs. limit=15.0
2023-11-23 04:04:52,644 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.259e+01 8.847e+01 9.648e+01 1.420e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 04:05:02,497 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9350, loss[loss=0.06337, simple_loss=0.08276, pruned_loss=0.01163, audio_tagging_loss=0.01035, over 15262.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09404, pruned_loss=0.01459, audio_tagging_loss=0.009068, over 3057155.21 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:05:08,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.29 vs. limit=22.5
2023-11-23 04:05:10,426 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334000
2023-11-23 04:05:16,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2226680.0, ans=0.0
2023-11-23 04:05:18,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2226680.0, ans=0.1
2023-11-23 04:05:25,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2226680.0, ans=0.05
2023-11-23 04:05:29,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2226746.6666666665, ans=0.0
2023-11-23 04:05:42,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2226813.3333333335, ans=0.0
2023-11-23 04:05:54,583 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.02 vs. limit=15.0
2023-11-23 04:06:06,798 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9400, loss[loss=0.07309, simple_loss=0.1021, pruned_loss=0.01324, audio_tagging_loss=0.008806, over 15778.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09354, pruned_loss=0.01444, audio_tagging_loss=0.00919, over 3048136.81 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:06:07,490 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.29 vs. limit=15.0
2023-11-23 04:06:14,279 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334050
2023-11-23 04:06:24,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2227013.3333333335, ans=0.1
2023-11-23 04:06:39,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2227080.0, ans=0.125
2023-11-23 04:06:39,754 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.84 vs. limit=15.0
2023-11-23 04:06:50,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2227146.6666666665, ans=0.1
2023-11-23 04:07:00,202 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.797e+01 8.264e+01 9.005e+01 9.544e+01 1.394e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 04:07:07,482 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:07:10,624 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9450, loss[loss=0.05019, simple_loss=0.06281, pruned_loss=0.008662, audio_tagging_loss=0.01012, over 14607.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09394, pruned_loss=0.01455, audio_tagging_loss=0.009151, over 3047191.33 frames. ], batch size: 59, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:07:18,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334100
2023-11-23 04:07:33,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2227346.6666666665, ans=10.0
2023-11-23 04:07:40,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2227413.3333333335, ans=0.1
2023-11-23 04:07:41,984 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:07:50,858 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.90 vs. limit=6.0
2023-11-23 04:07:55,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2227480.0, ans=0.125
2023-11-23 04:08:04,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2227546.6666666665, ans=0.125
2023-11-23 04:08:15,170 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9500, loss[loss=0.08238, simple_loss=0.1157, pruned_loss=0.01786, audio_tagging_loss=0.00668, over 15926.00 frames. ], tot_loss[loss=0.07079, simple_loss=0.09415, pruned_loss=0.01453, audio_tagging_loss=0.009186, over 3051113.75 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:08:15,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2227613.3333333335, ans=0.1
2023-11-23 04:08:21,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2227613.3333333335, ans=0.125
2023-11-23 04:08:22,749 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334150
2023-11-23 04:08:34,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.73 vs. limit=15.0
2023-11-23 04:08:41,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2227746.6666666665, ans=0.125
2023-11-23 04:09:03,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2227813.3333333335, ans=0.1
2023-11-23 04:09:04,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2227813.3333333335, ans=0.2
2023-11-23 04:09:08,994 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.198e+01 8.793e+01 9.435e+01 1.623e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 04:09:13,007 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:09:19,610 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9550, loss[loss=0.04583, simple_loss=0.05253, pruned_loss=0.007677, audio_tagging_loss=0.01189, over 15718.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.09292, pruned_loss=0.01426, audio_tagging_loss=0.0094, over 3061832.43 frames. ], batch size: 62, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:09:19,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2227946.6666666665, ans=0.125
2023-11-23 04:09:20,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2227946.6666666665, ans=0.125
2023-11-23 04:09:26,967 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334200
2023-11-23 04:09:53,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2228080.0, ans=0.125
2023-11-23 04:10:17,571 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.19 vs. limit=15.0
2023-11-23 04:10:18,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2228213.3333333335, ans=0.2
2023-11-23 04:10:23,896 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9600, loss[loss=0.04969, simple_loss=0.06206, pruned_loss=0.01005, audio_tagging_loss=0.008618, over 14880.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09305, pruned_loss=0.01429, audio_tagging_loss=0.009425, over 3062897.50 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:10:31,938 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334250
2023-11-23 04:10:44,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2228346.6666666665, ans=0.1
2023-11-23 04:10:52,497 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.93 vs. limit=15.0
2023-11-23 04:10:53,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2228413.3333333335, ans=0.125
2023-11-23 04:10:54,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2228413.3333333335, ans=0.1
2023-11-23 04:10:57,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.86 vs. limit=10.0
2023-11-23 04:11:00,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2228480.0, ans=0.125
2023-11-23 04:11:15,429 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:11:15,686 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.14 vs. limit=22.5
2023-11-23 04:11:17,374 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.563e+01 8.386e+01 9.005e+01 9.855e+01 1.264e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 04:11:22,009 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:11:28,322 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9650, loss[loss=0.06696, simple_loss=0.09091, pruned_loss=0.01412, audio_tagging_loss=0.007387, over 14516.00 frames. ], tot_loss[loss=0.07059, simple_loss=0.0935, pruned_loss=0.01446, audio_tagging_loss=0.009384, over 3051912.77 frames. ], batch size: 54, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:11:35,695 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334300
2023-11-23 04:11:36,413 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.65 vs. limit=15.0
2023-11-23 04:11:54,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2228746.6666666665, ans=0.0
2023-11-23 04:12:01,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2228746.6666666665, ans=0.0
2023-11-23 04:12:29,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2228880.0, ans=0.0
2023-11-23 04:12:31,999 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9700, loss[loss=0.04989, simple_loss=0.07079, pruned_loss=0.008577, audio_tagging_loss=0.005918, over 15396.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09285, pruned_loss=0.01412, audio_tagging_loss=0.009242, over 3048918.68 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:12:39,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334350
2023-11-23 04:13:10,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2229146.6666666665, ans=0.0
2023-11-23 04:13:13,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.37 vs. limit=10.0
2023-11-23 04:13:23,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2229213.3333333335, ans=0.125
2023-11-23 04:13:24,177 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.03 vs. limit=22.5
2023-11-23 04:13:27,065 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.979e+01 8.051e+01 8.815e+01 9.310e+01 1.256e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 04:13:32,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2229213.3333333335, ans=0.125
2023-11-23 04:13:36,210 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9750, loss[loss=0.07585, simple_loss=0.1004, pruned_loss=0.0178, audio_tagging_loss=0.007858, over 14385.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09245, pruned_loss=0.01407, audio_tagging_loss=0.009068, over 3045633.19 frames. ], batch size: 54, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:13:44,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334400
2023-11-23 04:13:44,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2229280.0, ans=0.125
2023-11-23 04:14:12,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2229413.3333333335, ans=0.0
2023-11-23 04:14:41,481 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9800, loss[loss=0.06512, simple_loss=0.09111, pruned_loss=0.01177, audio_tagging_loss=0.007803, over 14088.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09231, pruned_loss=0.01414, audio_tagging_loss=0.00905, over 3045835.46 frames. ], batch size: 53, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:14:42,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=12.0
2023-11-23 04:14:48,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334450
2023-11-23 04:14:49,357 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.70 vs. limit=15.0
2023-11-23 04:14:55,724 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.93 vs. limit=12.0
2023-11-23 04:15:04,543 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.03 vs. limit=15.0
2023-11-23 04:15:36,215 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.857e+01 8.531e+01 9.197e+01 9.757e+01 1.250e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 04:15:37,549 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:15:37,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2229880.0, ans=0.0
2023-11-23 04:15:38,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2229880.0, ans=0.125
2023-11-23 04:15:44,378 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.51 vs. limit=15.0
2023-11-23 04:15:45,020 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9850, loss[loss=0.07182, simple_loss=0.09654, pruned_loss=0.01643, audio_tagging_loss=0.007119, over 15101.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09308, pruned_loss=0.01432, audio_tagging_loss=0.008966, over 3048672.98 frames. ], batch size: 57, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:15:52,429 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334500
2023-11-23 04:16:00,372 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.81 vs. limit=15.0
2023-11-23 04:16:21,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2230080.0, ans=0.1
2023-11-23 04:16:23,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2230146.6666666665, ans=0.125
2023-11-23 04:16:34,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2230146.6666666665, ans=0.125
2023-11-23 04:16:48,577 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9900, loss[loss=0.06445, simple_loss=0.08227, pruned_loss=0.01467, audio_tagging_loss=0.008646, over 14253.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09307, pruned_loss=0.01432, audio_tagging_loss=0.008985, over 3046677.57 frames. ], batch size: 55, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:16:51,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2230280.0, ans=0.125
2023-11-23 04:16:57,354 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334550
2023-11-23 04:17:26,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2230480.0, ans=0.125
2023-11-23 04:17:35,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.29 vs. limit=10.0
2023-11-23 04:17:39,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2230546.6666666665, ans=0.2
2023-11-23 04:17:44,119 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.266e+01 8.858e+01 9.665e+01 1.141e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 04:17:48,534 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.13 vs. limit=22.5
2023-11-23 04:17:53,604 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 9950, loss[loss=0.06235, simple_loss=0.08337, pruned_loss=0.008498, audio_tagging_loss=0.01216, over 15167.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09253, pruned_loss=0.01415, audio_tagging_loss=0.009088, over 3051854.30 frames. ], batch size: 54, lr: 2.42e-03, grad_scale: 16.0
2023-11-23 04:18:00,934 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334600
2023-11-23 04:18:01,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2230613.3333333335, ans=0.0
2023-11-23 04:18:03,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2230613.3333333335, ans=0.125
2023-11-23 04:18:35,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2230813.3333333335, ans=0.1
2023-11-23 04:18:57,018 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10000, loss[loss=0.08063, simple_loss=0.1157, pruned_loss=0.016, audio_tagging_loss=0.006764, over 15589.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09234, pruned_loss=0.01408, audio_tagging_loss=0.009138, over 3046421.32 frames. ], batch size: 56, lr: 2.42e-03, grad_scale: 32.0
2023-11-23 04:19:04,411 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334650
2023-11-23 04:19:04,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2230946.6666666665, ans=0.07
2023-11-23 04:19:05,799 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:19:25,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2231080.0, ans=0.07
2023-11-23 04:19:25,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2231080.0, ans=0.125
2023-11-23 04:19:26,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2231080.0, ans=0.1
2023-11-23 04:19:32,852 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:19:50,322 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.38 vs. limit=10.0
2023-11-23 04:19:51,968 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.134e+01 8.342e+01 8.814e+01 9.358e+01 1.133e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 04:20:00,511 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10050, loss[loss=0.07958, simple_loss=0.1095, pruned_loss=0.01574, audio_tagging_loss=0.009085, over 13835.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09297, pruned_loss=0.01421, audio_tagging_loss=0.009102, over 3049362.28 frames. ], batch size: 52, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:20:08,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334700
2023-11-23 04:20:13,652 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.26 vs. limit=22.5
2023-11-23 04:20:23,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2231346.6666666665, ans=0.125
2023-11-23 04:20:52,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2231546.6666666665, ans=0.125
2023-11-23 04:21:06,023 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10100, loss[loss=0.06085, simple_loss=0.07353, pruned_loss=0.01307, audio_tagging_loss=0.01102, over 13890.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09331, pruned_loss=0.01432, audio_tagging_loss=0.009131, over 3047729.97 frames. ], batch size: 54, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:21:07,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2231613.3333333335, ans=0.0
2023-11-23 04:21:09,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2231613.3333333335, ans=0.125
2023-11-23 04:21:13,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334750
2023-11-23 04:21:25,655 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=13.67 vs. limit=15.0
2023-11-23 04:21:34,076 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.20 vs. limit=15.0
2023-11-23 04:21:39,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2231746.6666666665, ans=0.0
2023-11-23 04:21:43,885 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.90 vs. limit=15.0
2023-11-23 04:21:57,923 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:22:02,744 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.781e+01 8.395e+01 8.740e+01 9.651e+01 1.218e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 04:22:10,056 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10150, loss[loss=0.07775, simple_loss=0.1063, pruned_loss=0.0172, audio_tagging_loss=0.007388, over 14913.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09286, pruned_loss=0.01415, audio_tagging_loss=0.009301, over 3048089.15 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:22:12,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2231946.6666666665, ans=0.125
2023-11-23 04:22:17,501 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334800
2023-11-23 04:22:17,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2231946.6666666665, ans=0.125
2023-11-23 04:22:39,037 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:22:56,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2232146.6666666665, ans=0.1
2023-11-23 04:23:13,270 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10200, loss[loss=0.05711, simple_loss=0.0734, pruned_loss=0.01074, audio_tagging_loss=0.009667, over 15323.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09277, pruned_loss=0.01416, audio_tagging_loss=0.009259, over 3054797.21 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:23:15,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2232280.0, ans=0.1
2023-11-23 04:23:20,597 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334850
2023-11-23 04:23:26,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2232346.6666666665, ans=0.125
2023-11-23 04:23:35,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2232346.6666666665, ans=0.2
2023-11-23 04:23:37,117 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.53 vs. limit=15.0
2023-11-23 04:23:37,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0
2023-11-23 04:23:37,546 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:23:38,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.63 vs. limit=6.0
2023-11-23 04:24:09,464 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.987e+01 8.292e+01 8.929e+01 9.861e+01 1.255e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 04:24:17,282 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10250, loss[loss=0.08044, simple_loss=0.1094, pruned_loss=0.01799, audio_tagging_loss=0.007733, over 16042.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09404, pruned_loss=0.01449, audio_tagging_loss=0.009127, over 3060695.45 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:24:26,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334900
2023-11-23 04:24:32,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2232680.0, ans=0.0
2023-11-23 04:24:40,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2232680.0, ans=0.125
2023-11-23 04:24:47,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2232746.6666666665, ans=0.1
2023-11-23 04:24:49,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2232746.6666666665, ans=0.1
2023-11-23 04:25:21,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2232946.6666666665, ans=0.125
2023-11-23 04:25:22,432 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10300, loss[loss=0.07212, simple_loss=0.09987, pruned_loss=0.01503, audio_tagging_loss=0.007156, over 15737.00 frames. ], tot_loss[loss=0.07102, simple_loss=0.09439, pruned_loss=0.01466, audio_tagging_loss=0.009166, over 3064803.81 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:25:29,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 334950
2023-11-23 04:25:34,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2233013.3333333335, ans=0.125
2023-11-23 04:25:53,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2233080.0, ans=0.05
2023-11-23 04:25:56,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2233080.0, ans=0.0
2023-11-23 04:25:56,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2233080.0, ans=0.1
2023-11-23 04:26:03,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2233146.6666666665, ans=0.1
2023-11-23 04:26:17,082 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.94 vs. limit=5.0
2023-11-23 04:26:19,730 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.243e+01 9.019e+01 9.717e+01 1.166e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 04:26:25,988 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10350, loss[loss=0.06947, simple_loss=0.092, pruned_loss=0.01207, audio_tagging_loss=0.0114, over 15949.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09396, pruned_loss=0.01456, audio_tagging_loss=0.00929, over 3059246.24 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:26:33,508 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335000
2023-11-23 04:26:33,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2233280.0, ans=0.125
2023-11-23 04:26:33,977 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.99 vs. limit=15.0
2023-11-23 04:26:57,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2233413.3333333335, ans=0.125
2023-11-23 04:26:57,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2233413.3333333335, ans=0.0
2023-11-23 04:27:23,241 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.47 vs. limit=15.0
2023-11-23 04:27:30,470 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10400, loss[loss=0.09276, simple_loss=0.1223, pruned_loss=0.02284, audio_tagging_loss=0.008779, over 14689.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.0937, pruned_loss=0.01452, audio_tagging_loss=0.009378, over 3054189.85 frames. ], batch size: 53, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:27:31,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2233613.3333333335, ans=0.125
2023-11-23 04:27:35,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2233613.3333333335, ans=0.125
2023-11-23 04:27:39,154 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335050
2023-11-23 04:28:03,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2233746.6666666665, ans=0.04949747468305833
2023-11-23 04:28:06,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2233746.6666666665, ans=0.125
2023-11-23 04:28:30,420 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.149e+01 8.861e+01 9.460e+01 1.224e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 04:28:35,408 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10450, loss[loss=0.079, simple_loss=0.11, pruned_loss=0.01649, audio_tagging_loss=0.007501, over 16180.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09309, pruned_loss=0.0144, audio_tagging_loss=0.0093, over 3049561.54 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:28:43,402 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335100
2023-11-23 04:29:19,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2234146.6666666665, ans=0.125
2023-11-23 04:29:21,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2234146.6666666665, ans=0.04949747468305833
2023-11-23 04:29:39,633 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10500, loss[loss=0.06736, simple_loss=0.08909, pruned_loss=0.01452, audio_tagging_loss=0.008299, over 14619.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09337, pruned_loss=0.01448, audio_tagging_loss=0.009109, over 3044100.80 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:29:47,122 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335150
2023-11-23 04:29:47,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2234280.0, ans=0.07
2023-11-23 04:29:48,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2234280.0, ans=0.2
2023-11-23 04:29:54,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2234346.6666666665, ans=0.125
2023-11-23 04:29:57,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2234346.6666666665, ans=0.2
2023-11-23 04:30:03,837 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.88 vs. limit=15.0
2023-11-23 04:30:04,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2234413.3333333335, ans=0.2
2023-11-23 04:30:07,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2234413.3333333335, ans=0.125
2023-11-23 04:30:17,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2234480.0, ans=0.0
2023-11-23 04:30:27,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2234480.0, ans=0.0
2023-11-23 04:30:27,600 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:30:37,996 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.970e+01 8.079e+01 8.755e+01 9.423e+01 1.196e+02, threshold=1.751e+02, percent-clipped=0.0
2023-11-23 04:30:42,945 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10550, loss[loss=0.07333, simple_loss=0.09014, pruned_loss=0.01735, audio_tagging_loss=0.01091, over 13857.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09278, pruned_loss=0.01423, audio_tagging_loss=0.009043, over 3045222.96 frames. ], batch size: 53, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:30:51,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335200
2023-11-23 04:31:00,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2234680.0, ans=0.0
2023-11-23 04:31:32,253 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.09 vs. limit=15.0
2023-11-23 04:31:34,155 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0
2023-11-23 04:31:46,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2234946.6666666665, ans=0.2
2023-11-23 04:31:47,914 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10600, loss[loss=0.08741, simple_loss=0.1132, pruned_loss=0.02291, audio_tagging_loss=0.007875, over 15863.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09371, pruned_loss=0.01429, audio_tagging_loss=0.008931, over 3051302.66 frames. ], batch size: 61, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:31:55,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335250
2023-11-23 04:31:57,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2234946.6666666665, ans=0.025
2023-11-23 04:32:04,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2235013.3333333335, ans=0.1
2023-11-23 04:32:13,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2235080.0, ans=0.2
2023-11-23 04:32:18,528 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.28 vs. limit=22.5
2023-11-23 04:32:19,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2235080.0, ans=0.125
2023-11-23 04:32:27,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2235146.6666666665, ans=0.125
2023-11-23 04:32:27,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2235146.6666666665, ans=0.1
2023-11-23 04:32:37,389 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.16 vs. limit=15.0
2023-11-23 04:32:38,401 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:32:43,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2235213.3333333335, ans=0.0
2023-11-23 04:32:46,335 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.415e+01 8.292e+01 8.923e+01 9.577e+01 1.197e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 04:32:51,943 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10650, loss[loss=0.08042, simple_loss=0.1027, pruned_loss=0.01972, audio_tagging_loss=0.009329, over 15459.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09394, pruned_loss=0.01433, audio_tagging_loss=0.008924, over 3054279.34 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:32:53,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2235280.0, ans=0.07
2023-11-23 04:32:55,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2235280.0, ans=0.0
2023-11-23 04:32:55,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2235280.0, ans=0.1
2023-11-23 04:32:56,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.82 vs. limit=15.0
2023-11-23 04:32:57,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2235280.0, ans=0.09899494936611666
2023-11-23 04:32:57,597 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.92 vs. limit=15.0
2023-11-23 04:32:59,227 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335300
2023-11-23 04:33:12,259 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:33:28,541 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.84 vs. limit=10.0
2023-11-23 04:33:43,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2235546.6666666665, ans=0.035
2023-11-23 04:33:55,242 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10700, loss[loss=0.09815, simple_loss=0.1304, pruned_loss=0.02091, audio_tagging_loss=0.01203, over 14455.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09349, pruned_loss=0.01426, audio_tagging_loss=0.008852, over 3048034.48 frames. ], batch size: 54, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:34:03,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335350
2023-11-23 04:34:08,106 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.74 vs. limit=6.0
2023-11-23 04:34:15,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.26 vs. limit=6.0
2023-11-23 04:34:37,333 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.60 vs. limit=15.0
2023-11-23 04:34:54,570 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.130e+01 8.303e+01 8.893e+01 9.663e+01 2.175e+02, threshold=1.779e+02, percent-clipped=1.0
2023-11-23 04:34:56,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2235880.0, ans=0.07
2023-11-23 04:34:59,983 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10750, loss[loss=0.05977, simple_loss=0.07885, pruned_loss=0.01243, audio_tagging_loss=0.007918, over 14975.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09363, pruned_loss=0.01437, audio_tagging_loss=0.008868, over 3047922.17 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:35:06,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=2235946.6666666665, ans=0.1
2023-11-23 04:35:07,344 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335400
2023-11-23 04:35:13,459 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.27 vs. limit=22.5
2023-11-23 04:35:23,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2236013.3333333335, ans=0.125
2023-11-23 04:35:31,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2236080.0, ans=0.1
2023-11-23 04:35:42,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2236146.6666666665, ans=0.125
2023-11-23 04:35:49,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2236146.6666666665, ans=0.0
2023-11-23 04:35:59,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2236213.3333333335, ans=0.0
2023-11-23 04:36:01,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2236213.3333333335, ans=0.0
2023-11-23 04:36:03,432 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10800, loss[loss=0.05578, simple_loss=0.08213, pruned_loss=0.006697, audio_tagging_loss=0.00802, over 15164.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.0941, pruned_loss=0.01435, audio_tagging_loss=0.00884, over 3048870.52 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:36:03,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2236280.0, ans=0.0
2023-11-23 04:36:06,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2236280.0, ans=0.125
2023-11-23 04:36:10,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2236280.0, ans=0.0
2023-11-23 04:36:11,624 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335450
2023-11-23 04:36:22,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2236346.6666666665, ans=0.125
2023-11-23 04:36:22,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2236346.6666666665, ans=0.0
2023-11-23 04:36:42,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2236480.0, ans=0.125
2023-11-23 04:36:48,914 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.53 vs. limit=10.0
2023-11-23 04:36:56,392 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.69 vs. limit=6.0
2023-11-23 04:37:02,352 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.072e+01 8.725e+01 9.352e+01 1.267e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-23 04:37:07,366 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10850, loss[loss=0.0599, simple_loss=0.06158, pruned_loss=0.0158, audio_tagging_loss=0.01331, over 13833.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09374, pruned_loss=0.01431, audio_tagging_loss=0.009001, over 3053716.57 frames. ], batch size: 54, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:37:15,406 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335500
2023-11-23 04:37:23,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2236680.0, ans=0.125
2023-11-23 04:37:32,741 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.55 vs. limit=15.0
2023-11-23 04:37:33,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2236746.6666666665, ans=0.125
2023-11-23 04:37:39,676 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:37:57,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2236880.0, ans=0.0
2023-11-23 04:38:07,671 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:38:10,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2236946.6666666665, ans=0.125
2023-11-23 04:38:11,718 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10900, loss[loss=0.04765, simple_loss=0.062, pruned_loss=0.006936, audio_tagging_loss=0.009715, over 16409.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09333, pruned_loss=0.01428, audio_tagging_loss=0.009103, over 3049709.00 frames. ], batch size: 63, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:38:19,157 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335550
2023-11-23 04:39:10,067 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.435e+01 8.359e+01 8.884e+01 9.760e+01 1.616e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 04:39:14,987 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 10950, loss[loss=0.06457, simple_loss=0.07972, pruned_loss=0.01632, audio_tagging_loss=0.00839, over 14397.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09246, pruned_loss=0.01432, audio_tagging_loss=0.009168, over 3046444.42 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:39:22,538 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335600
2023-11-23 04:39:25,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2237280.0, ans=0.1
2023-11-23 04:39:46,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2237413.3333333335, ans=0.125
2023-11-23 04:40:07,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2237546.6666666665, ans=0.125
2023-11-23 04:40:19,243 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11000, loss[loss=0.08156, simple_loss=0.111, pruned_loss=0.01797, audio_tagging_loss=0.008068, over 15528.00 frames. ], tot_loss[loss=0.07051, simple_loss=0.09351, pruned_loss=0.01461, audio_tagging_loss=0.009139, over 3048952.13 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:40:26,592 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335650
2023-11-23 04:40:29,652 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:40:41,282 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.83 vs. limit=12.0
2023-11-23 04:40:52,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2237746.6666666665, ans=0.125
2023-11-23 04:40:59,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2237813.3333333335, ans=0.125
2023-11-23 04:41:19,574 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.086e+01 8.268e+01 8.870e+01 9.647e+01 1.227e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 04:41:23,744 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11050, loss[loss=0.06592, simple_loss=0.0898, pruned_loss=0.01237, audio_tagging_loss=0.008645, over 15029.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.09363, pruned_loss=0.01459, audio_tagging_loss=0.009172, over 3052191.92 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:41:31,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335700
2023-11-23 04:41:40,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2238013.3333333335, ans=0.2
2023-11-23 04:41:57,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2238080.0, ans=0.95
2023-11-23 04:42:08,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2238146.6666666665, ans=0.1
2023-11-23 04:42:13,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2238146.6666666665, ans=0.0
2023-11-23 04:42:27,705 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11100, loss[loss=0.07176, simple_loss=0.09255, pruned_loss=0.01688, audio_tagging_loss=0.008603, over 14856.00 frames. ], tot_loss[loss=0.07087, simple_loss=0.09401, pruned_loss=0.01453, audio_tagging_loss=0.009328, over 3055218.60 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:42:35,173 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335750
2023-11-23 04:42:46,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2238346.6666666665, ans=0.125
2023-11-23 04:42:58,623 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.42 vs. limit=22.5
2023-11-23 04:43:00,218 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.50 vs. limit=15.0
2023-11-23 04:43:16,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2238480.0, ans=0.1
2023-11-23 04:43:22,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2238546.6666666665, ans=0.1
2023-11-23 04:43:27,385 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.249e+01 8.341e+01 8.890e+01 9.757e+01 1.222e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-23 04:43:31,103 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11150, loss[loss=0.09517, simple_loss=0.1239, pruned_loss=0.02318, audio_tagging_loss=0.01004, over 15557.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.0933, pruned_loss=0.01425, audio_tagging_loss=0.009451, over 3048159.93 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 04:43:39,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335800
2023-11-23 04:43:40,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2238613.3333333335, ans=0.0
2023-11-23 04:43:42,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2238613.3333333335, ans=0.07
2023-11-23 04:44:07,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2238746.6666666665, ans=0.125
2023-11-23 04:44:18,429 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.46 vs. limit=12.0
2023-11-23 04:44:19,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2238813.3333333335, ans=0.0
2023-11-23 04:44:25,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2238880.0, ans=0.025
2023-11-23 04:44:35,963 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11200, loss[loss=0.07379, simple_loss=0.1019, pruned_loss=0.01431, audio_tagging_loss=0.008548, over 15702.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09354, pruned_loss=0.01434, audio_tagging_loss=0.009542, over 3047384.91 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:44:36,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2238946.6666666665, ans=0.07
2023-11-23 04:44:39,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2238946.6666666665, ans=0.125
2023-11-23 04:44:40,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2238946.6666666665, ans=0.125
2023-11-23 04:44:40,744 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.30 vs. limit=22.5
2023-11-23 04:44:44,532 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335850
2023-11-23 04:44:53,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2239013.3333333335, ans=0.2
2023-11-23 04:44:56,124 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.85 vs. limit=15.0
2023-11-23 04:45:15,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2239146.6666666665, ans=0.1
2023-11-23 04:45:36,867 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.985e+01 8.304e+01 9.043e+01 9.933e+01 1.318e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 04:45:39,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2239280.0, ans=0.125
2023-11-23 04:45:40,651 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11250, loss[loss=0.07443, simple_loss=0.1047, pruned_loss=0.01525, audio_tagging_loss=0.006819, over 15477.00 frames. ], tot_loss[loss=0.07048, simple_loss=0.09355, pruned_loss=0.01429, audio_tagging_loss=0.009411, over 3048097.33 frames. ], batch size: 58, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:45:48,214 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335900
2023-11-23 04:46:44,918 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11300, loss[loss=0.07098, simple_loss=0.09196, pruned_loss=0.01463, audio_tagging_loss=0.01037, over 15523.00 frames. ], tot_loss[loss=0.07093, simple_loss=0.09435, pruned_loss=0.01458, audio_tagging_loss=0.009178, over 3047224.61 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:46:52,366 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 335950
2023-11-23 04:47:00,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2239680.0, ans=0.125
2023-11-23 04:47:16,159 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:47:21,373 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.65 vs. limit=15.0
2023-11-23 04:47:27,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2239813.3333333335, ans=0.0
2023-11-23 04:47:31,132 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.27 vs. limit=10.0
2023-11-23 04:47:41,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2239880.0, ans=0.125
2023-11-23 04:47:43,898 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.080e+01 8.333e+01 8.937e+01 9.779e+01 1.201e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 04:47:47,578 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11350, loss[loss=0.07369, simple_loss=0.1042, pruned_loss=0.0121, audio_tagging_loss=0.009498, over 15334.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09398, pruned_loss=0.01459, audio_tagging_loss=0.009056, over 3044541.00 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:47:56,798 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336000
2023-11-23 04:47:58,909 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-336000.pt
2023-11-23 04:48:02,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2239946.6666666665, ans=0.1
2023-11-23 04:48:15,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2240013.3333333335, ans=0.1
2023-11-23 04:48:20,218 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.07 vs. limit=6.0
2023-11-23 04:48:56,660 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11400, loss[loss=0.06123, simple_loss=0.08493, pruned_loss=0.01177, audio_tagging_loss=0.007001, over 14987.00 frames. ], tot_loss[loss=0.071, simple_loss=0.09471, pruned_loss=0.01466, audio_tagging_loss=0.008983, over 3046221.13 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:49:03,996 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336050
2023-11-23 04:49:04,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2240280.0, ans=0.125
2023-11-23 04:49:05,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2240280.0, ans=0.125
2023-11-23 04:49:09,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2240346.6666666665, ans=0.125
2023-11-23 04:49:12,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2240346.6666666665, ans=0.125
2023-11-23 04:49:24,867 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:49:40,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2240480.0, ans=10.0
2023-11-23 04:49:45,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2240480.0, ans=0.035
2023-11-23 04:49:56,449 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.358e+01 8.216e+01 8.830e+01 9.675e+01 1.432e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 04:49:58,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2240546.6666666665, ans=0.09899494936611666
2023-11-23 04:50:00,155 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11450, loss[loss=0.07549, simple_loss=0.108, pruned_loss=0.01507, audio_tagging_loss=0.006436, over 15732.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09303, pruned_loss=0.01448, audio_tagging_loss=0.009004, over 3044613.54 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:50:07,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336100
2023-11-23 04:50:21,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2240680.0, ans=0.125
2023-11-23 04:50:28,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2240746.6666666665, ans=0.125
2023-11-23 04:50:30,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2240746.6666666665, ans=0.0
2023-11-23 04:50:55,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2240880.0, ans=0.0
2023-11-23 04:51:02,804 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11500, loss[loss=0.08226, simple_loss=0.1098, pruned_loss=0.01736, audio_tagging_loss=0.01003, over 15527.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09333, pruned_loss=0.01456, audio_tagging_loss=0.008996, over 3049237.52 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:51:09,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2240946.6666666665, ans=0.125
2023-11-23 04:51:11,518 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336150
2023-11-23 04:51:22,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2241013.3333333335, ans=0.1
2023-11-23 04:51:43,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2241146.6666666665, ans=0.0
2023-11-23 04:51:48,043 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 04:51:56,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2241213.3333333335, ans=0.025
2023-11-23 04:52:04,942 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.510e+01 8.416e+01 8.974e+01 9.531e+01 1.779e+02, threshold=1.795e+02, percent-clipped=1.0
2023-11-23 04:52:08,767 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11550, loss[loss=0.06692, simple_loss=0.08376, pruned_loss=0.01691, audio_tagging_loss=0.00813, over 14451.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.0929, pruned_loss=0.01437, audio_tagging_loss=0.009028, over 3051007.04 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:52:12,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2241280.0, ans=0.0
2023-11-23 04:52:16,201 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336200
2023-11-23 04:52:24,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2241346.6666666665, ans=10.0
2023-11-23 04:52:35,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2241413.3333333335, ans=0.0
2023-11-23 04:52:46,506 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 04:52:57,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.73 vs. limit=6.0
2023-11-23 04:53:00,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2241546.6666666665, ans=0.125
2023-11-23 04:53:12,026 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11600, loss[loss=0.07087, simple_loss=0.09607, pruned_loss=0.01343, audio_tagging_loss=0.009396, over 14142.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09398, pruned_loss=0.01442, audio_tagging_loss=0.008945, over 3057745.35 frames. ], batch size: 53, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:53:12,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2241613.3333333335, ans=15.0
2023-11-23 04:53:19,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336250
2023-11-23 04:53:27,291 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.51 vs. limit=15.0
2023-11-23 04:53:30,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2241680.0, ans=0.05
2023-11-23 04:53:40,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2241746.6666666665, ans=0.125
2023-11-23 04:53:51,935 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.22 vs. limit=22.5
2023-11-23 04:53:56,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2241813.3333333335, ans=0.0
2023-11-23 04:54:01,590 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.74 vs. limit=12.0
2023-11-23 04:54:03,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2241880.0, ans=0.125
2023-11-23 04:54:11,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2241880.0, ans=0.025
2023-11-23 04:54:12,068 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.276e+01 8.228e+01 9.009e+01 9.457e+01 1.103e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 04:54:15,869 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11650, loss[loss=0.06686, simple_loss=0.08604, pruned_loss=0.01462, audio_tagging_loss=0.009219, over 15264.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.0936, pruned_loss=0.01431, audio_tagging_loss=0.009012, over 3048623.98 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:54:20,355 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.61 vs. limit=22.5
2023-11-23 04:54:23,845 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336300
2023-11-23 04:54:25,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2241946.6666666665, ans=0.0
2023-11-23 04:54:38,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.10 vs. limit=6.0
2023-11-23 04:54:48,324 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.83 vs. limit=12.0
2023-11-23 04:54:50,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2242080.0, ans=0.0
2023-11-23 04:54:53,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2242080.0, ans=0.125
2023-11-23 04:55:08,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2242213.3333333335, ans=0.2
2023-11-23 04:55:21,584 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11700, loss[loss=0.07794, simple_loss=0.104, pruned_loss=0.01855, audio_tagging_loss=0.007377, over 14919.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09364, pruned_loss=0.01444, audio_tagging_loss=0.009082, over 3048142.84 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:55:29,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336350
2023-11-23 04:55:30,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2242280.0, ans=0.125
2023-11-23 04:55:36,251 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.05 vs. limit=15.0
2023-11-23 04:55:46,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2242413.3333333335, ans=0.125
2023-11-23 04:55:50,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2242413.3333333335, ans=0.1
2023-11-23 04:55:51,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2242413.3333333335, ans=0.0
2023-11-23 04:55:58,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2242480.0, ans=0.125
2023-11-23 04:56:07,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2242480.0, ans=0.04949747468305833
2023-11-23 04:56:21,992 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.889e+01 8.295e+01 8.847e+01 9.516e+01 1.261e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 04:56:25,699 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11750, loss[loss=0.05076, simple_loss=0.0587, pruned_loss=0.0113, audio_tagging_loss=0.01011, over 15595.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.0919, pruned_loss=0.01421, audio_tagging_loss=0.009257, over 3042717.82 frames. ], batch size: 59, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:56:26,308 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.92 vs. limit=15.0
2023-11-23 04:56:32,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2242613.3333333335, ans=0.0
2023-11-23 04:56:33,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336400
2023-11-23 04:56:39,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2242680.0, ans=15.0
2023-11-23 04:56:50,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2242746.6666666665, ans=0.0
2023-11-23 04:56:52,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2242746.6666666665, ans=0.0
2023-11-23 04:57:16,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2242880.0, ans=0.125
2023-11-23 04:57:21,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2242880.0, ans=0.2
2023-11-23 04:57:29,440 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11800, loss[loss=0.09, simple_loss=0.1193, pruned_loss=0.02214, audio_tagging_loss=0.008207, over 15376.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09149, pruned_loss=0.01412, audio_tagging_loss=0.009329, over 3040133.71 frames. ], batch size: 57, lr: 2.41e-03, grad_scale: 32.0
2023-11-23 04:57:36,953 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336450
2023-11-23 04:57:44,802 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.75 vs. limit=12.0
2023-11-23 04:57:58,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2243080.0, ans=0.0
2023-11-23 04:58:12,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2243146.6666666665, ans=0.0
2023-11-23 04:58:31,393 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.646e+01 8.315e+01 8.837e+01 9.445e+01 1.101e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 04:58:33,803 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11850, loss[loss=0.09486, simple_loss=0.13, pruned_loss=0.02209, audio_tagging_loss=0.007777, over 15391.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09113, pruned_loss=0.01402, audio_tagging_loss=0.009485, over 3032805.96 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:58:41,629 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336500
2023-11-23 04:58:47,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2243346.6666666665, ans=0.2
2023-11-23 04:58:57,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2243346.6666666665, ans=0.025
2023-11-23 04:59:14,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2243480.0, ans=0.0
2023-11-23 04:59:28,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2243546.6666666665, ans=0.125
2023-11-23 04:59:38,819 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11900, loss[loss=0.07872, simple_loss=0.1137, pruned_loss=0.01612, audio_tagging_loss=0.005761, over 14947.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09151, pruned_loss=0.01399, audio_tagging_loss=0.009461, over 3036936.90 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 04:59:46,236 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336550
2023-11-23 04:59:51,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2243680.0, ans=0.125
2023-11-23 04:59:56,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2243680.0, ans=0.04949747468305833
2023-11-23 05:00:10,990 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.19 vs. limit=22.5
2023-11-23 05:00:16,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2243813.3333333335, ans=0.0
2023-11-23 05:00:24,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2243813.3333333335, ans=0.09899494936611666
2023-11-23 05:00:25,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2243813.3333333335, ans=0.1
2023-11-23 05:00:40,662 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.380e+01 8.877e+01 9.570e+01 1.262e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 05:00:41,887 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 11950, loss[loss=0.07074, simple_loss=0.0994, pruned_loss=0.0113, audio_tagging_loss=0.009739, over 15205.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09096, pruned_loss=0.01391, audio_tagging_loss=0.00956, over 3039782.69 frames. ], batch size: 56, lr: 2.41e-03, grad_scale: 8.0
2023-11-23 05:00:49,039 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336600
2023-11-23 05:00:59,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2244013.3333333335, ans=0.125
2023-11-23 05:00:59,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2244013.3333333335, ans=0.2
2023-11-23 05:01:35,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2244213.3333333335, ans=0.0
2023-11-23 05:01:43,501 INFO [train_asr.py:1221] (0/4) Epoch 28, batch 12000, loss[loss=0.0749, simple_loss=0.1026, pruned_loss=0.01105, audio_tagging_loss=0.01253, over 14834.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09212, pruned_loss=0.01411, audio_tagging_loss=0.00956, over 3048744.09 frames. ], batch size: 55, lr: 2.41e-03, grad_scale: 16.0
2023-11-23 05:01:43,505 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 05:02:27,090 INFO [train_asr.py:1253] (0/4) Epoch 28, validation: loss=0.05897, simple_loss=0.05124, pruned_loss=0.005128, audio_tagging_loss=0.02822, over 4681554.00 frames. 
2023-11-23 05:02:27,091 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 05:02:27,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2244280.0, ans=0.1
2023-11-23 05:02:29,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2244280.0, ans=0.04949747468305833
2023-11-23 05:02:32,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2244280.0, ans=0.1
2023-11-23 05:02:34,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336650
2023-11-23 05:02:56,254 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-28.pt
2023-11-23 05:03:30,842 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 0, loss[loss=0.07629, simple_loss=0.07822, pruned_loss=0.01296, audio_tagging_loss=0.02421, over 15011.00 frames. ], tot_loss[loss=0.07629, simple_loss=0.07822, pruned_loss=0.01296, audio_tagging_loss=0.02421, over 15011.00 frames. ], batch size: 58, lr: 2.37e-03, grad_scale: 32.0
2023-11-23 05:03:30,846 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 05:04:08,423 INFO [train_asr.py:1253] (0/4) Epoch 29, validation: loss=0.05816, simple_loss=0.05122, pruned_loss=0.005095, audio_tagging_loss=0.02745, over 4681554.00 frames. 
2023-11-23 05:04:08,424 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 05:04:38,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.99 vs. limit=10.0
2023-11-23 05:04:40,857 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.622e+01 8.521e+01 9.209e+01 1.026e+02 2.736e+02, threshold=1.842e+02, percent-clipped=1.0
2023-11-23 05:04:49,520 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336700
2023-11-23 05:05:01,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2244706.6666666665, ans=0.125
2023-11-23 05:05:12,063 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 50, loss[loss=0.09448, simple_loss=0.1188, pruned_loss=0.02252, audio_tagging_loss=0.01257, over 15131.00 frames. ], tot_loss[loss=0.07613, simple_loss=0.08941, pruned_loss=0.0135, audio_tagging_loss=0.01793, over 693209.61 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:05:17,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2244773.3333333335, ans=0.125
2023-11-23 05:05:25,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2244840.0, ans=0.125
2023-11-23 05:05:29,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=2244840.0, ans=0.5
2023-11-23 05:05:46,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2244906.6666666665, ans=0.0
2023-11-23 05:05:51,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2244973.3333333335, ans=0.0
2023-11-23 05:05:54,168 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336750
2023-11-23 05:05:56,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2244973.3333333335, ans=0.125
2023-11-23 05:06:18,189 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 100, loss[loss=0.07932, simple_loss=0.1027, pruned_loss=0.01569, audio_tagging_loss=0.01229, over 15381.00 frames. ], tot_loss[loss=0.0777, simple_loss=0.09294, pruned_loss=0.01425, audio_tagging_loss=0.01697, over 1212705.25 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:06:26,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2245106.6666666665, ans=0.1
2023-11-23 05:06:37,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2245173.3333333335, ans=0.0
2023-11-23 05:06:44,284 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.75 vs. limit=10.0
2023-11-23 05:06:44,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2245240.0, ans=0.0
2023-11-23 05:06:49,667 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.633e+01 8.985e+01 9.652e+01 1.024e+02 1.304e+02, threshold=1.930e+02, percent-clipped=0.0
2023-11-23 05:06:52,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2245240.0, ans=0.2
2023-11-23 05:06:59,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336800
2023-11-23 05:07:10,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2245373.3333333335, ans=0.125
2023-11-23 05:07:22,217 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 150, loss[loss=0.09354, simple_loss=0.1265, pruned_loss=0.02041, audio_tagging_loss=0.009906, over 15123.00 frames. ], tot_loss[loss=0.07618, simple_loss=0.09363, pruned_loss=0.01426, audio_tagging_loss=0.0151, over 1615405.98 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:07:30,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2245440.0, ans=0.1
2023-11-23 05:07:56,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2245573.3333333335, ans=0.0
2023-11-23 05:08:04,393 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336850
2023-11-23 05:08:27,348 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 200, loss[loss=0.0708, simple_loss=0.09709, pruned_loss=0.01381, audio_tagging_loss=0.008445, over 16368.00 frames. ], tot_loss[loss=0.07464, simple_loss=0.09417, pruned_loss=0.01422, audio_tagging_loss=0.01333, over 1938526.55 frames. ], batch size: 63, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:08:37,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2245773.3333333335, ans=0.0
2023-11-23 05:08:56,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2245906.6666666665, ans=0.1
2023-11-23 05:09:00,522 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.383e+01 9.139e+01 9.827e+01 1.313e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 05:09:01,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2245906.6666666665, ans=0.0
2023-11-23 05:09:08,653 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336900
2023-11-23 05:09:16,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2245973.3333333335, ans=0.2
2023-11-23 05:09:23,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2246040.0, ans=0.0
2023-11-23 05:09:32,735 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 250, loss[loss=0.06575, simple_loss=0.08626, pruned_loss=0.01125, audio_tagging_loss=0.01137, over 14792.00 frames. ], tot_loss[loss=0.07286, simple_loss=0.09341, pruned_loss=0.01399, audio_tagging_loss=0.01216, over 2185050.28 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:09:35,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2246106.6666666665, ans=0.1
2023-11-23 05:09:36,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2246106.6666666665, ans=0.1
2023-11-23 05:10:12,461 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 336950
2023-11-23 05:10:36,221 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 300, loss[loss=0.05426, simple_loss=0.06668, pruned_loss=0.01079, audio_tagging_loss=0.01013, over 14172.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09342, pruned_loss=0.01401, audio_tagging_loss=0.01129, over 2374362.89 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:10:36,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2246440.0, ans=0.2
2023-11-23 05:10:40,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2246440.0, ans=0.0
2023-11-23 05:11:02,043 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.81 vs. limit=12.0
2023-11-23 05:11:10,507 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.241e+01 8.912e+01 9.854e+01 1.344e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 05:11:17,985 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337000
2023-11-23 05:11:23,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2246640.0, ans=0.125
2023-11-23 05:11:25,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2246640.0, ans=0.0
2023-11-23 05:11:29,881 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.08 vs. limit=6.0
2023-11-23 05:11:41,002 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 350, loss[loss=0.07004, simple_loss=0.08958, pruned_loss=0.01646, audio_tagging_loss=0.008782, over 14826.00 frames. ], tot_loss[loss=0.07126, simple_loss=0.09339, pruned_loss=0.01402, audio_tagging_loss=0.01055, over 2525653.28 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:12:20,239 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.18 vs. limit=22.5
2023-11-23 05:12:22,246 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337050
2023-11-23 05:12:28,730 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:12:28,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2246973.3333333335, ans=0.125
2023-11-23 05:12:31,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2247040.0, ans=0.125
2023-11-23 05:12:39,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2247040.0, ans=0.125
2023-11-23 05:12:40,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2247040.0, ans=10.0
2023-11-23 05:12:42,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2247040.0, ans=0.2
2023-11-23 05:12:45,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2247106.6666666665, ans=0.1
2023-11-23 05:12:46,384 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 400, loss[loss=0.05512, simple_loss=0.06973, pruned_loss=0.007773, audio_tagging_loss=0.01248, over 14987.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09305, pruned_loss=0.01414, audio_tagging_loss=0.01025, over 2640783.59 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:12:51,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2247106.6666666665, ans=0.0
2023-11-23 05:13:10,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2247240.0, ans=0.0
2023-11-23 05:13:18,411 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.176e+01 8.720e+01 9.339e+01 1.453e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-23 05:13:26,479 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337100
2023-11-23 05:13:38,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2247373.3333333335, ans=0.0
2023-11-23 05:13:50,280 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 450, loss[loss=0.09517, simple_loss=0.1282, pruned_loss=0.0229, audio_tagging_loss=0.008184, over 15298.00 frames. ], tot_loss[loss=0.07058, simple_loss=0.0929, pruned_loss=0.01415, audio_tagging_loss=0.009986, over 2719684.26 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:13:52,193 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.80 vs. limit=6.0
2023-11-23 05:14:00,320 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.20 vs. limit=10.0
2023-11-23 05:14:05,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2247506.6666666665, ans=0.125
2023-11-23 05:14:31,407 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337150
2023-11-23 05:14:46,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2247706.6666666665, ans=0.125
2023-11-23 05:14:50,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2247706.6666666665, ans=0.125
2023-11-23 05:14:53,630 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 500, loss[loss=0.07704, simple_loss=0.09714, pruned_loss=0.01916, audio_tagging_loss=0.009317, over 14635.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.0929, pruned_loss=0.01418, audio_tagging_loss=0.009836, over 2786589.02 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:15:26,969 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.484e+01 9.126e+01 9.707e+01 1.686e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 05:15:31,224 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.39 vs. limit=15.0
2023-11-23 05:15:34,363 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337200
2023-11-23 05:15:48,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2248040.0, ans=0.125
2023-11-23 05:15:57,894 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 550, loss[loss=0.08327, simple_loss=0.113, pruned_loss=0.0187, audio_tagging_loss=0.00805, over 15035.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09189, pruned_loss=0.01391, audio_tagging_loss=0.009821, over 2841525.52 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:16:02,907 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.66 vs. limit=15.0
2023-11-23 05:16:03,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2248106.6666666665, ans=0.0
2023-11-23 05:16:10,470 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.00 vs. limit=10.0
2023-11-23 05:16:20,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2248173.3333333335, ans=0.0
2023-11-23 05:16:27,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2248240.0, ans=0.125
2023-11-23 05:16:38,173 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337250
2023-11-23 05:16:50,412 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.93 vs. limit=10.0
2023-11-23 05:17:02,000 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 600, loss[loss=0.0684, simple_loss=0.0967, pruned_loss=0.01331, audio_tagging_loss=0.006742, over 15120.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.0916, pruned_loss=0.01387, audio_tagging_loss=0.009697, over 2877444.88 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:17:06,377 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.31 vs. limit=15.0
2023-11-23 05:17:11,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2248440.0, ans=0.95
2023-11-23 05:17:34,613 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.545e+01 8.479e+01 8.808e+01 9.540e+01 1.178e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 05:17:42,786 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337300
2023-11-23 05:17:52,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2248706.6666666665, ans=0.0
2023-11-23 05:17:55,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2248706.6666666665, ans=0.125
2023-11-23 05:18:05,044 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 650, loss[loss=0.06299, simple_loss=0.08648, pruned_loss=0.01172, audio_tagging_loss=0.00803, over 15407.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09199, pruned_loss=0.01402, audio_tagging_loss=0.009586, over 2913967.59 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:18:12,013 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.27 vs. limit=15.0
2023-11-23 05:18:14,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2248773.3333333335, ans=0.1
2023-11-23 05:18:15,493 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.07 vs. limit=10.0
2023-11-23 05:18:27,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2248840.0, ans=0.125
2023-11-23 05:18:47,037 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337350
2023-11-23 05:18:50,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2248973.3333333335, ans=0.0
2023-11-23 05:18:50,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2248973.3333333335, ans=0.0
2023-11-23 05:18:57,332 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.90 vs. limit=15.0
2023-11-23 05:19:06,495 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.39 vs. limit=12.0
2023-11-23 05:19:09,391 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 700, loss[loss=0.07586, simple_loss=0.1036, pruned_loss=0.01633, audio_tagging_loss=0.007708, over 15386.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.0919, pruned_loss=0.01396, audio_tagging_loss=0.009521, over 2951701.13 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:19:19,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2249106.6666666665, ans=0.125
2023-11-23 05:19:21,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2249106.6666666665, ans=0.125
2023-11-23 05:19:37,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2249240.0, ans=0.0
2023-11-23 05:19:40,806 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.53 vs. limit=22.5
2023-11-23 05:19:44,725 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.145e+01 8.167e+01 8.621e+01 9.555e+01 1.511e+02, threshold=1.724e+02, percent-clipped=0.0
2023-11-23 05:19:48,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2249306.6666666665, ans=0.0
2023-11-23 05:19:51,201 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337400
2023-11-23 05:19:57,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2249306.6666666665, ans=0.125
2023-11-23 05:20:15,763 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 750, loss[loss=0.06782, simple_loss=0.09437, pruned_loss=0.01229, audio_tagging_loss=0.008344, over 16096.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09278, pruned_loss=0.01421, audio_tagging_loss=0.00946, over 2980344.52 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:20:37,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2249506.6666666665, ans=0.0
2023-11-23 05:20:46,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2249573.3333333335, ans=0.0
2023-11-23 05:20:57,687 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337450
2023-11-23 05:21:19,963 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 800, loss[loss=0.07459, simple_loss=0.09376, pruned_loss=0.01742, audio_tagging_loss=0.01029, over 14949.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.09299, pruned_loss=0.0144, audio_tagging_loss=0.00953, over 2992425.07 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:21:27,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2249773.3333333335, ans=0.125
2023-11-23 05:21:38,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.45 vs. limit=22.5
2023-11-23 05:21:55,398 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.684e+01 8.531e+01 8.991e+01 9.709e+01 1.279e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 05:22:01,783 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337500
2023-11-23 05:22:06,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2249973.3333333335, ans=0.0
2023-11-23 05:22:20,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.64 vs. limit=22.5
2023-11-23 05:22:21,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2250040.0, ans=0.125
2023-11-23 05:22:24,153 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 850, loss[loss=0.06696, simple_loss=0.08439, pruned_loss=0.01465, audio_tagging_loss=0.01011, over 16715.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09246, pruned_loss=0.01429, audio_tagging_loss=0.009615, over 3007902.14 frames. ], batch size: 62, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:22:38,167 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.69 vs. limit=15.0
2023-11-23 05:23:06,276 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337550
2023-11-23 05:23:22,875 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.70 vs. limit=10.0
2023-11-23 05:23:26,146 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:23:28,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2250373.3333333335, ans=0.0
2023-11-23 05:23:30,317 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 900, loss[loss=0.06108, simple_loss=0.07882, pruned_loss=0.01384, audio_tagging_loss=0.007833, over 15338.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09226, pruned_loss=0.0141, audio_tagging_loss=0.00967, over 3009752.84 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:23:42,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2250506.6666666665, ans=0.0
2023-11-23 05:23:53,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2250506.6666666665, ans=0.125
2023-11-23 05:24:03,823 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.195e+01 8.718e+01 9.591e+01 1.345e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-23 05:24:04,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2250573.3333333335, ans=0.125
2023-11-23 05:24:11,542 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337600
2023-11-23 05:24:33,055 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.12 vs. limit=22.5
2023-11-23 05:24:34,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2250773.3333333335, ans=0.0
2023-11-23 05:24:35,050 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 950, loss[loss=0.06299, simple_loss=0.09, pruned_loss=0.01172, audio_tagging_loss=0.006275, over 15168.00 frames. ], tot_loss[loss=0.07023, simple_loss=0.09314, pruned_loss=0.01412, audio_tagging_loss=0.009537, over 3017585.08 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:24:46,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2250840.0, ans=0.125
2023-11-23 05:24:57,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2250840.0, ans=0.125
2023-11-23 05:25:17,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337650
2023-11-23 05:25:22,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2250973.3333333335, ans=0.0
2023-11-23 05:25:22,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2250973.3333333335, ans=0.1
2023-11-23 05:25:39,754 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1000, loss[loss=0.0572, simple_loss=0.07168, pruned_loss=0.0109, audio_tagging_loss=0.01046, over 14331.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09176, pruned_loss=0.01394, audio_tagging_loss=0.009418, over 3020054.89 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:25:40,308 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.24 vs. limit=15.0
2023-11-23 05:25:43,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2251106.6666666665, ans=0.1
2023-11-23 05:25:50,369 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.26 vs. limit=15.0
2023-11-23 05:25:55,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2251173.3333333335, ans=0.1
2023-11-23 05:25:58,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2251173.3333333335, ans=0.0
2023-11-23 05:26:07,700 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:26:10,770 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.21 vs. limit=15.0
2023-11-23 05:26:15,021 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.610e+01 8.282e+01 9.010e+01 9.966e+01 1.225e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 05:26:20,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2251306.6666666665, ans=0.1
2023-11-23 05:26:21,312 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337700
2023-11-23 05:26:21,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2251306.6666666665, ans=0.95
2023-11-23 05:26:23,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2251306.6666666665, ans=0.1
2023-11-23 05:26:26,046 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.51 vs. limit=5.0
2023-11-23 05:26:26,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2251306.6666666665, ans=0.025
2023-11-23 05:26:40,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2251373.3333333335, ans=0.0
2023-11-23 05:26:44,944 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1050, loss[loss=0.08477, simple_loss=0.1227, pruned_loss=0.01642, audio_tagging_loss=0.007009, over 15943.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09191, pruned_loss=0.01407, audio_tagging_loss=0.009312, over 3022996.18 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:26:53,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2251440.0, ans=0.2
2023-11-23 05:26:59,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2251506.6666666665, ans=0.1
2023-11-23 05:27:04,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2251506.6666666665, ans=0.0
2023-11-23 05:27:09,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2251573.3333333335, ans=0.125
2023-11-23 05:27:26,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337750
2023-11-23 05:27:31,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2251640.0, ans=0.125
2023-11-23 05:27:40,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2251706.6666666665, ans=0.2
2023-11-23 05:27:41,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2251706.6666666665, ans=0.0
2023-11-23 05:27:49,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2251773.3333333335, ans=0.125
2023-11-23 05:27:50,133 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1100, loss[loss=0.06, simple_loss=0.07118, pruned_loss=0.01283, audio_tagging_loss=0.01157, over 14990.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09146, pruned_loss=0.01402, audio_tagging_loss=0.009328, over 3021416.48 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:27:52,685 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:27:52,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2251773.3333333335, ans=0.0
2023-11-23 05:27:58,387 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=7.20 vs. limit=10.0
2023-11-23 05:27:58,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2251773.3333333335, ans=0.2
2023-11-23 05:27:59,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2251773.3333333335, ans=0.125
2023-11-23 05:28:25,468 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.248e+01 8.961e+01 9.560e+01 1.246e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 05:28:30,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2251973.3333333335, ans=0.2
2023-11-23 05:28:31,867 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337800
2023-11-23 05:28:54,916 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1150, loss[loss=0.05781, simple_loss=0.07957, pruned_loss=0.01062, audio_tagging_loss=0.007403, over 14739.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09143, pruned_loss=0.0139, audio_tagging_loss=0.009219, over 3025463.57 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:29:01,257 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:29:18,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2252173.3333333335, ans=0.0
2023-11-23 05:29:36,669 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337850
2023-11-23 05:29:39,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2252306.6666666665, ans=0.125
2023-11-23 05:29:47,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2252373.3333333335, ans=0.0
2023-11-23 05:30:00,373 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1200, loss[loss=0.06637, simple_loss=0.09352, pruned_loss=0.01309, audio_tagging_loss=0.006522, over 15979.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09157, pruned_loss=0.01388, audio_tagging_loss=0.009145, over 3037389.06 frames. ], batch size: 63, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:30:12,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2252506.6666666665, ans=0.0
2023-11-23 05:30:18,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2252506.6666666665, ans=0.0
2023-11-23 05:30:22,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2252506.6666666665, ans=0.125
2023-11-23 05:30:35,270 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.804e+01 8.344e+01 9.032e+01 9.683e+01 1.496e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 05:30:40,268 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337900
2023-11-23 05:31:04,509 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1250, loss[loss=0.07383, simple_loss=0.1036, pruned_loss=0.01168, audio_tagging_loss=0.01034, over 14761.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09108, pruned_loss=0.0139, audio_tagging_loss=0.009191, over 3041777.51 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:31:11,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2252773.3333333335, ans=0.0
2023-11-23 05:31:41,956 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:31:45,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 337950
2023-11-23 05:31:58,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2253040.0, ans=0.125
2023-11-23 05:32:07,643 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1300, loss[loss=0.06002, simple_loss=0.08212, pruned_loss=0.009982, audio_tagging_loss=0.008973, over 15567.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09106, pruned_loss=0.01384, audio_tagging_loss=0.009027, over 3043957.99 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:32:10,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2253106.6666666665, ans=0.1
2023-11-23 05:32:33,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2253240.0, ans=0.125
2023-11-23 05:32:38,150 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.65 vs. limit=15.0
2023-11-23 05:32:39,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.29 vs. limit=12.0
2023-11-23 05:32:44,466 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.370e+01 8.150e+01 8.858e+01 9.270e+01 1.252e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 05:32:47,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2253306.6666666665, ans=0.0
2023-11-23 05:32:49,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338000
2023-11-23 05:32:51,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2253306.6666666665, ans=0.05
2023-11-23 05:33:13,806 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1350, loss[loss=0.04778, simple_loss=0.05633, pruned_loss=0.008194, audio_tagging_loss=0.01142, over 14070.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.0905, pruned_loss=0.01375, audio_tagging_loss=0.009173, over 3052601.31 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:33:18,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2253440.0, ans=0.1
2023-11-23 05:33:24,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2253506.6666666665, ans=0.1
2023-11-23 05:33:27,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2253506.6666666665, ans=0.125
2023-11-23 05:33:53,327 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338050
2023-11-23 05:33:59,468 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:34:17,346 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1400, loss[loss=0.05446, simple_loss=0.06283, pruned_loss=0.008945, audio_tagging_loss=0.01411, over 14781.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09014, pruned_loss=0.0138, audio_tagging_loss=0.009312, over 3054674.12 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:34:41,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2253906.6666666665, ans=0.125
2023-11-23 05:34:53,597 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.459e+01 9.010e+01 9.793e+01 1.707e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 05:34:58,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338100
2023-11-23 05:35:20,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2254106.6666666665, ans=0.125
2023-11-23 05:35:21,300 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1450, loss[loss=0.04905, simple_loss=0.06143, pruned_loss=0.007023, audio_tagging_loss=0.01131, over 14934.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09102, pruned_loss=0.01409, audio_tagging_loss=0.009307, over 3043130.55 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:35:54,743 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.37 vs. limit=15.0
2023-11-23 05:36:01,644 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338150
2023-11-23 05:36:14,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2254373.3333333335, ans=0.1
2023-11-23 05:36:18,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2254373.3333333335, ans=0.1
2023-11-23 05:36:24,826 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1500, loss[loss=0.068, simple_loss=0.08797, pruned_loss=0.01203, audio_tagging_loss=0.01198, over 14357.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09137, pruned_loss=0.01432, audio_tagging_loss=0.009349, over 3030965.09 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:36:26,863 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.40 vs. limit=15.0
2023-11-23 05:36:33,648 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.36 vs. limit=15.0
2023-11-23 05:36:36,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2254506.6666666665, ans=0.125
2023-11-23 05:36:43,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2254506.6666666665, ans=0.2
2023-11-23 05:36:44,924 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.92 vs. limit=15.0
2023-11-23 05:36:46,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2254506.6666666665, ans=0.1
2023-11-23 05:36:48,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2254506.6666666665, ans=0.125
2023-11-23 05:37:00,757 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.437e+01 9.139e+01 9.780e+01 1.863e+02, threshold=1.828e+02, percent-clipped=1.0
2023-11-23 05:37:05,861 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338200
2023-11-23 05:37:26,523 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-23 05:37:29,288 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1550, loss[loss=0.06443, simple_loss=0.09301, pruned_loss=0.01055, audio_tagging_loss=0.007375, over 14733.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09173, pruned_loss=0.01441, audio_tagging_loss=0.009449, over 3034202.78 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:37:32,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2254773.3333333335, ans=0.0
2023-11-23 05:37:36,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff2.min_abs, batch_count=2254773.3333333335, ans=0.1
2023-11-23 05:37:47,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2254840.0, ans=0.0
2023-11-23 05:37:48,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2254840.0, ans=0.2
2023-11-23 05:37:51,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2254840.0, ans=0.0
2023-11-23 05:37:56,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2254906.6666666665, ans=0.125
2023-11-23 05:37:58,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2254906.6666666665, ans=0.0
2023-11-23 05:38:10,785 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338250
2023-11-23 05:38:13,738 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.12 vs. limit=15.0
2023-11-23 05:38:16,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=14.15 vs. limit=15.0
2023-11-23 05:38:32,705 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1600, loss[loss=0.06113, simple_loss=0.0749, pruned_loss=0.01262, audio_tagging_loss=0.01107, over 14469.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.09281, pruned_loss=0.01456, audio_tagging_loss=0.009432, over 3034444.38 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:38:46,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2255173.3333333335, ans=0.025
2023-11-23 05:39:09,183 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.790e+01 8.320e+01 8.917e+01 9.601e+01 1.213e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 05:39:13,004 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338300
2023-11-23 05:39:19,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2255306.6666666665, ans=0.0
2023-11-23 05:39:35,863 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1650, loss[loss=0.05732, simple_loss=0.07297, pruned_loss=0.01025, audio_tagging_loss=0.01059, over 15169.00 frames. ], tot_loss[loss=0.06999, simple_loss=0.09217, pruned_loss=0.01446, audio_tagging_loss=0.009444, over 3039599.90 frames. ], batch size: 58, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:39:42,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2255440.0, ans=0.2
2023-11-23 05:39:45,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2255440.0, ans=0.125
2023-11-23 05:40:15,717 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338350
2023-11-23 05:40:25,944 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.22 vs. limit=10.0
2023-11-23 05:40:39,304 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1700, loss[loss=0.08072, simple_loss=0.1163, pruned_loss=0.01375, audio_tagging_loss=0.008833, over 15159.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09142, pruned_loss=0.01431, audio_tagging_loss=0.00961, over 3041465.94 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:40:46,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2255773.3333333335, ans=0.2
2023-11-23 05:40:56,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2255840.0, ans=0.125
2023-11-23 05:41:10,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2255906.6666666665, ans=0.125
2023-11-23 05:41:16,149 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.211e+01 9.004e+01 9.824e+01 1.268e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 05:41:19,984 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338400
2023-11-23 05:41:36,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2256040.0, ans=0.125
2023-11-23 05:41:42,304 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1750, loss[loss=0.06805, simple_loss=0.08707, pruned_loss=0.01472, audio_tagging_loss=0.009791, over 14357.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09061, pruned_loss=0.01409, audio_tagging_loss=0.009515, over 3040296.35 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:41:46,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2256106.6666666665, ans=0.04949747468305833
2023-11-23 05:41:55,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2256173.3333333335, ans=0.1
2023-11-23 05:41:56,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2256173.3333333335, ans=0.125
2023-11-23 05:41:57,088 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.51 vs. limit=15.0
2023-11-23 05:42:09,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2256240.0, ans=0.125
2023-11-23 05:42:22,891 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338450
2023-11-23 05:42:45,366 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1800, loss[loss=0.05242, simple_loss=0.07244, pruned_loss=0.008814, audio_tagging_loss=0.007388, over 15815.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09058, pruned_loss=0.0139, audio_tagging_loss=0.009324, over 3039406.92 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:42:45,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2256440.0, ans=0.125
2023-11-23 05:42:57,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2256506.6666666665, ans=0.2
2023-11-23 05:42:59,017 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.92 vs. limit=15.0
2023-11-23 05:43:12,681 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.07 vs. limit=15.0
2023-11-23 05:43:14,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2256573.3333333335, ans=0.1
2023-11-23 05:43:21,550 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.187e+01 9.056e+01 9.555e+01 1.230e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 05:43:25,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338500
2023-11-23 05:43:32,141 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.36 vs. limit=22.5
2023-11-23 05:43:48,543 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1850, loss[loss=0.06114, simple_loss=0.06936, pruned_loss=0.01518, audio_tagging_loss=0.01129, over 13969.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09124, pruned_loss=0.01407, audio_tagging_loss=0.009238, over 3041842.73 frames. ], batch size: 55, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:43:51,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2256773.3333333335, ans=0.125
2023-11-23 05:44:29,131 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338550
2023-11-23 05:44:51,031 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1900, loss[loss=0.06261, simple_loss=0.08785, pruned_loss=0.01021, audio_tagging_loss=0.008472, over 15301.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09154, pruned_loss=0.01395, audio_tagging_loss=0.009153, over 3037349.65 frames. ], batch size: 59, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:45:28,235 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.552e+01 8.593e+01 9.216e+01 1.001e+02 1.158e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-23 05:45:32,192 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338600
2023-11-23 05:45:54,442 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 1950, loss[loss=0.05868, simple_loss=0.07259, pruned_loss=0.01294, audio_tagging_loss=0.009447, over 14840.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09128, pruned_loss=0.01403, audio_tagging_loss=0.009166, over 3036341.43 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:45:54,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2257440.0, ans=0.5
2023-11-23 05:46:17,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2257506.6666666665, ans=0.125
2023-11-23 05:46:34,865 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338650
2023-11-23 05:46:35,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2257640.0, ans=0.0
2023-11-23 05:46:57,996 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2000, loss[loss=0.07834, simple_loss=0.1073, pruned_loss=0.01678, audio_tagging_loss=0.007902, over 14514.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09066, pruned_loss=0.01393, audio_tagging_loss=0.009203, over 3031187.25 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:47:03,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2257773.3333333335, ans=0.1
2023-11-23 05:47:08,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2257773.3333333335, ans=0.2
2023-11-23 05:47:14,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2257840.0, ans=0.1
2023-11-23 05:47:19,613 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.11 vs. limit=15.0
2023-11-23 05:47:24,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2257906.6666666665, ans=0.1
2023-11-23 05:47:27,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2257906.6666666665, ans=0.1
2023-11-23 05:47:33,484 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.084e+01 8.322e+01 9.117e+01 1.012e+02 1.277e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-23 05:47:37,943 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338700
2023-11-23 05:48:00,709 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2050, loss[loss=0.06878, simple_loss=0.09689, pruned_loss=0.0117, audio_tagging_loss=0.008632, over 16716.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09279, pruned_loss=0.01418, audio_tagging_loss=0.009073, over 3037040.55 frames. ], batch size: 61, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:48:07,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2258106.6666666665, ans=0.125
2023-11-23 05:48:08,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2258106.6666666665, ans=0.0
2023-11-23 05:48:13,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2258173.3333333335, ans=0.0
2023-11-23 05:48:13,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2258173.3333333335, ans=0.09899494936611666
2023-11-23 05:48:18,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2258173.3333333335, ans=0.125
2023-11-23 05:48:19,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2258173.3333333335, ans=0.1
2023-11-23 05:48:24,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2258240.0, ans=0.125
2023-11-23 05:48:24,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2258240.0, ans=0.125
2023-11-23 05:48:37,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2258306.6666666665, ans=0.0
2023-11-23 05:48:40,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=2258306.6666666665, ans=0.05
2023-11-23 05:48:41,304 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338750
2023-11-23 05:48:57,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2258373.3333333335, ans=0.125
2023-11-23 05:48:58,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2258373.3333333335, ans=0.125
2023-11-23 05:49:03,151 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2100, loss[loss=0.08302, simple_loss=0.1121, pruned_loss=0.01728, audio_tagging_loss=0.009701, over 15008.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09219, pruned_loss=0.01396, audio_tagging_loss=0.009013, over 3037135.54 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:49:09,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2258440.0, ans=0.0
2023-11-23 05:49:22,822 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.47 vs. limit=22.5
2023-11-23 05:49:28,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2258573.3333333335, ans=0.0
2023-11-23 05:49:28,841 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.03 vs. limit=15.0
2023-11-23 05:49:36,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2258573.3333333335, ans=0.125
2023-11-23 05:49:40,174 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.396e+01 9.168e+01 1.020e+02 1.328e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 05:49:43,933 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338800
2023-11-23 05:49:52,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2258706.6666666665, ans=0.2
2023-11-23 05:49:55,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2258706.6666666665, ans=0.125
2023-11-23 05:50:00,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2258706.6666666665, ans=0.125
2023-11-23 05:50:07,665 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2150, loss[loss=0.05524, simple_loss=0.07158, pruned_loss=0.01103, audio_tagging_loss=0.008417, over 14981.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09322, pruned_loss=0.01421, audio_tagging_loss=0.00898, over 3037775.58 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:50:19,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2258840.0, ans=0.125
2023-11-23 05:50:45,284 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:50:47,807 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338850
2023-11-23 05:51:02,501 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.21 vs. limit=15.0
2023-11-23 05:51:04,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2259040.0, ans=0.125
2023-11-23 05:51:11,696 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2200, loss[loss=0.06963, simple_loss=0.09091, pruned_loss=0.01466, audio_tagging_loss=0.00951, over 16468.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09201, pruned_loss=0.01401, audio_tagging_loss=0.0091, over 3043603.72 frames. ], batch size: 62, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:51:12,082 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:51:15,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2259106.6666666665, ans=0.0
2023-11-23 05:51:24,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2259173.3333333335, ans=0.125
2023-11-23 05:51:35,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2259240.0, ans=0.2
2023-11-23 05:51:46,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2259240.0, ans=0.125
2023-11-23 05:51:50,018 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.646e+01 8.527e+01 9.201e+01 9.842e+01 1.279e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 05:51:53,527 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338900
2023-11-23 05:52:13,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2259373.3333333335, ans=0.0
2023-11-23 05:52:14,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.18 vs. limit=10.0
2023-11-23 05:52:16,100 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2250, loss[loss=0.06381, simple_loss=0.07989, pruned_loss=0.01287, audio_tagging_loss=0.011, over 14983.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09172, pruned_loss=0.01389, audio_tagging_loss=0.00911, over 3041878.79 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:52:23,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2259440.0, ans=0.0
2023-11-23 05:52:55,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.64 vs. limit=22.5
2023-11-23 05:52:56,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.18 vs. limit=15.0
2023-11-23 05:52:58,331 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 338950
2023-11-23 05:52:58,806 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.29 vs. limit=15.0
2023-11-23 05:52:59,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2259640.0, ans=0.1
2023-11-23 05:53:04,664 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 05:53:06,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2259640.0, ans=0.2
2023-11-23 05:53:08,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2259706.6666666665, ans=0.0
2023-11-23 05:53:22,031 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2300, loss[loss=0.05108, simple_loss=0.06806, pruned_loss=0.0089, audio_tagging_loss=0.008144, over 16009.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09108, pruned_loss=0.01386, audio_tagging_loss=0.009196, over 3039193.91 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:53:27,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2259773.3333333335, ans=0.125
2023-11-23 05:53:55,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2259906.6666666665, ans=0.2
2023-11-23 05:54:00,273 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.833e+01 8.330e+01 8.807e+01 9.666e+01 1.881e+02, threshold=1.761e+02, percent-clipped=1.0
2023-11-23 05:54:02,938 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339000
2023-11-23 05:54:19,388 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 05:54:27,450 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2350, loss[loss=0.06181, simple_loss=0.08707, pruned_loss=0.009678, audio_tagging_loss=0.0086, over 15337.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09152, pruned_loss=0.01404, audio_tagging_loss=0.009238, over 3039541.52 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:54:31,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2260106.6666666665, ans=0.0
2023-11-23 05:54:37,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2260106.6666666665, ans=0.125
2023-11-23 05:54:45,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2260173.3333333335, ans=0.125
2023-11-23 05:55:08,371 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339050
2023-11-23 05:55:15,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2260306.6666666665, ans=0.0
2023-11-23 05:55:21,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2260373.3333333335, ans=0.0
2023-11-23 05:55:23,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2260373.3333333335, ans=0.125
2023-11-23 05:55:23,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2260373.3333333335, ans=0.125
2023-11-23 05:55:26,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2260373.3333333335, ans=0.0
2023-11-23 05:55:30,935 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2400, loss[loss=0.09046, simple_loss=0.1321, pruned_loss=0.01869, audio_tagging_loss=0.005722, over 15274.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09204, pruned_loss=0.01403, audio_tagging_loss=0.009314, over 3042993.56 frames. ], batch size: 54, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:56:09,723 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.287e+01 8.924e+01 9.833e+01 1.260e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 05:56:12,357 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339100
2023-11-23 05:56:23,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2260706.6666666665, ans=0.125
2023-11-23 05:56:23,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2260706.6666666665, ans=0.125
2023-11-23 05:56:34,936 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2450, loss[loss=0.06109, simple_loss=0.07993, pruned_loss=0.01204, audio_tagging_loss=0.009092, over 15629.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09142, pruned_loss=0.01381, audio_tagging_loss=0.009547, over 3043402.02 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 05:56:45,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2260773.3333333335, ans=0.125
2023-11-23 05:57:15,353 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339150
2023-11-23 05:57:38,396 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2500, loss[loss=0.07582, simple_loss=0.1032, pruned_loss=0.01585, audio_tagging_loss=0.008385, over 15916.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.092, pruned_loss=0.01389, audio_tagging_loss=0.00948, over 3050483.95 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:58:18,163 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.151e+01 8.943e+01 9.911e+01 1.178e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 05:58:19,514 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339200
2023-11-23 05:58:41,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2261440.0, ans=0.0
2023-11-23 05:58:42,713 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2550, loss[loss=0.06155, simple_loss=0.08446, pruned_loss=0.01289, audio_tagging_loss=0.006432, over 15778.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09163, pruned_loss=0.01397, audio_tagging_loss=0.009312, over 3046642.06 frames. ], batch size: 60, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:58:45,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2261440.0, ans=0.1
2023-11-23 05:58:46,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2261440.0, ans=0.0
2023-11-23 05:58:55,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2261506.6666666665, ans=0.0
2023-11-23 05:58:57,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2261506.6666666665, ans=0.0
2023-11-23 05:59:23,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339250
2023-11-23 05:59:30,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2261640.0, ans=0.125
2023-11-23 05:59:45,926 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2600, loss[loss=0.07528, simple_loss=0.106, pruned_loss=0.01359, audio_tagging_loss=0.008711, over 15555.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09158, pruned_loss=0.01399, audio_tagging_loss=0.00924, over 3054870.62 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 05:59:57,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.whiten.whitening_limit, batch_count=2261773.3333333335, ans=12.0
2023-11-23 06:00:07,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2261840.0, ans=0.1
2023-11-23 06:00:09,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2261840.0, ans=0.0
2023-11-23 06:00:18,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2261906.6666666665, ans=0.05
2023-11-23 06:00:25,725 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.329e+01 8.953e+01 9.650e+01 1.435e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 06:00:27,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339300
2023-11-23 06:00:33,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_ff3.min_abs, batch_count=2261973.3333333335, ans=0.2
2023-11-23 06:00:36,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2262040.0, ans=0.0
2023-11-23 06:00:50,472 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2650, loss[loss=0.05887, simple_loss=0.08088, pruned_loss=0.009229, audio_tagging_loss=0.009199, over 15378.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09171, pruned_loss=0.01399, audio_tagging_loss=0.00918, over 3054509.20 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:01:00,487 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:01:04,646 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.55 vs. limit=15.0
2023-11-23 06:01:18,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2262240.0, ans=0.09899494936611666
2023-11-23 06:01:27,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2262306.6666666665, ans=0.125
2023-11-23 06:01:31,605 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339350
2023-11-23 06:01:40,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2262373.3333333335, ans=0.1
2023-11-23 06:01:53,648 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2700, loss[loss=0.0781, simple_loss=0.09033, pruned_loss=0.02188, audio_tagging_loss=0.01106, over 14338.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09163, pruned_loss=0.014, audio_tagging_loss=0.009168, over 3054992.45 frames. ], batch size: 56, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:02:03,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2262440.0, ans=0.125
2023-11-23 06:02:09,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2262506.6666666665, ans=0.125
2023-11-23 06:02:13,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.58 vs. limit=15.0
2023-11-23 06:02:16,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2262506.6666666665, ans=0.0
2023-11-23 06:02:33,765 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.633e+01 8.218e+01 8.946e+01 9.636e+01 1.335e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 06:02:35,138 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339400
2023-11-23 06:02:35,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2262640.0, ans=0.07
2023-11-23 06:02:49,085 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.34 vs. limit=15.0
2023-11-23 06:02:49,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2262706.6666666665, ans=0.2
2023-11-23 06:02:58,441 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2750, loss[loss=0.07039, simple_loss=0.09802, pruned_loss=0.01191, audio_tagging_loss=0.009465, over 14471.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09182, pruned_loss=0.01396, audio_tagging_loss=0.009162, over 3054402.15 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:03:02,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=15.0
2023-11-23 06:03:18,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2262840.0, ans=0.0
2023-11-23 06:03:39,644 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339450
2023-11-23 06:03:41,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2262973.3333333335, ans=0.125
2023-11-23 06:03:52,617 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:04:03,064 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2800, loss[loss=0.08456, simple_loss=0.1174, pruned_loss=0.01746, audio_tagging_loss=0.008398, over 15158.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09152, pruned_loss=0.01408, audio_tagging_loss=0.009197, over 3045866.89 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 32.0
2023-11-23 06:04:15,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2263173.3333333335, ans=0.0
2023-11-23 06:04:19,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2263173.3333333335, ans=0.0
2023-11-23 06:04:31,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2263240.0, ans=0.0
2023-11-23 06:04:43,995 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.600e+01 8.163e+01 8.713e+01 9.405e+01 1.279e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-23 06:04:44,146 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339500
2023-11-23 06:04:59,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2263373.3333333335, ans=0.0
2023-11-23 06:05:06,458 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2850, loss[loss=0.07083, simple_loss=0.09944, pruned_loss=0.01406, audio_tagging_loss=0.007049, over 14520.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09174, pruned_loss=0.01414, audio_tagging_loss=0.00918, over 3046248.91 frames. ], batch size: 53, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:05:08,340 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.14 vs. limit=6.0
2023-11-23 06:05:11,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2263440.0, ans=0.2
2023-11-23 06:05:11,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2263440.0, ans=0.125
2023-11-23 06:05:15,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2263440.0, ans=0.0
2023-11-23 06:05:23,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2263506.6666666665, ans=0.1
2023-11-23 06:05:31,482 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.02 vs. limit=15.0
2023-11-23 06:05:36,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2263573.3333333335, ans=0.05
2023-11-23 06:05:48,765 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339550
2023-11-23 06:05:51,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2263640.0, ans=0.0
2023-11-23 06:05:57,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2263706.6666666665, ans=0.125
2023-11-23 06:06:00,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2263706.6666666665, ans=0.125
2023-11-23 06:06:11,465 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2900, loss[loss=0.06846, simple_loss=0.08878, pruned_loss=0.01463, audio_tagging_loss=0.009438, over 15380.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09248, pruned_loss=0.01432, audio_tagging_loss=0.009105, over 3041893.03 frames. ], batch size: 57, lr: 2.36e-03, grad_scale: 16.0
2023-11-23 06:06:47,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2263906.6666666665, ans=0.2
2023-11-23 06:06:52,981 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.150e+01 8.750e+01 9.470e+01 1.345e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-23 06:06:53,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339600
2023-11-23 06:06:53,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2263973.3333333335, ans=10.0
2023-11-23 06:07:12,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2264040.0, ans=0.125
2023-11-23 06:07:15,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2264040.0, ans=0.07
2023-11-23 06:07:17,754 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 2950, loss[loss=0.07346, simple_loss=0.101, pruned_loss=0.01347, audio_tagging_loss=0.009494, over 14753.00 frames. ], tot_loss[loss=0.07053, simple_loss=0.09376, pruned_loss=0.01455, audio_tagging_loss=0.0091, over 3047774.80 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:07:19,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2264106.6666666665, ans=0.0
2023-11-23 06:07:31,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2264173.3333333335, ans=0.2
2023-11-23 06:07:46,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2264240.0, ans=0.125
2023-11-23 06:07:51,609 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.01 vs. limit=15.0
2023-11-23 06:07:59,554 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339650
2023-11-23 06:08:22,256 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3000, loss[loss=0.04866, simple_loss=0.05237, pruned_loss=0.009372, audio_tagging_loss=0.0131, over 14456.00 frames. ], tot_loss[loss=0.07094, simple_loss=0.09463, pruned_loss=0.01456, audio_tagging_loss=0.009065, over 3052501.96 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:08:22,260 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 06:09:05,318 INFO [train_asr.py:1253] (0/4) Epoch 29, validation: loss=0.05823, simple_loss=0.05127, pruned_loss=0.005185, audio_tagging_loss=0.02741, over 4681554.00 frames. 
2023-11-23 06:09:05,319 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 06:09:09,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2264440.0, ans=0.2
2023-11-23 06:09:28,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2264506.6666666665, ans=0.125
2023-11-23 06:09:30,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2264573.3333333335, ans=0.125
2023-11-23 06:09:34,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2264573.3333333335, ans=0.0
2023-11-23 06:09:46,385 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.265e+01 8.432e+01 8.978e+01 9.876e+01 1.396e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 06:09:46,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339700
2023-11-23 06:10:02,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2264706.6666666665, ans=0.125
2023-11-23 06:10:08,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2264706.6666666665, ans=0.125
2023-11-23 06:10:10,837 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3050, loss[loss=0.03603, simple_loss=0.04048, pruned_loss=0.004008, audio_tagging_loss=0.01178, over 15763.00 frames. ], tot_loss[loss=0.07056, simple_loss=0.09388, pruned_loss=0.01446, audio_tagging_loss=0.009166, over 3053488.71 frames. ], batch size: 64, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:10:11,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2264773.3333333335, ans=0.0
2023-11-23 06:10:23,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2264840.0, ans=0.0
2023-11-23 06:10:46,327 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:10:49,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2264973.3333333335, ans=0.125
2023-11-23 06:10:52,460 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339750
2023-11-23 06:11:06,795 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.03 vs. limit=22.5
2023-11-23 06:11:14,624 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3100, loss[loss=0.06731, simple_loss=0.08719, pruned_loss=0.01257, audio_tagging_loss=0.01115, over 15602.00 frames. ], tot_loss[loss=0.07068, simple_loss=0.09401, pruned_loss=0.01448, audio_tagging_loss=0.009199, over 3053553.36 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 8.0
2023-11-23 06:11:17,793 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.43 vs. limit=12.0
2023-11-23 06:11:18,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2265106.6666666665, ans=0.0
2023-11-23 06:11:33,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2265173.3333333335, ans=0.0
2023-11-23 06:11:55,554 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339800
2023-11-23 06:11:56,579 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.703e+01 8.232e+01 8.826e+01 9.441e+01 1.388e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 06:12:18,035 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3150, loss[loss=0.07763, simple_loss=0.1139, pruned_loss=0.01347, audio_tagging_loss=0.007225, over 15400.00 frames. ], tot_loss[loss=0.07086, simple_loss=0.0943, pruned_loss=0.01447, audio_tagging_loss=0.009238, over 3060740.59 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 8.0
2023-11-23 06:12:36,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2265506.6666666665, ans=0.0
2023-11-23 06:12:59,787 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339850
2023-11-23 06:13:22,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2265773.3333333335, ans=0.1
2023-11-23 06:13:23,921 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3200, loss[loss=0.04548, simple_loss=0.0621, pruned_loss=0.002912, audio_tagging_loss=0.01151, over 14164.00 frames. ], tot_loss[loss=0.07085, simple_loss=0.09402, pruned_loss=0.01449, audio_tagging_loss=0.009349, over 3060240.45 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:13:36,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2265840.0, ans=0.125
2023-11-23 06:13:43,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2265840.0, ans=0.1
2023-11-23 06:13:46,501 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.17 vs. limit=10.0
2023-11-23 06:13:46,763 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.31 vs. limit=15.0
2023-11-23 06:14:04,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339900
2023-11-23 06:14:05,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2265973.3333333335, ans=0.125
2023-11-23 06:14:06,191 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.473e+01 8.333e+01 9.166e+01 9.819e+01 1.242e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 06:14:13,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2266040.0, ans=0.125
2023-11-23 06:14:26,815 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3250, loss[loss=0.05613, simple_loss=0.05122, pruned_loss=0.01173, audio_tagging_loss=0.01878, over 14605.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09287, pruned_loss=0.01424, audio_tagging_loss=0.009494, over 3051622.78 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:14:29,904 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.03 vs. limit=22.5
2023-11-23 06:14:35,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2266106.6666666665, ans=0.2
2023-11-23 06:14:42,424 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.38 vs. limit=15.0
2023-11-23 06:14:56,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2266240.0, ans=0.125
2023-11-23 06:14:56,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2266240.0, ans=10.0
2023-11-23 06:15:01,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2266240.0, ans=0.5
2023-11-23 06:15:06,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2266306.6666666665, ans=0.125
2023-11-23 06:15:08,465 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 339950
2023-11-23 06:15:30,093 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.11 vs. limit=15.0
2023-11-23 06:15:30,556 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3300, loss[loss=0.05395, simple_loss=0.07205, pruned_loss=0.00662, audio_tagging_loss=0.01131, over 15292.00 frames. ], tot_loss[loss=0.07049, simple_loss=0.09336, pruned_loss=0.01427, audio_tagging_loss=0.009536, over 3056122.77 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:15:40,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2266440.0, ans=0.2
2023-11-23 06:15:50,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2266506.6666666665, ans=0.0
2023-11-23 06:16:07,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2266573.3333333335, ans=0.125
2023-11-23 06:16:12,648 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340000
2023-11-23 06:16:13,718 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.818e+01 8.298e+01 8.900e+01 9.803e+01 1.322e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 06:16:14,157 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-340000.pt
2023-11-23 06:16:30,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2266706.6666666665, ans=0.125
2023-11-23 06:16:34,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2266706.6666666665, ans=0.1
2023-11-23 06:16:40,089 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3350, loss[loss=0.07842, simple_loss=0.1017, pruned_loss=0.01795, audio_tagging_loss=0.009638, over 15922.00 frames. ], tot_loss[loss=0.07078, simple_loss=0.09376, pruned_loss=0.01445, audio_tagging_loss=0.009453, over 3057901.12 frames. ], batch size: 60, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:16:55,174 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.43 vs. limit=15.0
2023-11-23 06:17:08,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2266906.6666666665, ans=0.125
2023-11-23 06:17:17,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2266973.3333333335, ans=0.05
2023-11-23 06:17:20,213 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340050
2023-11-23 06:17:29,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2266973.3333333335, ans=0.0
2023-11-23 06:17:43,970 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3400, loss[loss=0.0619, simple_loss=0.08218, pruned_loss=0.01249, audio_tagging_loss=0.008325, over 15784.00 frames. ], tot_loss[loss=0.07071, simple_loss=0.09406, pruned_loss=0.01439, audio_tagging_loss=0.009297, over 3056660.91 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:17:54,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2267106.6666666665, ans=0.0
2023-11-23 06:18:25,577 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340100
2023-11-23 06:18:26,635 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 8.359e+01 8.997e+01 9.733e+01 1.260e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 06:18:27,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2267306.6666666665, ans=0.0
2023-11-23 06:18:47,389 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3450, loss[loss=0.06986, simple_loss=0.09134, pruned_loss=0.01406, audio_tagging_loss=0.01012, over 14886.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.0939, pruned_loss=0.01438, audio_tagging_loss=0.009102, over 3054421.00 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:18:48,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2267440.0, ans=0.0
2023-11-23 06:19:09,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2267506.6666666665, ans=0.125
2023-11-23 06:19:14,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2267573.3333333335, ans=0.0
2023-11-23 06:19:15,288 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.24 vs. limit=22.5
2023-11-23 06:19:23,029 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.83 vs. limit=10.0
2023-11-23 06:19:29,716 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340150
2023-11-23 06:19:31,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2267640.0, ans=0.0
2023-11-23 06:19:53,630 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3500, loss[loss=0.06279, simple_loss=0.08658, pruned_loss=0.009381, audio_tagging_loss=0.01012, over 15851.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09374, pruned_loss=0.0143, audio_tagging_loss=0.009019, over 3046048.88 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:20:15,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2267840.0, ans=0.125
2023-11-23 06:20:20,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2267906.6666666665, ans=0.0
2023-11-23 06:20:25,097 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:20:27,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2267906.6666666665, ans=0.125
2023-11-23 06:20:34,333 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340200
2023-11-23 06:20:35,338 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.175e+01 8.873e+01 9.452e+01 1.244e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 06:20:47,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2268040.0, ans=0.125
2023-11-23 06:20:58,544 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3550, loss[loss=0.04921, simple_loss=0.05854, pruned_loss=0.01137, audio_tagging_loss=0.008567, over 16351.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.0923, pruned_loss=0.01395, audio_tagging_loss=0.00902, over 3049329.35 frames. ], batch size: 62, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:21:19,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2268173.3333333335, ans=0.0
2023-11-23 06:21:39,472 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340250
2023-11-23 06:21:57,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2268373.3333333335, ans=0.125
2023-11-23 06:22:01,965 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3600, loss[loss=0.06364, simple_loss=0.09231, pruned_loss=0.009808, audio_tagging_loss=0.007681, over 14609.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09278, pruned_loss=0.01423, audio_tagging_loss=0.009009, over 3046367.78 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:22:21,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2268506.6666666665, ans=0.2
2023-11-23 06:22:24,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2268506.6666666665, ans=0.125
2023-11-23 06:22:43,599 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340300
2023-11-23 06:22:44,650 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.233e+01 8.921e+01 9.911e+01 1.235e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 06:22:45,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2268640.0, ans=0.125
2023-11-23 06:22:47,679 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.61 vs. limit=10.0
2023-11-23 06:23:07,321 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3650, loss[loss=0.06654, simple_loss=0.09672, pruned_loss=0.01153, audio_tagging_loss=0.006647, over 15548.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09315, pruned_loss=0.01438, audio_tagging_loss=0.008934, over 3046605.75 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:23:10,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2268773.3333333335, ans=0.0
2023-11-23 06:23:20,367 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.54 vs. limit=15.0
2023-11-23 06:23:25,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2268840.0, ans=0.0
2023-11-23 06:23:35,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2268906.6666666665, ans=0.0
2023-11-23 06:23:42,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2268906.6666666665, ans=0.1
2023-11-23 06:23:47,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340350
2023-11-23 06:23:53,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2268973.3333333335, ans=0.125
2023-11-23 06:24:11,252 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3700, loss[loss=0.06086, simple_loss=0.07727, pruned_loss=0.0139, audio_tagging_loss=0.008333, over 15229.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09276, pruned_loss=0.01435, audio_tagging_loss=0.008882, over 3051070.05 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:24:32,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2269173.3333333335, ans=0.015
2023-11-23 06:24:39,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2269240.0, ans=0.125
2023-11-23 06:24:40,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2269240.0, ans=0.125
2023-11-23 06:24:46,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2269240.0, ans=0.125
2023-11-23 06:24:52,636 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340400
2023-11-23 06:24:55,304 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.367e+01 8.831e+01 9.767e+01 1.153e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 06:25:11,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2269373.3333333335, ans=0.125
2023-11-23 06:25:16,075 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3750, loss[loss=0.06119, simple_loss=0.07714, pruned_loss=0.01145, audio_tagging_loss=0.01118, over 15039.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09228, pruned_loss=0.01435, audio_tagging_loss=0.009035, over 3052647.61 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:25:57,883 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340450
2023-11-23 06:25:58,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2269640.0, ans=0.2
2023-11-23 06:25:59,028 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:26:14,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2269706.6666666665, ans=0.09899494936611666
2023-11-23 06:26:20,472 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3800, loss[loss=0.07646, simple_loss=0.1014, pruned_loss=0.01863, audio_tagging_loss=0.007119, over 15315.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09241, pruned_loss=0.01431, audio_tagging_loss=0.009088, over 3052754.47 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:26:28,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2269773.3333333335, ans=0.2
2023-11-23 06:26:31,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2269773.3333333335, ans=0.0
2023-11-23 06:26:38,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2269840.0, ans=0.1
2023-11-23 06:26:45,395 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.82 vs. limit=15.0
2023-11-23 06:26:46,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2269906.6666666665, ans=0.0
2023-11-23 06:26:53,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2269906.6666666665, ans=0.0
2023-11-23 06:27:01,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340500
2023-11-23 06:27:04,359 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.380e+01 8.994e+01 9.826e+01 1.163e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 06:27:04,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2269973.3333333335, ans=0.125
2023-11-23 06:27:09,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2269973.3333333335, ans=0.05
2023-11-23 06:27:26,058 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3850, loss[loss=0.05901, simple_loss=0.0751, pruned_loss=0.01155, audio_tagging_loss=0.009904, over 15732.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09276, pruned_loss=0.01436, audio_tagging_loss=0.009159, over 3049790.20 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:27:33,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2270106.6666666665, ans=0.2
2023-11-23 06:27:44,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.39 vs. limit=15.0
2023-11-23 06:27:45,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2270173.3333333335, ans=0.125
2023-11-23 06:28:04,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2270306.6666666665, ans=0.125
2023-11-23 06:28:06,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.44 vs. limit=22.5
2023-11-23 06:28:08,179 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340550
2023-11-23 06:28:12,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2270306.6666666665, ans=0.1
2023-11-23 06:28:14,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2270306.6666666665, ans=0.125
2023-11-23 06:28:19,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.92 vs. limit=10.0
2023-11-23 06:28:31,088 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3900, loss[loss=0.07945, simple_loss=0.101, pruned_loss=0.02024, audio_tagging_loss=0.00871, over 14512.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09343, pruned_loss=0.01434, audio_tagging_loss=0.009118, over 3048783.37 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:28:32,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2270440.0, ans=0.2
2023-11-23 06:28:37,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2270440.0, ans=0.0
2023-11-23 06:29:05,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2270573.3333333335, ans=0.2
2023-11-23 06:29:11,718 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340600
2023-11-23 06:29:13,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2270640.0, ans=0.0
2023-11-23 06:29:14,347 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.444e+01 8.394e+01 8.868e+01 9.858e+01 1.292e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 06:29:20,792 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.41 vs. limit=6.0
2023-11-23 06:29:35,329 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 3950, loss[loss=0.0571, simple_loss=0.07208, pruned_loss=0.008972, audio_tagging_loss=0.01208, over 14619.00 frames. ], tot_loss[loss=0.07028, simple_loss=0.09346, pruned_loss=0.01436, audio_tagging_loss=0.009191, over 3041293.55 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:30:05,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2270906.6666666665, ans=0.1
2023-11-23 06:30:16,658 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340650
2023-11-23 06:30:24,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2270973.3333333335, ans=0.2
2023-11-23 06:30:27,903 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.76 vs. limit=22.5
2023-11-23 06:30:34,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2271040.0, ans=0.1
2023-11-23 06:30:40,563 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4000, loss[loss=0.07723, simple_loss=0.1054, pruned_loss=0.01526, audio_tagging_loss=0.009265, over 15145.00 frames. ], tot_loss[loss=0.0704, simple_loss=0.0936, pruned_loss=0.01428, audio_tagging_loss=0.009312, over 3042967.20 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:30:40,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2271106.6666666665, ans=0.125
2023-11-23 06:31:01,383 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.21 vs. limit=6.0
2023-11-23 06:31:21,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2271306.6666666665, ans=0.2
2023-11-23 06:31:22,163 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340700
2023-11-23 06:31:25,687 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.305e+01 8.425e+01 8.951e+01 9.689e+01 1.650e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 06:31:44,120 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4050, loss[loss=0.06927, simple_loss=0.08273, pruned_loss=0.0165, audio_tagging_loss=0.01141, over 14201.00 frames. ], tot_loss[loss=0.07075, simple_loss=0.09415, pruned_loss=0.01437, audio_tagging_loss=0.009311, over 3039201.24 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:31:45,422 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:31:47,410 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.09 vs. limit=10.0
2023-11-23 06:32:04,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2271506.6666666665, ans=10.0
2023-11-23 06:32:04,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2271506.6666666665, ans=0.0
2023-11-23 06:32:07,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2271506.6666666665, ans=0.2
2023-11-23 06:32:12,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2271573.3333333335, ans=0.125
2023-11-23 06:32:13,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.75 vs. limit=22.5
2023-11-23 06:32:26,212 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340750
2023-11-23 06:32:32,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2271640.0, ans=0.0
2023-11-23 06:32:48,644 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4100, loss[loss=0.08875, simple_loss=0.1221, pruned_loss=0.0215, audio_tagging_loss=0.006191, over 14929.00 frames. ], tot_loss[loss=0.07076, simple_loss=0.09426, pruned_loss=0.01438, audio_tagging_loss=0.009246, over 3040716.43 frames. ], batch size: 52, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:32:50,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2271773.3333333335, ans=0.125
2023-11-23 06:33:06,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2271840.0, ans=0.0
2023-11-23 06:33:09,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2271840.0, ans=0.125
2023-11-23 06:33:24,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2271906.6666666665, ans=0.2
2023-11-23 06:33:30,037 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340800
2023-11-23 06:33:34,564 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.442e+01 8.896e+01 9.495e+01 1.100e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 06:33:54,750 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4150, loss[loss=0.09525, simple_loss=0.1269, pruned_loss=0.0235, audio_tagging_loss=0.008321, over 15077.00 frames. ], tot_loss[loss=0.07036, simple_loss=0.09408, pruned_loss=0.01424, audio_tagging_loss=0.009084, over 3041200.72 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:33:55,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2272106.6666666665, ans=0.125
2023-11-23 06:33:57,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2272106.6666666665, ans=0.125
2023-11-23 06:34:18,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2272240.0, ans=0.125
2023-11-23 06:34:22,393 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.61 vs. limit=15.0
2023-11-23 06:34:36,378 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340850
2023-11-23 06:34:38,719 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:34:42,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2272306.6666666665, ans=0.1
2023-11-23 06:34:49,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2272373.3333333335, ans=0.0
2023-11-23 06:34:57,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2272440.0, ans=0.125
2023-11-23 06:34:58,257 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4200, loss[loss=0.06082, simple_loss=0.08141, pruned_loss=0.01065, audio_tagging_loss=0.009466, over 14961.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09457, pruned_loss=0.01427, audio_tagging_loss=0.008967, over 3045298.79 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:35:00,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2272440.0, ans=0.125
2023-11-23 06:35:03,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2272440.0, ans=0.125
2023-11-23 06:35:13,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2272506.6666666665, ans=0.0
2023-11-23 06:35:16,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2272506.6666666665, ans=0.0
2023-11-23 06:35:40,246 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340900
2023-11-23 06:35:43,732 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.311e+01 8.855e+01 9.584e+01 1.424e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 06:35:51,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2272706.6666666665, ans=0.125
2023-11-23 06:36:02,081 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4250, loss[loss=0.0641, simple_loss=0.08389, pruned_loss=0.01169, audio_tagging_loss=0.01047, over 15018.00 frames. ], tot_loss[loss=0.07046, simple_loss=0.09475, pruned_loss=0.01428, audio_tagging_loss=0.008799, over 3049862.98 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:36:07,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2272773.3333333335, ans=0.2
2023-11-23 06:36:15,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2272840.0, ans=0.07
2023-11-23 06:36:29,452 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.17 vs. limit=10.0
2023-11-23 06:36:35,913 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.61 vs. limit=12.0
2023-11-23 06:36:43,945 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 340950
2023-11-23 06:36:48,214 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.35 vs. limit=15.0
2023-11-23 06:37:05,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2273040.0, ans=0.125
2023-11-23 06:37:07,792 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4300, loss[loss=0.07026, simple_loss=0.09534, pruned_loss=0.01448, audio_tagging_loss=0.008108, over 14602.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09445, pruned_loss=0.01428, audio_tagging_loss=0.008822, over 3047703.37 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:37:09,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2273106.6666666665, ans=0.125
2023-11-23 06:37:10,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2273106.6666666665, ans=0.025
2023-11-23 06:37:14,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.89 vs. limit=15.0
2023-11-23 06:37:18,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2273173.3333333335, ans=0.1
2023-11-23 06:37:25,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2273173.3333333335, ans=0.125
2023-11-23 06:37:37,671 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:37:48,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341000
2023-11-23 06:37:52,142 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:37:53,085 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.426e+01 8.312e+01 8.993e+01 9.823e+01 1.635e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 06:38:05,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2273373.3333333335, ans=0.0
2023-11-23 06:38:06,125 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.25 vs. limit=22.5
2023-11-23 06:38:10,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2273440.0, ans=0.2
2023-11-23 06:38:11,589 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4350, loss[loss=0.07182, simple_loss=0.1002, pruned_loss=0.01207, audio_tagging_loss=0.00963, over 14832.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09514, pruned_loss=0.01447, audio_tagging_loss=0.008793, over 3037895.32 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:38:43,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2273573.3333333335, ans=0.125
2023-11-23 06:38:52,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341050
2023-11-23 06:38:53,139 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=7.52 vs. limit=15.0
2023-11-23 06:38:59,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2273640.0, ans=0.1
2023-11-23 06:39:14,632 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4400, loss[loss=0.05837, simple_loss=0.07392, pruned_loss=0.01019, audio_tagging_loss=0.01122, over 14670.00 frames. ], tot_loss[loss=0.07109, simple_loss=0.09534, pruned_loss=0.01469, audio_tagging_loss=0.008734, over 3042463.45 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:39:17,656 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.19 vs. limit=12.0
2023-11-23 06:39:22,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2273773.3333333335, ans=0.1
2023-11-23 06:39:26,824 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.47 vs. limit=15.0
2023-11-23 06:39:45,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.73 vs. limit=15.0
2023-11-23 06:39:50,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2273906.6666666665, ans=0.125
2023-11-23 06:39:55,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341100
2023-11-23 06:40:00,191 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.327e+01 8.740e+01 9.683e+01 1.205e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 06:40:14,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2274040.0, ans=0.125
2023-11-23 06:40:17,257 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.77 vs. limit=15.0
2023-11-23 06:40:19,124 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4450, loss[loss=0.07701, simple_loss=0.09786, pruned_loss=0.01741, audio_tagging_loss=0.01066, over 15739.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09539, pruned_loss=0.01471, audio_tagging_loss=0.008799, over 3046658.24 frames. ], batch size: 59, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:40:58,336 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341150
2023-11-23 06:41:00,095 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.42 vs. limit=22.5
2023-11-23 06:41:12,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2274373.3333333335, ans=0.125
2023-11-23 06:41:16,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2274373.3333333335, ans=0.125
2023-11-23 06:41:20,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2274373.3333333335, ans=0.1
2023-11-23 06:41:22,093 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4500, loss[loss=0.05098, simple_loss=0.06854, pruned_loss=0.00951, audio_tagging_loss=0.007203, over 14734.00 frames. ], tot_loss[loss=0.07145, simple_loss=0.09591, pruned_loss=0.01475, audio_tagging_loss=0.008749, over 3056819.94 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:41:22,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2274440.0, ans=10.0
2023-11-23 06:41:43,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2274506.6666666665, ans=0.0
2023-11-23 06:41:49,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2274573.3333333335, ans=10.0
2023-11-23 06:42:01,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2274640.0, ans=0.125
2023-11-23 06:42:03,437 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341200
2023-11-23 06:42:08,628 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 8.193e+01 8.685e+01 9.597e+01 1.191e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-23 06:42:12,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2274706.6666666665, ans=0.125
2023-11-23 06:42:16,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2274706.6666666665, ans=0.125
2023-11-23 06:42:25,798 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4550, loss[loss=0.06531, simple_loss=0.07593, pruned_loss=0.01572, audio_tagging_loss=0.01163, over 13713.00 frames. ], tot_loss[loss=0.0715, simple_loss=0.0956, pruned_loss=0.01478, audio_tagging_loss=0.008916, over 3053353.25 frames. ], batch size: 54, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:42:29,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2274773.3333333335, ans=0.0
2023-11-23 06:42:41,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2274840.0, ans=0.0
2023-11-23 06:43:04,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2274973.3333333335, ans=0.0
2023-11-23 06:43:06,624 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341250
2023-11-23 06:43:09,240 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:43:11,437 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 06:43:21,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2275040.0, ans=0.2
2023-11-23 06:43:28,969 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4600, loss[loss=0.08375, simple_loss=0.09625, pruned_loss=0.02702, audio_tagging_loss=0.008606, over 15044.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09529, pruned_loss=0.01484, audio_tagging_loss=0.008946, over 3051207.41 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:44:05,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2275306.6666666665, ans=0.0
2023-11-23 06:44:08,903 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341300
2023-11-23 06:44:13,201 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.39 vs. limit=15.0
2023-11-23 06:44:13,567 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.895e+01 8.306e+01 8.944e+01 9.725e+01 1.153e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 06:44:21,490 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:44:26,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2275373.3333333335, ans=0.125
2023-11-23 06:44:31,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2275440.0, ans=0.125
2023-11-23 06:44:32,734 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4650, loss[loss=0.0844, simple_loss=0.1085, pruned_loss=0.01915, audio_tagging_loss=0.01101, over 14720.00 frames. ], tot_loss[loss=0.07127, simple_loss=0.09499, pruned_loss=0.01472, audio_tagging_loss=0.009056, over 3046097.12 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:44:42,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2275440.0, ans=0.125
2023-11-23 06:45:13,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341350
2023-11-23 06:45:17,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2275640.0, ans=0.2
2023-11-23 06:45:20,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2275640.0, ans=0.04949747468305833
2023-11-23 06:45:25,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2275706.6666666665, ans=0.0
2023-11-23 06:45:34,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2275706.6666666665, ans=0.0
2023-11-23 06:45:36,237 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4700, loss[loss=0.08224, simple_loss=0.1041, pruned_loss=0.02082, audio_tagging_loss=0.009366, over 14985.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09478, pruned_loss=0.0147, audio_tagging_loss=0.009163, over 3046384.30 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:46:00,106 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:46:18,057 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341400
2023-11-23 06:46:23,124 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.995e+01 8.437e+01 9.019e+01 9.512e+01 1.168e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 06:46:26,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten.whitening_limit, batch_count=2275973.3333333335, ans=15.0
2023-11-23 06:46:30,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2276040.0, ans=0.1
2023-11-23 06:46:40,573 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4750, loss[loss=0.07415, simple_loss=0.09875, pruned_loss=0.01502, audio_tagging_loss=0.009751, over 15810.00 frames. ], tot_loss[loss=0.07082, simple_loss=0.09416, pruned_loss=0.01449, audio_tagging_loss=0.009244, over 3049588.36 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:46:45,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.57 vs. limit=10.0
2023-11-23 06:47:05,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2276173.3333333335, ans=0.125
2023-11-23 06:47:18,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.71 vs. limit=5.0
2023-11-23 06:47:21,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.71 vs. limit=15.0
2023-11-23 06:47:22,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341450
2023-11-23 06:47:46,391 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4800, loss[loss=0.07101, simple_loss=0.1007, pruned_loss=0.01309, audio_tagging_loss=0.007587, over 15195.00 frames. ], tot_loss[loss=0.07011, simple_loss=0.09308, pruned_loss=0.01413, audio_tagging_loss=0.009436, over 3050299.89 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:47:54,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2276440.0, ans=0.0
2023-11-23 06:48:03,506 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:48:07,107 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:48:11,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.37 vs. limit=10.0
2023-11-23 06:48:25,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2276640.0, ans=0.1
2023-11-23 06:48:27,919 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341500
2023-11-23 06:48:31,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2276640.0, ans=0.125
2023-11-23 06:48:33,945 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.380e+01 8.789e+01 9.333e+01 1.267e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-23 06:48:35,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2276640.0, ans=0.125
2023-11-23 06:48:50,512 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4850, loss[loss=0.0892, simple_loss=0.1173, pruned_loss=0.02115, audio_tagging_loss=0.009391, over 16397.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09228, pruned_loss=0.01414, audio_tagging_loss=0.009581, over 3045345.29 frames. ], batch size: 60, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:48:58,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2276773.3333333335, ans=0.1
2023-11-23 06:49:21,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2276906.6666666665, ans=0.125
2023-11-23 06:49:26,225 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.66 vs. limit=15.0
2023-11-23 06:49:32,091 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341550
2023-11-23 06:49:33,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2276973.3333333335, ans=0.125
2023-11-23 06:49:38,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2276973.3333333335, ans=0.125
2023-11-23 06:49:43,092 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 06:49:43,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2277040.0, ans=0.125
2023-11-23 06:49:54,001 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4900, loss[loss=0.07916, simple_loss=0.1115, pruned_loss=0.01789, audio_tagging_loss=0.005514, over 15538.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09283, pruned_loss=0.01418, audio_tagging_loss=0.009441, over 3043448.84 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:49:54,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2277106.6666666665, ans=0.5
2023-11-23 06:49:54,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.13 vs. limit=15.0
2023-11-23 06:50:00,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2277106.6666666665, ans=0.125
2023-11-23 06:50:03,338 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.94 vs. limit=15.0
2023-11-23 06:50:35,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.93 vs. limit=15.0
2023-11-23 06:50:35,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341600
2023-11-23 06:50:41,946 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.445e+01 8.899e+01 9.661e+01 1.338e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 06:50:57,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2277373.3333333335, ans=0.125
2023-11-23 06:50:59,330 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 4950, loss[loss=0.06533, simple_loss=0.09085, pruned_loss=0.01335, audio_tagging_loss=0.006563, over 14698.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09299, pruned_loss=0.0142, audio_tagging_loss=0.00918, over 3039539.92 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:51:02,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2277440.0, ans=0.125
2023-11-23 06:51:16,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.19 vs. limit=10.0
2023-11-23 06:51:39,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2277640.0, ans=0.125
2023-11-23 06:51:40,524 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341650
2023-11-23 06:51:40,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2277640.0, ans=0.1
2023-11-23 06:51:49,704 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.06 vs. limit=15.0
2023-11-23 06:52:03,335 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5000, loss[loss=0.06896, simple_loss=0.09218, pruned_loss=0.01534, audio_tagging_loss=0.007532, over 15082.00 frames. ], tot_loss[loss=0.07029, simple_loss=0.09368, pruned_loss=0.0143, audio_tagging_loss=0.009147, over 3042657.88 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:52:17,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2277840.0, ans=0.0
2023-11-23 06:52:31,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2277906.6666666665, ans=0.125
2023-11-23 06:52:34,446 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.63 vs. limit=15.0
2023-11-23 06:52:38,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2277906.6666666665, ans=0.0
2023-11-23 06:52:43,696 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341700
2023-11-23 06:52:47,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2277973.3333333335, ans=0.125
2023-11-23 06:52:50,303 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.671e+01 8.065e+01 8.894e+01 1.001e+02 1.433e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 06:53:06,449 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5050, loss[loss=0.05252, simple_loss=0.07334, pruned_loss=0.007521, audio_tagging_loss=0.008322, over 15415.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09334, pruned_loss=0.01408, audio_tagging_loss=0.009019, over 3041062.24 frames. ], batch size: 60, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:53:11,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2278106.6666666665, ans=0.125
2023-11-23 06:53:26,728 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0
2023-11-23 06:53:31,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2278240.0, ans=0.2
2023-11-23 06:53:47,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341750
2023-11-23 06:54:10,955 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5100, loss[loss=0.06886, simple_loss=0.09293, pruned_loss=0.01456, audio_tagging_loss=0.007845, over 14918.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09205, pruned_loss=0.01386, audio_tagging_loss=0.009058, over 3042979.38 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:54:16,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2278440.0, ans=0.0
2023-11-23 06:54:25,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2278506.6666666665, ans=0.125
2023-11-23 06:54:36,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2278573.3333333335, ans=0.2
2023-11-23 06:54:44,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2278573.3333333335, ans=0.125
2023-11-23 06:54:47,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2278640.0, ans=0.0
2023-11-23 06:54:48,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2278640.0, ans=0.1
2023-11-23 06:54:52,229 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341800
2023-11-23 06:54:55,030 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.74 vs. limit=15.0
2023-11-23 06:54:59,228 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 7.874e+01 8.463e+01 9.175e+01 1.215e+02, threshold=1.693e+02, percent-clipped=0.0
2023-11-23 06:55:00,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2278640.0, ans=0.95
2023-11-23 06:55:04,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2278706.6666666665, ans=0.05
2023-11-23 06:55:05,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2278706.6666666665, ans=0.125
2023-11-23 06:55:10,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2278706.6666666665, ans=0.0
2023-11-23 06:55:10,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2278706.6666666665, ans=0.2
2023-11-23 06:55:15,521 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5150, loss[loss=0.07356, simple_loss=0.1067, pruned_loss=0.01356, audio_tagging_loss=0.006678, over 14331.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09233, pruned_loss=0.0139, audio_tagging_loss=0.009008, over 3038976.93 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:55:25,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2278773.3333333335, ans=0.0
2023-11-23 06:55:45,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2278906.6666666665, ans=0.125
2023-11-23 06:55:57,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341850
2023-11-23 06:56:20,328 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5200, loss[loss=0.06134, simple_loss=0.08347, pruned_loss=0.01013, audio_tagging_loss=0.009477, over 14579.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09235, pruned_loss=0.01386, audio_tagging_loss=0.009026, over 3035080.78 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 06:56:37,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2279173.3333333335, ans=0.035
2023-11-23 06:56:38,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2279173.3333333335, ans=0.125
2023-11-23 06:56:50,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2279240.0, ans=0.125
2023-11-23 06:56:51,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2279240.0, ans=0.5
2023-11-23 06:56:58,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2279306.6666666665, ans=0.0
2023-11-23 06:57:01,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341900
2023-11-23 06:57:06,335 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.40 vs. limit=6.0
2023-11-23 06:57:09,216 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.516e+01 8.360e+01 8.790e+01 9.384e+01 1.181e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-23 06:57:25,767 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5250, loss[loss=0.05839, simple_loss=0.08009, pruned_loss=0.009754, audio_tagging_loss=0.008596, over 16473.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09265, pruned_loss=0.01401, audio_tagging_loss=0.008996, over 3036889.03 frames. ], batch size: 60, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:57:33,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2279440.0, ans=0.0
2023-11-23 06:57:45,115 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.67 vs. limit=6.0
2023-11-23 06:57:55,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2279573.3333333335, ans=0.125
2023-11-23 06:58:06,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 341950
2023-11-23 06:58:06,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2279640.0, ans=0.2
2023-11-23 06:58:29,708 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5300, loss[loss=0.07644, simple_loss=0.1046, pruned_loss=0.01423, audio_tagging_loss=0.0099, over 14769.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09293, pruned_loss=0.01409, audio_tagging_loss=0.009, over 3037180.51 frames. ], batch size: 53, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:58:29,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2279773.3333333335, ans=0.125
2023-11-23 06:58:29,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2279773.3333333335, ans=0.0
2023-11-23 06:58:47,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2279840.0, ans=0.1
2023-11-23 06:58:52,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2279840.0, ans=0.05
2023-11-23 06:58:53,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2279906.6666666665, ans=0.0
2023-11-23 06:59:11,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342000
2023-11-23 06:59:18,965 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.762e+01 8.363e+01 8.739e+01 9.478e+01 1.273e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 06:59:30,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2280040.0, ans=0.125
2023-11-23 06:59:33,717 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5350, loss[loss=0.05716, simple_loss=0.07318, pruned_loss=0.009421, audio_tagging_loss=0.01115, over 14681.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09251, pruned_loss=0.01398, audio_tagging_loss=0.00901, over 3032371.20 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 06:59:53,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2280173.3333333335, ans=0.0
2023-11-23 07:00:04,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2280240.0, ans=0.125
2023-11-23 07:00:11,039 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.80 vs. limit=22.5
2023-11-23 07:00:15,461 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342050
2023-11-23 07:00:29,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2280373.3333333335, ans=0.1
2023-11-23 07:00:38,808 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5400, loss[loss=0.08745, simple_loss=0.1179, pruned_loss=0.02077, audio_tagging_loss=0.007746, over 15480.00 frames. ], tot_loss[loss=0.07033, simple_loss=0.09393, pruned_loss=0.01428, audio_tagging_loss=0.009081, over 3041719.24 frames. ], batch size: 58, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:00:41,398 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.50 vs. limit=15.0
2023-11-23 07:00:53,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2280506.6666666665, ans=0.125
2023-11-23 07:01:10,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2280573.3333333335, ans=0.125
2023-11-23 07:01:10,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2280573.3333333335, ans=0.025
2023-11-23 07:01:19,556 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342100
2023-11-23 07:01:27,618 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.934e+01 8.345e+01 9.025e+01 9.492e+01 1.163e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 07:01:32,850 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.06 vs. limit=15.0
2023-11-23 07:01:43,279 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5450, loss[loss=0.05445, simple_loss=0.07057, pruned_loss=0.009997, audio_tagging_loss=0.009163, over 15909.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09241, pruned_loss=0.01406, audio_tagging_loss=0.009194, over 3045410.33 frames. ], batch size: 61, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:01:48,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2280773.3333333335, ans=0.125
2023-11-23 07:02:00,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.78 vs. limit=10.0
2023-11-23 07:02:12,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2280906.6666666665, ans=0.0
2023-11-23 07:02:24,755 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342150
2023-11-23 07:02:38,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2281040.0, ans=0.0
2023-11-23 07:02:46,743 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5500, loss[loss=0.056, simple_loss=0.07099, pruned_loss=0.0104, audio_tagging_loss=0.0101, over 14868.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09307, pruned_loss=0.01413, audio_tagging_loss=0.009184, over 3046836.83 frames. ], batch size: 57, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:02:49,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2281106.6666666665, ans=0.125
2023-11-23 07:02:58,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2281173.3333333335, ans=0.025
2023-11-23 07:02:59,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2281173.3333333335, ans=0.09899494936611666
2023-11-23 07:03:07,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2281173.3333333335, ans=0.125
2023-11-23 07:03:23,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2281240.0, ans=0.0
2023-11-23 07:03:28,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342200
2023-11-23 07:03:31,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2281306.6666666665, ans=0.2
2023-11-23 07:03:35,750 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.376e+01 8.901e+01 9.455e+01 1.073e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 07:03:38,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2281373.3333333335, ans=0.125
2023-11-23 07:03:46,947 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.00 vs. limit=15.0
2023-11-23 07:03:48,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2281373.3333333335, ans=0.0
2023-11-23 07:03:51,710 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5550, loss[loss=0.06916, simple_loss=0.08789, pruned_loss=0.01277, audio_tagging_loss=0.01244, over 15489.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09218, pruned_loss=0.01394, audio_tagging_loss=0.009237, over 3048915.78 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:04:11,504 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.97 vs. limit=15.0
2023-11-23 07:04:17,482 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.09 vs. limit=15.0
2023-11-23 07:04:19,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2281573.3333333335, ans=0.1
2023-11-23 07:04:22,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2281573.3333333335, ans=0.125
2023-11-23 07:04:24,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2281573.3333333335, ans=0.125
2023-11-23 07:04:31,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342250
2023-11-23 07:04:55,523 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5600, loss[loss=0.06739, simple_loss=0.07844, pruned_loss=0.01457, audio_tagging_loss=0.0136, over 14463.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.0926, pruned_loss=0.01391, audio_tagging_loss=0.009268, over 3051365.18 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 07:05:36,320 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342300
2023-11-23 07:05:38,720 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 07:05:43,430 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.528e+01 8.225e+01 8.879e+01 9.451e+01 1.680e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-23 07:05:57,946 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5650, loss[loss=0.06601, simple_loss=0.08856, pruned_loss=0.01381, audio_tagging_loss=0.007911, over 15391.00 frames. ], tot_loss[loss=0.069, simple_loss=0.0913, pruned_loss=0.01379, audio_tagging_loss=0.009561, over 3039344.87 frames. ], batch size: 56, lr: 2.35e-03, grad_scale: 32.0
2023-11-23 07:06:34,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2282240.0, ans=0.125
2023-11-23 07:06:36,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2282306.6666666665, ans=0.125
2023-11-23 07:06:39,064 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342350
2023-11-23 07:06:49,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2282373.3333333335, ans=0.125
2023-11-23 07:06:51,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2282373.3333333335, ans=0.05
2023-11-23 07:06:52,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2282373.3333333335, ans=0.0
2023-11-23 07:07:01,242 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5700, loss[loss=0.06538, simple_loss=0.08227, pruned_loss=0.0115, audio_tagging_loss=0.01275, over 16458.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09125, pruned_loss=0.01374, audio_tagging_loss=0.009555, over 3044583.65 frames. ], batch size: 60, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:07:10,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2282440.0, ans=10.0
2023-11-23 07:07:15,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2282506.6666666665, ans=0.125
2023-11-23 07:07:16,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2282506.6666666665, ans=0.2
2023-11-23 07:07:18,037 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.40 vs. limit=22.5
2023-11-23 07:07:31,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2282573.3333333335, ans=0.0
2023-11-23 07:07:33,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2282573.3333333335, ans=0.125
2023-11-23 07:07:41,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342400
2023-11-23 07:07:50,744 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.447e+01 8.964e+01 9.780e+01 1.182e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 07:08:00,638 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.04 vs. limit=15.0
2023-11-23 07:08:06,387 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5750, loss[loss=0.05759, simple_loss=0.08001, pruned_loss=0.00943, audio_tagging_loss=0.008154, over 16727.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09136, pruned_loss=0.01377, audio_tagging_loss=0.009379, over 3047020.35 frames. ], batch size: 66, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:08:20,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2282840.0, ans=0.1
2023-11-23 07:08:28,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2282840.0, ans=0.0
2023-11-23 07:08:33,043 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.38 vs. limit=5.0
2023-11-23 07:08:42,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2282906.6666666665, ans=0.0
2023-11-23 07:08:45,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2282973.3333333335, ans=0.0
2023-11-23 07:08:48,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342450
2023-11-23 07:09:09,966 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5800, loss[loss=0.07757, simple_loss=0.1074, pruned_loss=0.01415, audio_tagging_loss=0.009725, over 14084.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09298, pruned_loss=0.01412, audio_tagging_loss=0.00922, over 3033461.67 frames. ], batch size: 55, lr: 2.35e-03, grad_scale: 16.0
2023-11-23 07:09:14,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2283106.6666666665, ans=0.125
2023-11-23 07:09:27,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2283173.3333333335, ans=0.125
2023-11-23 07:09:31,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2283173.3333333335, ans=0.0
2023-11-23 07:09:33,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2283173.3333333335, ans=0.015
2023-11-23 07:09:40,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2283240.0, ans=0.0
2023-11-23 07:09:42,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.49 vs. limit=15.0
2023-11-23 07:09:42,567 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.59 vs. limit=15.0
2023-11-23 07:09:51,946 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342500
2023-11-23 07:09:52,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2283306.6666666665, ans=0.125
2023-11-23 07:10:00,543 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.596e+01 8.186e+01 8.952e+01 9.751e+01 1.304e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 07:10:14,086 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5850, loss[loss=0.09617, simple_loss=0.1326, pruned_loss=0.02387, audio_tagging_loss=0.005977, over 15183.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09223, pruned_loss=0.01392, audio_tagging_loss=0.00918, over 3041653.52 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:10:19,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2283440.0, ans=0.0
2023-11-23 07:10:22,323 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.41 vs. limit=6.0
2023-11-23 07:10:27,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2283506.6666666665, ans=0.0
2023-11-23 07:10:36,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2283506.6666666665, ans=0.125
2023-11-23 07:10:46,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2283573.3333333335, ans=0.125
2023-11-23 07:10:48,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2283573.3333333335, ans=0.125
2023-11-23 07:10:50,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2283573.3333333335, ans=0.125
2023-11-23 07:10:55,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342550
2023-11-23 07:11:10,912 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.50 vs. limit=15.0
2023-11-23 07:11:19,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2283773.3333333335, ans=0.2
2023-11-23 07:11:20,072 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5900, loss[loss=0.05523, simple_loss=0.07097, pruned_loss=0.009415, audio_tagging_loss=0.01033, over 15042.00 frames. ], tot_loss[loss=0.06972, simple_loss=0.09295, pruned_loss=0.01411, audio_tagging_loss=0.009126, over 3044511.67 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:12:01,271 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342600
2023-11-23 07:12:10,820 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.390e+01 8.881e+01 9.843e+01 1.391e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-23 07:12:13,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2284040.0, ans=0.0
2023-11-23 07:12:24,217 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 5950, loss[loss=0.05637, simple_loss=0.07553, pruned_loss=0.009541, audio_tagging_loss=0.00907, over 14459.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09299, pruned_loss=0.01412, audio_tagging_loss=0.00907, over 3044644.44 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:12:25,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2284106.6666666665, ans=0.125
2023-11-23 07:12:41,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2284173.3333333335, ans=0.0
2023-11-23 07:12:52,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2284240.0, ans=0.125
2023-11-23 07:13:05,421 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342650
2023-11-23 07:13:08,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2284306.6666666665, ans=0.125
2023-11-23 07:13:27,467 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6000, loss[loss=0.06424, simple_loss=0.08923, pruned_loss=0.01029, audio_tagging_loss=0.009339, over 14255.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09252, pruned_loss=0.01404, audio_tagging_loss=0.009036, over 3034901.17 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:13:27,470 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 07:14:10,553 INFO [train_asr.py:1253] (0/4) Epoch 29, validation: loss=0.05847, simple_loss=0.05123, pruned_loss=0.005068, audio_tagging_loss=0.02778, over 4681554.00 frames. 
2023-11-23 07:14:10,553 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 07:14:20,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2284440.0, ans=0.1
2023-11-23 07:14:31,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2284506.6666666665, ans=0.1
2023-11-23 07:14:51,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342700
2023-11-23 07:14:55,079 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 07:15:00,591 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.089e+01 8.161e+01 8.758e+01 9.491e+01 1.345e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 07:15:08,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2284706.6666666665, ans=0.125
2023-11-23 07:15:14,051 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6050, loss[loss=0.06812, simple_loss=0.09296, pruned_loss=0.01285, audio_tagging_loss=0.008791, over 14412.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09342, pruned_loss=0.01433, audio_tagging_loss=0.009018, over 3040146.06 frames. ], batch size: 53, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:15:34,784 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.83 vs. limit=10.0
2023-11-23 07:15:55,594 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342750
2023-11-23 07:16:14,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2285040.0, ans=0.0
2023-11-23 07:16:17,508 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6100, loss[loss=0.06354, simple_loss=0.08711, pruned_loss=0.01032, audio_tagging_loss=0.009663, over 14918.00 frames. ], tot_loss[loss=0.07008, simple_loss=0.09364, pruned_loss=0.01424, audio_tagging_loss=0.00902, over 3043665.28 frames. ], batch size: 52, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:16:26,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2285106.6666666665, ans=0.125
2023-11-23 07:16:34,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2285173.3333333335, ans=0.2
2023-11-23 07:16:59,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342800
2023-11-23 07:17:03,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2285306.6666666665, ans=0.125
2023-11-23 07:17:04,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2285306.6666666665, ans=0.125
2023-11-23 07:17:08,162 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 8.337e+01 9.020e+01 9.718e+01 1.256e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 07:17:23,756 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6150, loss[loss=0.06209, simple_loss=0.07677, pruned_loss=0.01377, audio_tagging_loss=0.009935, over 16291.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09305, pruned_loss=0.01421, audio_tagging_loss=0.009105, over 3044655.25 frames. ], batch size: 62, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:17:36,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2285506.6666666665, ans=0.0
2023-11-23 07:17:44,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2285506.6666666665, ans=0.0
2023-11-23 07:17:56,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2285573.3333333335, ans=0.1
2023-11-23 07:17:57,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2285573.3333333335, ans=0.125
2023-11-23 07:18:02,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2285640.0, ans=0.2
2023-11-23 07:18:05,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342850
2023-11-23 07:18:11,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2285640.0, ans=0.125
2023-11-23 07:18:17,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2285706.6666666665, ans=0.95
2023-11-23 07:18:22,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2285706.6666666665, ans=0.0
2023-11-23 07:18:28,763 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6200, loss[loss=0.05285, simple_loss=0.07427, pruned_loss=0.008075, audio_tagging_loss=0.007635, over 14033.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.0922, pruned_loss=0.01399, audio_tagging_loss=0.009236, over 3042724.98 frames. ], batch size: 53, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:19:07,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2285973.3333333335, ans=0.1
2023-11-23 07:19:09,328 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342900
2023-11-23 07:19:16,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2285973.3333333335, ans=0.0
2023-11-23 07:19:19,542 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.777e+01 8.276e+01 8.850e+01 9.566e+01 1.179e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 07:19:31,990 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6250, loss[loss=0.07243, simple_loss=0.09243, pruned_loss=0.01357, audio_tagging_loss=0.01264, over 15848.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09213, pruned_loss=0.01412, audio_tagging_loss=0.009301, over 3047315.50 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:19:42,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2286106.6666666665, ans=0.125
2023-11-23 07:20:04,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2286240.0, ans=0.125
2023-11-23 07:20:13,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2286306.6666666665, ans=0.0
2023-11-23 07:20:14,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 342950
2023-11-23 07:20:31,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=2286373.3333333335, ans=0.1
2023-11-23 07:20:37,768 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6300, loss[loss=0.06275, simple_loss=0.08495, pruned_loss=0.01074, audio_tagging_loss=0.00953, over 15658.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.0923, pruned_loss=0.01418, audio_tagging_loss=0.009337, over 3039311.16 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:21:14,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2286640.0, ans=0.125
2023-11-23 07:21:18,285 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343000
2023-11-23 07:21:29,490 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.995e+01 8.227e+01 8.680e+01 9.495e+01 1.255e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-23 07:21:39,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2286706.6666666665, ans=0.0
2023-11-23 07:21:42,429 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6350, loss[loss=0.06933, simple_loss=0.08453, pruned_loss=0.01782, audio_tagging_loss=0.009249, over 14541.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09157, pruned_loss=0.01408, audio_tagging_loss=0.009376, over 3047042.14 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:21:52,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2286773.3333333335, ans=0.125
2023-11-23 07:22:23,565 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343050
2023-11-23 07:22:44,481 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.37 vs. limit=12.0
2023-11-23 07:22:46,274 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6400, loss[loss=0.06971, simple_loss=0.08853, pruned_loss=0.01619, audio_tagging_loss=0.009258, over 15434.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09138, pruned_loss=0.01396, audio_tagging_loss=0.00947, over 3045653.73 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:23:20,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2287240.0, ans=0.1
2023-11-23 07:23:20,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2287240.0, ans=0.125
2023-11-23 07:23:23,910 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.21 vs. limit=10.0
2023-11-23 07:23:28,281 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343100
2023-11-23 07:23:37,926 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 8.257e+01 8.871e+01 9.376e+01 1.262e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 07:23:39,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2287373.3333333335, ans=0.0
2023-11-23 07:23:47,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2287373.3333333335, ans=0.1
2023-11-23 07:23:51,288 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6450, loss[loss=0.08567, simple_loss=0.1161, pruned_loss=0.02069, audio_tagging_loss=0.006919, over 16752.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09115, pruned_loss=0.01405, audio_tagging_loss=0.009465, over 3040094.64 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:23:54,988 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.57 vs. limit=12.0
2023-11-23 07:23:59,610 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:24:23,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2287573.3333333335, ans=0.2
2023-11-23 07:24:29,322 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.84 vs. limit=15.0
2023-11-23 07:24:32,184 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343150
2023-11-23 07:24:55,838 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6500, loss[loss=0.07115, simple_loss=0.1066, pruned_loss=0.0114, audio_tagging_loss=0.006433, over 16297.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09117, pruned_loss=0.01387, audio_tagging_loss=0.009477, over 3046130.22 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:25:23,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2287906.6666666665, ans=0.125
2023-11-23 07:25:37,692 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343200
2023-11-23 07:25:46,097 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.53 vs. limit=15.0
2023-11-23 07:25:47,698 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.151e+01 8.781e+01 9.682e+01 1.216e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-23 07:25:52,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2288040.0, ans=0.125
2023-11-23 07:26:00,755 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6550, loss[loss=0.0735, simple_loss=0.1025, pruned_loss=0.01324, audio_tagging_loss=0.009017, over 15027.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09094, pruned_loss=0.01377, audio_tagging_loss=0.009413, over 3048473.92 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:26:01,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2288106.6666666665, ans=0.125
2023-11-23 07:26:06,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2288106.6666666665, ans=0.125
2023-11-23 07:26:09,171 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.99 vs. limit=10.0
2023-11-23 07:26:19,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2288173.3333333335, ans=0.0
2023-11-23 07:26:33,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2288240.0, ans=0.0
2023-11-23 07:26:42,176 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343250
2023-11-23 07:26:57,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2288373.3333333335, ans=0.125
2023-11-23 07:27:01,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2288373.3333333335, ans=0.5
2023-11-23 07:27:05,403 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6600, loss[loss=0.06139, simple_loss=0.08477, pruned_loss=0.01284, audio_tagging_loss=0.006168, over 14574.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09092, pruned_loss=0.01389, audio_tagging_loss=0.009321, over 3039155.51 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:27:46,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343300
2023-11-23 07:27:59,922 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.340e+01 9.003e+01 9.613e+01 1.234e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 07:28:05,582 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.45 vs. limit=15.0
2023-11-23 07:28:11,241 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6650, loss[loss=0.0663, simple_loss=0.0908, pruned_loss=0.01206, audio_tagging_loss=0.00884, over 16239.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09118, pruned_loss=0.01397, audio_tagging_loss=0.009205, over 3039606.32 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:28:18,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2288773.3333333335, ans=0.125
2023-11-23 07:28:31,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2288840.0, ans=0.125
2023-11-23 07:28:44,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2288906.6666666665, ans=0.125
2023-11-23 07:28:52,946 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343350
2023-11-23 07:28:54,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2288973.3333333335, ans=0.2
2023-11-23 07:29:14,902 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6700, loss[loss=0.05208, simple_loss=0.06714, pruned_loss=0.01058, audio_tagging_loss=0.007935, over 14710.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09078, pruned_loss=0.01383, audio_tagging_loss=0.009166, over 3039766.98 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:29:25,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2289106.6666666665, ans=0.125
2023-11-23 07:29:56,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343400
2023-11-23 07:30:07,555 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.148e+01 8.906e+01 9.681e+01 1.677e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 07:30:08,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2289373.3333333335, ans=0.1
2023-11-23 07:30:16,319 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.69 vs. limit=10.0
2023-11-23 07:30:19,377 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6750, loss[loss=0.08416, simple_loss=0.1026, pruned_loss=0.0231, audio_tagging_loss=0.009785, over 15599.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09176, pruned_loss=0.01406, audio_tagging_loss=0.009141, over 3033627.24 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:30:35,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2289506.6666666665, ans=15.0
2023-11-23 07:30:36,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2289506.6666666665, ans=0.0
2023-11-23 07:30:41,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2289506.6666666665, ans=0.04949747468305833
2023-11-23 07:30:43,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2289506.6666666665, ans=0.1
2023-11-23 07:30:50,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2289573.3333333335, ans=10.0
2023-11-23 07:30:53,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2289573.3333333335, ans=0.125
2023-11-23 07:30:59,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2289640.0, ans=0.0
2023-11-23 07:31:00,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343450
2023-11-23 07:31:24,902 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6800, loss[loss=0.06492, simple_loss=0.0867, pruned_loss=0.01325, audio_tagging_loss=0.00832, over 16045.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09174, pruned_loss=0.01407, audio_tagging_loss=0.009078, over 3036197.34 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:31:49,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2289906.6666666665, ans=0.125
2023-11-23 07:31:52,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2289906.6666666665, ans=0.125
2023-11-23 07:32:00,840 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:32:06,725 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343500
2023-11-23 07:32:08,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2289973.3333333335, ans=0.1
2023-11-23 07:32:10,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2289973.3333333335, ans=0.07
2023-11-23 07:32:17,720 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.151e+01 8.874e+01 9.532e+01 1.488e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 07:32:28,868 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6850, loss[loss=0.09764, simple_loss=0.1283, pruned_loss=0.02591, audio_tagging_loss=0.007585, over 14430.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09224, pruned_loss=0.01409, audio_tagging_loss=0.008937, over 3040123.47 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:32:51,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2290173.3333333335, ans=0.125
2023-11-23 07:32:56,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2290240.0, ans=0.0
2023-11-23 07:32:57,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2290240.0, ans=0.95
2023-11-23 07:33:00,079 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:33:08,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2290306.6666666665, ans=0.1
2023-11-23 07:33:11,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343550
2023-11-23 07:33:33,167 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6900, loss[loss=0.06335, simple_loss=0.08815, pruned_loss=0.01124, audio_tagging_loss=0.008034, over 15031.00 frames. ], tot_loss[loss=0.06978, simple_loss=0.09324, pruned_loss=0.01427, audio_tagging_loss=0.008884, over 3041950.73 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:33:57,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2290506.6666666665, ans=0.015
2023-11-23 07:34:06,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2290573.3333333335, ans=0.0
2023-11-23 07:34:14,562 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343600
2023-11-23 07:34:18,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2290640.0, ans=0.1
2023-11-23 07:34:22,083 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 07:34:26,250 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.590e+01 8.208e+01 8.939e+01 9.660e+01 1.242e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 07:34:28,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2290706.6666666665, ans=0.1
2023-11-23 07:34:39,259 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 6950, loss[loss=0.04654, simple_loss=0.04961, pruned_loss=0.007773, audio_tagging_loss=0.01397, over 14787.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09289, pruned_loss=0.01417, audio_tagging_loss=0.008985, over 3045043.13 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:34:44,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2290773.3333333335, ans=0.0
2023-11-23 07:34:51,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2290840.0, ans=0.0
2023-11-23 07:34:55,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2290840.0, ans=0.0
2023-11-23 07:35:09,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2290906.6666666665, ans=0.0
2023-11-23 07:35:19,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343650
2023-11-23 07:35:20,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2290973.3333333335, ans=0.0
2023-11-23 07:35:42,398 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7000, loss[loss=0.05026, simple_loss=0.06487, pruned_loss=0.008584, audio_tagging_loss=0.009235, over 15544.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09301, pruned_loss=0.01421, audio_tagging_loss=0.009054, over 3050032.10 frames. ], batch size: 63, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:35:52,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2291106.6666666665, ans=0.1
2023-11-23 07:35:58,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2291173.3333333335, ans=0.125
2023-11-23 07:36:05,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2291173.3333333335, ans=0.0
2023-11-23 07:36:13,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2291240.0, ans=0.0
2023-11-23 07:36:19,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2291240.0, ans=0.2
2023-11-23 07:36:24,118 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343700
2023-11-23 07:36:26,925 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.82 vs. limit=15.0
2023-11-23 07:36:34,906 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.812e+01 8.255e+01 8.925e+01 9.735e+01 1.377e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 07:36:35,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2291373.3333333335, ans=0.125
2023-11-23 07:36:40,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2291373.3333333335, ans=0.125
2023-11-23 07:36:40,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.91 vs. limit=15.0
2023-11-23 07:36:43,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2291373.3333333335, ans=0.1
2023-11-23 07:36:45,869 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7050, loss[loss=0.08419, simple_loss=0.1139, pruned_loss=0.0206, audio_tagging_loss=0.006647, over 15697.00 frames. ], tot_loss[loss=0.0703, simple_loss=0.0938, pruned_loss=0.01433, audio_tagging_loss=0.009072, over 3045444.33 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:37:15,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2291573.3333333335, ans=0.07
2023-11-23 07:37:16,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2291573.3333333335, ans=0.0
2023-11-23 07:37:26,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2291640.0, ans=0.125
2023-11-23 07:37:26,817 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.77 vs. limit=15.0
2023-11-23 07:37:27,563 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343750
2023-11-23 07:37:28,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2291640.0, ans=0.125
2023-11-23 07:37:39,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=2291706.6666666665, ans=0.1
2023-11-23 07:37:51,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2291773.3333333335, ans=0.2
2023-11-23 07:37:52,042 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7100, loss[loss=0.06635, simple_loss=0.07836, pruned_loss=0.01759, audio_tagging_loss=0.009583, over 13995.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09313, pruned_loss=0.01426, audio_tagging_loss=0.009115, over 3046667.73 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:37:57,686 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.39 vs. limit=15.0
2023-11-23 07:38:29,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2291973.3333333335, ans=0.95
2023-11-23 07:38:32,215 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343800
2023-11-23 07:38:36,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2291973.3333333335, ans=0.0
2023-11-23 07:38:43,962 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.34 vs. limit=12.0
2023-11-23 07:38:45,445 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.319e+01 8.862e+01 9.684e+01 1.135e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 07:38:45,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2292040.0, ans=0.05
2023-11-23 07:38:50,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2292040.0, ans=0.0
2023-11-23 07:38:53,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2292040.0, ans=0.125
2023-11-23 07:38:54,782 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.54 vs. limit=22.5
2023-11-23 07:38:56,654 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7150, loss[loss=0.08821, simple_loss=0.1151, pruned_loss=0.02088, audio_tagging_loss=0.009792, over 15814.00 frames. ], tot_loss[loss=0.07067, simple_loss=0.09397, pruned_loss=0.01446, audio_tagging_loss=0.009217, over 3052536.75 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:39:01,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2292106.6666666665, ans=0.07
2023-11-23 07:39:05,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2292106.6666666665, ans=0.125
2023-11-23 07:39:14,739 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.79 vs. limit=10.0
2023-11-23 07:39:38,207 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343850
2023-11-23 07:39:41,164 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.88 vs. limit=10.0
2023-11-23 07:39:45,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2292306.6666666665, ans=0.125
2023-11-23 07:40:00,103 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7200, loss[loss=0.06571, simple_loss=0.08721, pruned_loss=0.014, audio_tagging_loss=0.008105, over 14532.00 frames. ], tot_loss[loss=0.07083, simple_loss=0.09413, pruned_loss=0.01447, audio_tagging_loss=0.009295, over 3056907.84 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:40:18,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2292506.6666666665, ans=0.0
2023-11-23 07:40:20,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2292506.6666666665, ans=0.0
2023-11-23 07:40:41,603 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343900
2023-11-23 07:40:52,453 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.883e+01 8.353e+01 8.830e+01 9.779e+01 1.772e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 07:40:54,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2292706.6666666665, ans=0.125
2023-11-23 07:41:05,597 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7250, loss[loss=0.09048, simple_loss=0.1168, pruned_loss=0.02291, audio_tagging_loss=0.009153, over 15923.00 frames. ], tot_loss[loss=0.07042, simple_loss=0.0934, pruned_loss=0.01427, audio_tagging_loss=0.009448, over 3053258.67 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:41:24,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2292840.0, ans=0.0
2023-11-23 07:41:45,481 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 343950
2023-11-23 07:41:45,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2292973.3333333335, ans=0.2
2023-11-23 07:41:46,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2292973.3333333335, ans=0.125
2023-11-23 07:41:51,245 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.07 vs. limit=15.0
2023-11-23 07:42:09,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2293106.6666666665, ans=0.1
2023-11-23 07:42:10,345 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7300, loss[loss=0.04809, simple_loss=0.05911, pruned_loss=0.008834, audio_tagging_loss=0.009706, over 14966.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09293, pruned_loss=0.01409, audio_tagging_loss=0.00929, over 3054031.90 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:42:37,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2293240.0, ans=0.0
2023-11-23 07:42:46,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2293240.0, ans=0.125
2023-11-23 07:42:50,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2293306.6666666665, ans=0.125
2023-11-23 07:42:51,457 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344000
2023-11-23 07:42:52,948 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-344000.pt
2023-11-23 07:43:04,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2293373.3333333335, ans=0.0
2023-11-23 07:43:06,586 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.353e+01 8.994e+01 9.658e+01 1.142e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 07:43:14,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2293373.3333333335, ans=0.125
2023-11-23 07:43:17,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2293440.0, ans=0.0
2023-11-23 07:43:17,908 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7350, loss[loss=0.07885, simple_loss=0.1145, pruned_loss=0.01418, audio_tagging_loss=0.00742, over 15974.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09268, pruned_loss=0.01405, audio_tagging_loss=0.009173, over 3047895.45 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:43:20,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2293440.0, ans=0.0
2023-11-23 07:43:24,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2293440.0, ans=0.125
2023-11-23 07:44:00,179 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344050
2023-11-23 07:44:00,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2293640.0, ans=0.0
2023-11-23 07:44:06,919 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.39 vs. limit=15.0
2023-11-23 07:44:07,204 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.94 vs. limit=15.0
2023-11-23 07:44:08,131 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.09 vs. limit=10.0
2023-11-23 07:44:16,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2293706.6666666665, ans=0.2
2023-11-23 07:44:23,617 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7400, loss[loss=0.05095, simple_loss=0.05147, pruned_loss=0.01092, audio_tagging_loss=0.0143, over 15132.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09259, pruned_loss=0.01398, audio_tagging_loss=0.009124, over 3045197.42 frames. ], batch size: 61, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:44:34,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2293773.3333333335, ans=0.1
2023-11-23 07:44:54,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2293906.6666666665, ans=0.1
2023-11-23 07:45:04,056 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344100
2023-11-23 07:45:15,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2294040.0, ans=0.0
2023-11-23 07:45:17,733 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.613e+01 8.228e+01 8.684e+01 9.636e+01 1.290e+02, threshold=1.737e+02, percent-clipped=0.0
2023-11-23 07:45:28,114 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7450, loss[loss=0.06602, simple_loss=0.08015, pruned_loss=0.012, audio_tagging_loss=0.01395, over 14312.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09331, pruned_loss=0.0141, audio_tagging_loss=0.009073, over 3045945.08 frames. ], batch size: 53, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:45:49,203 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:46:09,188 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344150
2023-11-23 07:46:12,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2294306.6666666665, ans=0.125
2023-11-23 07:46:12,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2294306.6666666665, ans=0.125
2023-11-23 07:46:28,521 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.66 vs. limit=22.5
2023-11-23 07:46:29,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2294373.3333333335, ans=0.0
2023-11-23 07:46:30,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2294440.0, ans=0.0
2023-11-23 07:46:31,663 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7500, loss[loss=0.06014, simple_loss=0.0801, pruned_loss=0.007954, audio_tagging_loss=0.01213, over 15001.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09238, pruned_loss=0.01401, audio_tagging_loss=0.008961, over 3044572.37 frames. ], batch size: 61, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:46:51,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2294506.6666666665, ans=0.125
2023-11-23 07:47:13,242 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344200
2023-11-23 07:47:16,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2294640.0, ans=0.0
2023-11-23 07:47:25,790 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.933e+01 8.286e+01 8.811e+01 9.468e+01 1.775e+02, threshold=1.762e+02, percent-clipped=1.0
2023-11-23 07:47:31,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.73 vs. limit=15.0
2023-11-23 07:47:36,015 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7550, loss[loss=0.0767, simple_loss=0.11, pruned_loss=0.01524, audio_tagging_loss=0.006471, over 16054.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09203, pruned_loss=0.01398, audio_tagging_loss=0.008983, over 3052321.16 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:47:44,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.28 vs. limit=22.5
2023-11-23 07:48:02,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2294906.6666666665, ans=0.125
2023-11-23 07:48:04,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2294906.6666666665, ans=0.2
2023-11-23 07:48:09,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2294906.6666666665, ans=0.0
2023-11-23 07:48:11,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2294906.6666666665, ans=0.125
2023-11-23 07:48:11,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2294906.6666666665, ans=0.125
2023-11-23 07:48:15,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2294973.3333333335, ans=0.07
2023-11-23 07:48:17,162 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344250
2023-11-23 07:48:30,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2295040.0, ans=0.125
2023-11-23 07:48:34,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.40 vs. limit=15.0
2023-11-23 07:48:41,118 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7600, loss[loss=0.06118, simple_loss=0.08078, pruned_loss=0.01141, audio_tagging_loss=0.009383, over 15703.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09212, pruned_loss=0.01399, audio_tagging_loss=0.009062, over 3052960.19 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:49:12,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2295240.0, ans=0.2
2023-11-23 07:49:15,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2295240.0, ans=0.2
2023-11-23 07:49:22,989 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344300
2023-11-23 07:49:25,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2295306.6666666665, ans=0.1
2023-11-23 07:49:35,027 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.607e+01 8.274e+01 8.656e+01 9.484e+01 1.340e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-23 07:49:45,378 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7650, loss[loss=0.04931, simple_loss=0.05154, pruned_loss=0.007837, audio_tagging_loss=0.0157, over 15313.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09076, pruned_loss=0.01386, audio_tagging_loss=0.009159, over 3046751.47 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:49:58,249 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.45 vs. limit=22.5
2023-11-23 07:50:11,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2295573.3333333335, ans=0.2
2023-11-23 07:50:19,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2295573.3333333335, ans=0.125
2023-11-23 07:50:19,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2295573.3333333335, ans=0.0
2023-11-23 07:50:23,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2295640.0, ans=0.125
2023-11-23 07:50:24,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2295640.0, ans=0.2
2023-11-23 07:50:26,579 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344350
2023-11-23 07:50:47,176 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0
2023-11-23 07:50:48,884 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7700, loss[loss=0.05026, simple_loss=0.06713, pruned_loss=0.007541, audio_tagging_loss=0.009149, over 14927.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09124, pruned_loss=0.0138, audio_tagging_loss=0.009108, over 3044367.19 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:51:00,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2295773.3333333335, ans=0.125
2023-11-23 07:51:01,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2295840.0, ans=10.0
2023-11-23 07:51:06,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2295840.0, ans=0.125
2023-11-23 07:51:23,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2295906.6666666665, ans=0.125
2023-11-23 07:51:26,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2295973.3333333335, ans=0.125
2023-11-23 07:51:30,231 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344400
2023-11-23 07:51:30,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2295973.3333333335, ans=0.125
2023-11-23 07:51:38,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2295973.3333333335, ans=0.125
2023-11-23 07:51:44,896 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.308e+01 8.349e+01 8.922e+01 9.569e+01 1.181e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 07:51:46,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2296040.0, ans=0.1
2023-11-23 07:51:49,750 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.05 vs. limit=15.0
2023-11-23 07:51:50,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2296040.0, ans=0.0
2023-11-23 07:51:54,188 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7750, loss[loss=0.07503, simple_loss=0.1051, pruned_loss=0.01391, audio_tagging_loss=0.00856, over 15147.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09219, pruned_loss=0.01388, audio_tagging_loss=0.009156, over 3046151.28 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:52:32,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2296306.6666666665, ans=0.0
2023-11-23 07:52:35,159 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344450
2023-11-23 07:52:40,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2296306.6666666665, ans=0.125
2023-11-23 07:52:51,704 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 07:52:57,558 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7800, loss[loss=0.0688, simple_loss=0.09367, pruned_loss=0.0161, audio_tagging_loss=0.005872, over 15457.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09195, pruned_loss=0.014, audio_tagging_loss=0.009133, over 3048239.54 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:53:23,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2296573.3333333335, ans=0.125
2023-11-23 07:53:27,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2296573.3333333335, ans=0.0
2023-11-23 07:53:32,741 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.19 vs. limit=22.5
2023-11-23 07:53:35,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2296640.0, ans=0.1
2023-11-23 07:53:39,383 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344500
2023-11-23 07:53:44,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2296640.0, ans=0.125
2023-11-23 07:53:48,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2296706.6666666665, ans=0.0
2023-11-23 07:53:53,392 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.147e+01 8.634e+01 9.399e+01 1.178e+02, threshold=1.727e+02, percent-clipped=0.0
2023-11-23 07:54:01,885 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7850, loss[loss=0.07479, simple_loss=0.09155, pruned_loss=0.01577, audio_tagging_loss=0.01325, over 15714.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09192, pruned_loss=0.01393, audio_tagging_loss=0.00924, over 3051805.90 frames. ], batch size: 60, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:54:13,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2296773.3333333335, ans=0.0
2023-11-23 07:54:24,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2296840.0, ans=0.0
2023-11-23 07:54:25,211 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.10 vs. limit=10.0
2023-11-23 07:54:33,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2296906.6666666665, ans=0.125
2023-11-23 07:54:43,724 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344550
2023-11-23 07:54:51,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2296973.3333333335, ans=0.0
2023-11-23 07:55:07,320 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7900, loss[loss=0.07259, simple_loss=0.1027, pruned_loss=0.01199, audio_tagging_loss=0.009259, over 15298.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09245, pruned_loss=0.01388, audio_tagging_loss=0.00929, over 3051639.72 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:55:09,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2297106.6666666665, ans=0.125
2023-11-23 07:55:09,345 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.43 vs. limit=12.0
2023-11-23 07:55:24,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2297173.3333333335, ans=0.0
2023-11-23 07:55:31,415 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0
2023-11-23 07:55:48,154 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344600
2023-11-23 07:56:03,005 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.370e+01 9.081e+01 9.662e+01 1.223e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 07:56:11,612 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 7950, loss[loss=0.0731, simple_loss=0.1018, pruned_loss=0.01251, audio_tagging_loss=0.009693, over 15842.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09202, pruned_loss=0.01361, audio_tagging_loss=0.009397, over 3052309.12 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 07:56:23,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2297506.6666666665, ans=0.0
2023-11-23 07:56:25,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2297506.6666666665, ans=0.125
2023-11-23 07:56:26,807 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 07:56:28,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.25 vs. limit=15.0
2023-11-23 07:56:34,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2297506.6666666665, ans=0.1
2023-11-23 07:56:52,890 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344650
2023-11-23 07:56:55,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2297640.0, ans=0.035
2023-11-23 07:56:55,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.80 vs. limit=15.0
2023-11-23 07:57:14,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2297773.3333333335, ans=0.025
2023-11-23 07:57:15,716 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8000, loss[loss=0.07158, simple_loss=0.1053, pruned_loss=0.01243, audio_tagging_loss=0.00651, over 14471.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09057, pruned_loss=0.01346, audio_tagging_loss=0.009576, over 3041744.91 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:57:37,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2297840.0, ans=0.1
2023-11-23 07:57:48,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2297906.6666666665, ans=0.125
2023-11-23 07:57:56,976 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344700
2023-11-23 07:58:00,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2297973.3333333335, ans=0.1
2023-11-23 07:58:04,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2297973.3333333335, ans=0.0
2023-11-23 07:58:06,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2298040.0, ans=0.0
2023-11-23 07:58:09,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2298040.0, ans=0.015
2023-11-23 07:58:10,964 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.163e+01 8.824e+01 9.486e+01 1.093e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 07:58:13,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=15.0
2023-11-23 07:58:21,512 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8050, loss[loss=0.0628, simple_loss=0.07999, pruned_loss=0.01239, audio_tagging_loss=0.01042, over 15297.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09107, pruned_loss=0.01362, audio_tagging_loss=0.00952, over 3038515.74 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:58:25,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2298106.6666666665, ans=0.125
2023-11-23 07:58:30,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2298106.6666666665, ans=0.125
2023-11-23 07:58:44,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2298173.3333333335, ans=0.0
2023-11-23 07:58:51,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2298240.0, ans=0.0
2023-11-23 07:58:53,561 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.59 vs. limit=15.0
2023-11-23 07:58:56,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2298240.0, ans=0.125
2023-11-23 07:59:02,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344750
2023-11-23 07:59:20,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2298373.3333333335, ans=0.0
2023-11-23 07:59:25,550 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8100, loss[loss=0.08669, simple_loss=0.1135, pruned_loss=0.02135, audio_tagging_loss=0.008606, over 15219.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09182, pruned_loss=0.01384, audio_tagging_loss=0.009512, over 3043133.95 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 07:59:35,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2298440.0, ans=0.125
2023-11-23 07:59:37,445 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.15 vs. limit=10.0
2023-11-23 07:59:41,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=2298506.6666666665, ans=0.05
2023-11-23 07:59:48,902 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.14 vs. limit=22.5
2023-11-23 08:00:05,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2298640.0, ans=0.125
2023-11-23 08:00:07,352 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344800
2023-11-23 08:00:15,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff3.min_abs, batch_count=2298640.0, ans=0.2
2023-11-23 08:00:22,312 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.280e+01 8.423e+01 8.748e+01 9.445e+01 1.319e+02, threshold=1.750e+02, percent-clipped=0.0
2023-11-23 08:00:29,757 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8150, loss[loss=0.06038, simple_loss=0.08477, pruned_loss=0.01148, audio_tagging_loss=0.006515, over 15482.00 frames. ], tot_loss[loss=0.06993, simple_loss=0.0933, pruned_loss=0.01411, audio_tagging_loss=0.009162, over 3045579.31 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:00:54,549 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.07 vs. limit=15.0
2023-11-23 08:00:54,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.61 vs. limit=15.0
2023-11-23 08:00:55,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=2298906.6666666665, ans=0.05
2023-11-23 08:00:58,475 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.12 vs. limit=22.5
2023-11-23 08:01:09,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2298973.3333333335, ans=0.1
2023-11-23 08:01:11,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344850
2023-11-23 08:01:20,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2299040.0, ans=0.125
2023-11-23 08:01:27,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2299040.0, ans=0.125
2023-11-23 08:01:34,469 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8200, loss[loss=0.07048, simple_loss=0.0951, pruned_loss=0.01375, audio_tagging_loss=0.00918, over 14242.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09365, pruned_loss=0.01408, audio_tagging_loss=0.009094, over 3049949.51 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:01:34,532 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:01:34,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2299106.6666666665, ans=0.0
2023-11-23 08:01:47,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2299173.3333333335, ans=0.125
2023-11-23 08:02:09,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2299240.0, ans=0.125
2023-11-23 08:02:14,524 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344900
2023-11-23 08:02:31,499 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.892e+01 8.311e+01 8.960e+01 9.864e+01 1.172e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 08:02:39,026 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8250, loss[loss=0.05321, simple_loss=0.06393, pruned_loss=0.01079, audio_tagging_loss=0.01045, over 15887.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.0943, pruned_loss=0.01426, audio_tagging_loss=0.009059, over 3049560.39 frames. ], batch size: 61, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:03:05,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2299573.3333333335, ans=0.07
2023-11-23 08:03:21,203 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 344950
2023-11-23 08:03:21,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2299640.0, ans=0.125
2023-11-23 08:03:42,990 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8300, loss[loss=0.08274, simple_loss=0.1163, pruned_loss=0.01623, audio_tagging_loss=0.008382, over 15686.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09456, pruned_loss=0.01419, audio_tagging_loss=0.009067, over 3060157.27 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:03:48,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2299773.3333333335, ans=0.0
2023-11-23 08:04:10,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2299906.6666666665, ans=0.125
2023-11-23 08:04:10,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2299906.6666666665, ans=0.1
2023-11-23 08:04:18,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2299906.6666666665, ans=0.0
2023-11-23 08:04:18,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2299906.6666666665, ans=0.1
2023-11-23 08:04:23,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345000
2023-11-23 08:04:34,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2300040.0, ans=0.125
2023-11-23 08:04:38,781 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.704e+01 8.216e+01 8.783e+01 9.423e+01 1.176e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-23 08:04:46,573 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8350, loss[loss=0.0484, simple_loss=0.05533, pruned_loss=0.009531, audio_tagging_loss=0.0112, over 15132.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09317, pruned_loss=0.01404, audio_tagging_loss=0.009142, over 3049890.91 frames. ], batch size: 58, lr: 2.34e-03, grad_scale: 16.0
2023-11-23 08:05:04,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2300173.3333333335, ans=0.1
2023-11-23 08:05:27,870 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345050
2023-11-23 08:05:32,164 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.40 vs. limit=15.0
2023-11-23 08:05:45,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2300373.3333333335, ans=0.125
2023-11-23 08:05:51,647 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8400, loss[loss=0.06768, simple_loss=0.09255, pruned_loss=0.01235, audio_tagging_loss=0.009057, over 15019.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09275, pruned_loss=0.01411, audio_tagging_loss=0.009175, over 3051645.37 frames. ], batch size: 55, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:06:05,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2300506.6666666665, ans=0.125
2023-11-23 08:06:08,687 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.56 vs. limit=15.0
2023-11-23 08:06:16,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2300573.3333333335, ans=0.125
2023-11-23 08:06:23,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2300573.3333333335, ans=0.125
2023-11-23 08:06:28,159 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.35 vs. limit=15.0
2023-11-23 08:06:29,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2300640.0, ans=0.0
2023-11-23 08:06:33,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345100
2023-11-23 08:06:47,799 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.105e+01 8.014e+01 8.846e+01 9.500e+01 1.669e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 08:06:48,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2300706.6666666665, ans=0.95
2023-11-23 08:06:50,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2300706.6666666665, ans=0.125
2023-11-23 08:06:55,295 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8450, loss[loss=0.06767, simple_loss=0.09702, pruned_loss=0.009919, audio_tagging_loss=0.009245, over 15203.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09295, pruned_loss=0.01421, audio_tagging_loss=0.009136, over 3054951.79 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:07:36,242 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345150
2023-11-23 08:07:37,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2300973.3333333335, ans=0.2
2023-11-23 08:07:40,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2300973.3333333335, ans=0.2
2023-11-23 08:07:54,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2301040.0, ans=0.125
2023-11-23 08:07:58,713 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8500, loss[loss=0.06791, simple_loss=0.09065, pruned_loss=0.01222, audio_tagging_loss=0.01037, over 15335.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09413, pruned_loss=0.01434, audio_tagging_loss=0.009063, over 3055144.05 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:08:31,244 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.84 vs. limit=6.0
2023-11-23 08:08:33,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2301240.0, ans=10.0
2023-11-23 08:08:39,457 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345200
2023-11-23 08:08:55,830 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.021e+01 8.259e+01 8.832e+01 9.401e+01 1.675e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 08:09:03,825 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8550, loss[loss=0.0674, simple_loss=0.09301, pruned_loss=0.01288, audio_tagging_loss=0.008012, over 15066.00 frames. ], tot_loss[loss=0.07038, simple_loss=0.09386, pruned_loss=0.01433, audio_tagging_loss=0.009125, over 3049909.78 frames. ], batch size: 54, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:09:16,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2301506.6666666665, ans=0.125
2023-11-23 08:09:37,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2301573.3333333335, ans=0.125
2023-11-23 08:09:44,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345250
2023-11-23 08:10:00,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2301706.6666666665, ans=0.0
2023-11-23 08:10:02,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2301706.6666666665, ans=0.09899494936611666
2023-11-23 08:10:07,669 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8600, loss[loss=0.05869, simple_loss=0.07719, pruned_loss=0.009986, audio_tagging_loss=0.0101, over 15371.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09254, pruned_loss=0.01402, audio_tagging_loss=0.00924, over 3049019.92 frames. ], batch size: 59, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:10:19,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2301840.0, ans=0.1
2023-11-23 08:10:49,396 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345300
2023-11-23 08:10:49,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2301973.3333333335, ans=0.0
2023-11-23 08:10:58,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2302040.0, ans=0.0
2023-11-23 08:11:03,888 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.822e+01 8.320e+01 8.873e+01 9.523e+01 1.247e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 08:11:11,303 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8650, loss[loss=0.09123, simple_loss=0.1209, pruned_loss=0.02262, audio_tagging_loss=0.008171, over 15250.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.0935, pruned_loss=0.01421, audio_tagging_loss=0.009212, over 3050354.66 frames. ], batch size: 56, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:11:35,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2302173.3333333335, ans=0.125
2023-11-23 08:11:50,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2302306.6666666665, ans=0.07
2023-11-23 08:11:52,593 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345350
2023-11-23 08:12:07,849 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.77 vs. limit=10.0
2023-11-23 08:12:09,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2302373.3333333335, ans=0.0
2023-11-23 08:12:09,745 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.52 vs. limit=22.5
2023-11-23 08:12:15,896 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.59 vs. limit=10.0
2023-11-23 08:12:16,469 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8700, loss[loss=0.08205, simple_loss=0.1112, pruned_loss=0.01662, audio_tagging_loss=0.009835, over 15564.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09388, pruned_loss=0.01423, audio_tagging_loss=0.009274, over 3052301.18 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:12:16,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2302440.0, ans=0.125
2023-11-23 08:12:44,291 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.12 vs. limit=12.0
2023-11-23 08:12:50,831 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.82 vs. limit=22.5
2023-11-23 08:12:54,840 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.92 vs. limit=22.5
2023-11-23 08:12:57,893 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345400
2023-11-23 08:13:07,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2302706.6666666665, ans=0.125
2023-11-23 08:13:13,363 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.372e+01 9.197e+01 9.763e+01 1.279e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 08:13:18,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2302706.6666666665, ans=0.125
2023-11-23 08:13:20,777 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8750, loss[loss=0.06862, simple_loss=0.09937, pruned_loss=0.01158, audio_tagging_loss=0.007353, over 15787.00 frames. ], tot_loss[loss=0.07136, simple_loss=0.09486, pruned_loss=0.01467, audio_tagging_loss=0.00926, over 3056061.58 frames. ], batch size: 57, lr: 2.34e-03, grad_scale: 32.0
2023-11-23 08:13:27,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2302773.3333333335, ans=0.0
2023-11-23 08:13:28,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2302773.3333333335, ans=0.125
2023-11-23 08:13:29,966 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0
2023-11-23 08:13:31,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2302840.0, ans=0.0
2023-11-23 08:13:34,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2302840.0, ans=0.2
2023-11-23 08:14:01,404 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345450
2023-11-23 08:14:04,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2302973.3333333335, ans=0.0
2023-11-23 08:14:13,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2303040.0, ans=0.0
2023-11-23 08:14:24,235 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8800, loss[loss=0.07386, simple_loss=0.09974, pruned_loss=0.0131, audio_tagging_loss=0.01089, over 16306.00 frames. ], tot_loss[loss=0.07125, simple_loss=0.09466, pruned_loss=0.01456, audio_tagging_loss=0.009349, over 3052607.99 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:14:38,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2303173.3333333335, ans=0.1
2023-11-23 08:14:51,471 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.17 vs. limit=15.0
2023-11-23 08:14:58,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2303240.0, ans=0.0
2023-11-23 08:15:01,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2303306.6666666665, ans=0.125
2023-11-23 08:15:03,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2303306.6666666665, ans=0.125
2023-11-23 08:15:04,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345500
2023-11-23 08:15:07,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2303306.6666666665, ans=0.125
2023-11-23 08:15:20,175 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.259e+01 8.339e+01 9.015e+01 9.606e+01 1.190e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 08:15:27,864 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.36 vs. limit=15.0
2023-11-23 08:15:28,110 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8850, loss[loss=0.04235, simple_loss=0.04705, pruned_loss=0.009037, audio_tagging_loss=0.009792, over 14776.00 frames. ], tot_loss[loss=0.07074, simple_loss=0.09371, pruned_loss=0.01442, audio_tagging_loss=0.009466, over 3049453.22 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:15:29,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2303440.0, ans=0.125
2023-11-23 08:15:34,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2303440.0, ans=0.125
2023-11-23 08:15:39,113 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:15:46,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2303506.6666666665, ans=0.125
2023-11-23 08:15:59,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2303573.3333333335, ans=0.07
2023-11-23 08:16:03,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2303573.3333333335, ans=0.1
2023-11-23 08:16:04,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2303640.0, ans=0.125
2023-11-23 08:16:08,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345550
2023-11-23 08:16:31,260 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8900, loss[loss=0.08816, simple_loss=0.1181, pruned_loss=0.019, audio_tagging_loss=0.01013, over 15893.00 frames. ], tot_loss[loss=0.07092, simple_loss=0.09375, pruned_loss=0.01468, audio_tagging_loss=0.009369, over 3051639.46 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:16:37,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2303773.3333333335, ans=0.1
2023-11-23 08:16:42,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2303840.0, ans=0.125
2023-11-23 08:17:01,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2303906.6666666665, ans=0.2
2023-11-23 08:17:09,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2303973.3333333335, ans=0.0
2023-11-23 08:17:11,891 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345600
2023-11-23 08:17:16,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2303973.3333333335, ans=0.07
2023-11-23 08:17:27,490 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.364e+01 9.019e+01 9.831e+01 1.164e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 08:17:34,817 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 8950, loss[loss=0.05564, simple_loss=0.07358, pruned_loss=0.0107, audio_tagging_loss=0.008152, over 14484.00 frames. ], tot_loss[loss=0.07108, simple_loss=0.09433, pruned_loss=0.01477, audio_tagging_loss=0.009143, over 3046098.03 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:17:58,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2304173.3333333335, ans=0.125
2023-11-23 08:18:16,378 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345650
2023-11-23 08:18:23,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2304306.6666666665, ans=0.2
2023-11-23 08:18:40,006 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9000, loss[loss=0.06678, simple_loss=0.08029, pruned_loss=0.01417, audio_tagging_loss=0.01247, over 16328.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.09387, pruned_loss=0.01468, audio_tagging_loss=0.009114, over 3045189.88 frames. ], batch size: 62, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:18:40,009 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 08:19:23,265 INFO [train_asr.py:1253] (0/4) Epoch 29, validation: loss=0.05895, simple_loss=0.05118, pruned_loss=0.005121, audio_tagging_loss=0.02824, over 4681554.00 frames. 
2023-11-23 08:19:23,266 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 08:19:41,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2304506.6666666665, ans=0.125
2023-11-23 08:19:58,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2304573.3333333335, ans=0.125
2023-11-23 08:20:04,375 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345700
2023-11-23 08:20:06,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.86 vs. limit=10.0
2023-11-23 08:20:07,281 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.45 vs. limit=15.0
2023-11-23 08:20:09,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2304640.0, ans=0.125
2023-11-23 08:20:19,507 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.885e+01 8.225e+01 9.124e+01 9.733e+01 1.162e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 08:20:22,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2304706.6666666665, ans=0.125
2023-11-23 08:20:27,048 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9050, loss[loss=0.05856, simple_loss=0.07914, pruned_loss=0.01101, audio_tagging_loss=0.007985, over 15085.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09289, pruned_loss=0.0143, audio_tagging_loss=0.009118, over 3044115.19 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:20:52,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2304906.6666666665, ans=0.125
2023-11-23 08:20:56,149 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.59 vs. limit=15.0
2023-11-23 08:21:08,198 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345750
2023-11-23 08:21:19,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2305040.0, ans=0.2
2023-11-23 08:21:31,419 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9100, loss[loss=0.07406, simple_loss=0.1072, pruned_loss=0.01488, audio_tagging_loss=0.005582, over 15683.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09241, pruned_loss=0.01424, audio_tagging_loss=0.009064, over 3043551.11 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:21:36,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2305106.6666666665, ans=0.0
2023-11-23 08:21:59,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2305240.0, ans=0.07
2023-11-23 08:22:12,302 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345800
2023-11-23 08:22:28,186 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.178e+01 8.924e+01 9.840e+01 1.213e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 08:22:34,160 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9150, loss[loss=0.04845, simple_loss=0.05839, pruned_loss=0.007194, audio_tagging_loss=0.01206, over 14538.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09206, pruned_loss=0.01422, audio_tagging_loss=0.009022, over 3043564.10 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:22:34,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2305440.0, ans=0.0
2023-11-23 08:22:45,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=2305506.6666666665, ans=0.1
2023-11-23 08:23:15,363 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345850
2023-11-23 08:23:22,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2305640.0, ans=0.0
2023-11-23 08:23:37,719 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9200, loss[loss=0.06733, simple_loss=0.08803, pruned_loss=0.01548, audio_tagging_loss=0.007829, over 14550.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09203, pruned_loss=0.01414, audio_tagging_loss=0.008958, over 3043022.67 frames. ], batch size: 54, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:23:43,009 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:23:44,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2305773.3333333335, ans=0.125
2023-11-23 08:23:54,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2305840.0, ans=0.1
2023-11-23 08:24:18,696 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345900
2023-11-23 08:24:29,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2306040.0, ans=0.125
2023-11-23 08:24:34,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2306040.0, ans=0.125
2023-11-23 08:24:35,642 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.303e+01 8.816e+01 9.419e+01 1.160e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 08:24:40,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2306040.0, ans=0.125
2023-11-23 08:24:42,461 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9250, loss[loss=0.05745, simple_loss=0.07581, pruned_loss=0.009899, audio_tagging_loss=0.009649, over 15299.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09275, pruned_loss=0.01431, audio_tagging_loss=0.009005, over 3041124.83 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:24:47,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2306106.6666666665, ans=0.125
2023-11-23 08:24:47,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2306106.6666666665, ans=0.125
2023-11-23 08:25:05,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2306240.0, ans=0.0
2023-11-23 08:25:06,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2306240.0, ans=0.125
2023-11-23 08:25:07,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2306240.0, ans=0.125
2023-11-23 08:25:22,703 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 345950
2023-11-23 08:25:22,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2306306.6666666665, ans=0.125
2023-11-23 08:25:34,284 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:25:40,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2306373.3333333335, ans=0.125
2023-11-23 08:25:44,891 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9300, loss[loss=0.06527, simple_loss=0.08946, pruned_loss=0.01238, audio_tagging_loss=0.008163, over 14785.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09179, pruned_loss=0.01399, audio_tagging_loss=0.009107, over 3041633.70 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:26:23,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2306640.0, ans=0.125
2023-11-23 08:26:25,334 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.16 vs. limit=10.0
2023-11-23 08:26:25,851 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346000
2023-11-23 08:26:27,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2306640.0, ans=0.07
2023-11-23 08:26:39,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2306706.6666666665, ans=0.0
2023-11-23 08:26:41,829 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.561e+01 8.507e+01 8.976e+01 9.625e+01 1.274e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-23 08:26:48,006 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9350, loss[loss=0.08818, simple_loss=0.1235, pruned_loss=0.01965, audio_tagging_loss=0.006779, over 14706.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09209, pruned_loss=0.01414, audio_tagging_loss=0.009173, over 3044249.59 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:27:08,814 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.88 vs. limit=15.0
2023-11-23 08:27:16,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2306906.6666666665, ans=0.125
2023-11-23 08:27:20,033 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.06 vs. limit=12.0
2023-11-23 08:27:28,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2306973.3333333335, ans=0.1
2023-11-23 08:27:29,176 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346050
2023-11-23 08:27:31,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2306973.3333333335, ans=0.125
2023-11-23 08:27:34,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2306973.3333333335, ans=0.2
2023-11-23 08:27:53,083 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9400, loss[loss=0.07508, simple_loss=0.09143, pruned_loss=0.018, audio_tagging_loss=0.01137, over 16475.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09131, pruned_loss=0.01405, audio_tagging_loss=0.009294, over 3042297.44 frames. ], batch size: 62, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:27:55,034 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.73 vs. limit=15.0
2023-11-23 08:28:08,193 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.00 vs. limit=15.0
2023-11-23 08:28:12,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2307173.3333333335, ans=0.125
2023-11-23 08:28:25,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2307240.0, ans=0.0
2023-11-23 08:28:33,011 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346100
2023-11-23 08:28:39,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0
2023-11-23 08:28:43,833 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.08 vs. limit=6.0
2023-11-23 08:28:51,280 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.981e+01 8.336e+01 8.834e+01 9.625e+01 1.227e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 08:28:53,804 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:28:56,218 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9450, loss[loss=0.05263, simple_loss=0.07373, pruned_loss=0.006515, audio_tagging_loss=0.009246, over 15082.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09225, pruned_loss=0.01422, audio_tagging_loss=0.009293, over 3047876.63 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:29:07,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.90 vs. limit=15.0
2023-11-23 08:29:08,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2307506.6666666665, ans=0.95
2023-11-23 08:29:09,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2307506.6666666665, ans=0.125
2023-11-23 08:29:11,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2307506.6666666665, ans=0.2
2023-11-23 08:29:32,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.27 vs. limit=22.5
2023-11-23 08:29:36,708 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346150
2023-11-23 08:29:42,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2307640.0, ans=0.0
2023-11-23 08:29:58,755 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9500, loss[loss=0.07322, simple_loss=0.1031, pruned_loss=0.01414, audio_tagging_loss=0.007514, over 15394.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09193, pruned_loss=0.01412, audio_tagging_loss=0.009418, over 3049314.06 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:29:58,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2307773.3333333335, ans=0.125
2023-11-23 08:30:04,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2307773.3333333335, ans=0.2
2023-11-23 08:30:10,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2307840.0, ans=0.0
2023-11-23 08:30:20,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2307840.0, ans=0.125
2023-11-23 08:30:22,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2307840.0, ans=0.1
2023-11-23 08:30:31,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2307906.6666666665, ans=0.04949747468305833
2023-11-23 08:30:39,544 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346200
2023-11-23 08:30:56,978 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.410e+01 8.909e+01 9.826e+01 1.512e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 08:31:03,149 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9550, loss[loss=0.06981, simple_loss=0.1031, pruned_loss=0.0106, audio_tagging_loss=0.00767, over 14504.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09217, pruned_loss=0.01402, audio_tagging_loss=0.009407, over 3043872.92 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:31:08,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2308106.6666666665, ans=0.1
2023-11-23 08:31:14,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.07 vs. limit=15.0
2023-11-23 08:31:43,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346250
2023-11-23 08:32:07,624 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9600, loss[loss=0.07277, simple_loss=0.1043, pruned_loss=0.01503, audio_tagging_loss=0.005583, over 16391.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.0912, pruned_loss=0.01399, audio_tagging_loss=0.009457, over 3046833.81 frames. ], batch size: 61, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:32:09,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2308440.0, ans=0.0
2023-11-23 08:32:27,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.98 vs. limit=10.0
2023-11-23 08:32:30,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2308506.6666666665, ans=0.0
2023-11-23 08:32:48,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2308640.0, ans=0.0
2023-11-23 08:32:49,784 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346300
2023-11-23 08:33:06,935 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.195e+01 8.302e+01 9.179e+01 9.827e+01 1.321e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-23 08:33:09,116 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.66 vs. limit=10.0
2023-11-23 08:33:09,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2308706.6666666665, ans=0.0
2023-11-23 08:33:11,999 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9650, loss[loss=0.06008, simple_loss=0.07657, pruned_loss=0.01184, audio_tagging_loss=0.00995, over 15271.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09141, pruned_loss=0.01406, audio_tagging_loss=0.009416, over 3035837.81 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:33:15,861 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:33:19,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2308773.3333333335, ans=0.125
2023-11-23 08:33:40,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2308906.6666666665, ans=0.125
2023-11-23 08:33:46,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.57 vs. limit=10.0
2023-11-23 08:33:53,653 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346350
2023-11-23 08:33:57,803 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.53 vs. limit=15.0
2023-11-23 08:34:05,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2309040.0, ans=0.0
2023-11-23 08:34:10,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2309040.0, ans=0.0
2023-11-23 08:34:15,592 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9700, loss[loss=0.08603, simple_loss=0.1232, pruned_loss=0.01944, audio_tagging_loss=0.004997, over 15631.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09198, pruned_loss=0.01417, audio_tagging_loss=0.009168, over 3038828.81 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:34:37,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2309173.3333333335, ans=0.125
2023-11-23 08:34:41,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2309173.3333333335, ans=0.125
2023-11-23 08:34:58,288 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346400
2023-11-23 08:35:02,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2309306.6666666665, ans=0.125
2023-11-23 08:35:02,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2309306.6666666665, ans=0.1
2023-11-23 08:35:03,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2309306.6666666665, ans=0.125
2023-11-23 08:35:03,662 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:35:07,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2309373.3333333335, ans=0.125
2023-11-23 08:35:11,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2309373.3333333335, ans=0.0
2023-11-23 08:35:19,606 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.905e+01 8.371e+01 8.992e+01 9.698e+01 1.313e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 08:35:21,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2309373.3333333335, ans=0.125
2023-11-23 08:35:21,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2309373.3333333335, ans=0.125
2023-11-23 08:35:23,358 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9750, loss[loss=0.07105, simple_loss=0.103, pruned_loss=0.01261, audio_tagging_loss=0.006927, over 15569.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09274, pruned_loss=0.0141, audio_tagging_loss=0.009037, over 3038593.80 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:35:26,080 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:35:34,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2309506.6666666665, ans=0.1
2023-11-23 08:35:37,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2309506.6666666665, ans=0.1
2023-11-23 08:35:38,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2309506.6666666665, ans=0.0
2023-11-23 08:35:51,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2309573.3333333335, ans=0.1
2023-11-23 08:36:04,798 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346450
2023-11-23 08:36:15,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2309706.6666666665, ans=0.1
2023-11-23 08:36:24,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2309706.6666666665, ans=0.2
2023-11-23 08:36:27,488 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9800, loss[loss=0.07696, simple_loss=0.09761, pruned_loss=0.01694, audio_tagging_loss=0.01122, over 14568.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09303, pruned_loss=0.0142, audio_tagging_loss=0.009038, over 3035159.83 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:36:28,365 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.36 vs. limit=15.0
2023-11-23 08:36:43,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2309840.0, ans=0.1
2023-11-23 08:37:03,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2309906.6666666665, ans=0.125
2023-11-23 08:37:09,355 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346500
2023-11-23 08:37:23,991 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:37:27,538 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.413e+01 9.129e+01 9.676e+01 1.290e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-23 08:37:29,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2310040.0, ans=0.04949747468305833
2023-11-23 08:37:31,289 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9850, loss[loss=0.05734, simple_loss=0.06868, pruned_loss=0.009534, audio_tagging_loss=0.01347, over 14679.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09335, pruned_loss=0.01425, audio_tagging_loss=0.008951, over 3040827.89 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:37:40,023 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.98 vs. limit=22.5
2023-11-23 08:37:48,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2310173.3333333335, ans=0.125
2023-11-23 08:37:49,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2310173.3333333335, ans=0.125
2023-11-23 08:37:52,798 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=13.77 vs. limit=22.5
2023-11-23 08:37:57,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2310240.0, ans=0.0
2023-11-23 08:37:58,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2310240.0, ans=0.125
2023-11-23 08:38:12,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.71 vs. limit=12.0
2023-11-23 08:38:12,936 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346550
2023-11-23 08:38:20,576 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.10 vs. limit=12.0
2023-11-23 08:38:36,337 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9900, loss[loss=0.05806, simple_loss=0.07578, pruned_loss=0.008928, audio_tagging_loss=0.01124, over 15225.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09354, pruned_loss=0.01432, audio_tagging_loss=0.008932, over 3040741.49 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:39:02,412 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.00 vs. limit=15.0
2023-11-23 08:39:17,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346600
2023-11-23 08:39:21,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2310640.0, ans=0.125
2023-11-23 08:39:36,796 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.313e+01 8.245e+01 9.085e+01 9.612e+01 1.420e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 08:39:40,510 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 9950, loss[loss=0.0551, simple_loss=0.07466, pruned_loss=0.007038, audio_tagging_loss=0.01073, over 15628.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09294, pruned_loss=0.01417, audio_tagging_loss=0.008978, over 3039957.78 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:40:14,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2310906.6666666665, ans=0.0
2023-11-23 08:40:15,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2310906.6666666665, ans=0.0
2023-11-23 08:40:22,043 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346650
2023-11-23 08:40:33,516 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.89 vs. limit=15.0
2023-11-23 08:40:43,977 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10000, loss[loss=0.06367, simple_loss=0.08452, pruned_loss=0.01323, audio_tagging_loss=0.00818, over 15679.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.0922, pruned_loss=0.014, audio_tagging_loss=0.009113, over 3043610.99 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:40:50,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2311106.6666666665, ans=0.125
2023-11-23 08:40:53,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2311106.6666666665, ans=0.0
2023-11-23 08:40:55,505 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.09 vs. limit=6.0
2023-11-23 08:41:16,602 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.04 vs. limit=15.0
2023-11-23 08:41:24,588 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346700
2023-11-23 08:41:44,039 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.283e+01 8.893e+01 9.595e+01 1.369e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 08:41:47,689 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10050, loss[loss=0.05668, simple_loss=0.07472, pruned_loss=0.008725, audio_tagging_loss=0.01059, over 15456.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09183, pruned_loss=0.01384, audio_tagging_loss=0.009134, over 3053243.10 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:41:51,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2311440.0, ans=0.0
2023-11-23 08:42:20,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2311573.3333333335, ans=0.125
2023-11-23 08:42:28,473 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346750
2023-11-23 08:42:35,222 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:42:42,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2311706.6666666665, ans=0.1
2023-11-23 08:42:45,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2311706.6666666665, ans=0.0
2023-11-23 08:42:51,636 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10100, loss[loss=0.05823, simple_loss=0.07405, pruned_loss=0.01199, audio_tagging_loss=0.009214, over 14490.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.09319, pruned_loss=0.01413, audio_tagging_loss=0.009112, over 3054248.66 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:43:07,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2311840.0, ans=0.1
2023-11-23 08:43:32,351 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346800
2023-11-23 08:43:35,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2311973.3333333335, ans=0.125
2023-11-23 08:43:43,424 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:43:52,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2312040.0, ans=0.125
2023-11-23 08:43:53,035 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.400e+01 8.955e+01 9.679e+01 1.203e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 08:43:55,584 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10150, loss[loss=0.08799, simple_loss=0.1216, pruned_loss=0.0187, audio_tagging_loss=0.008508, over 15184.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09285, pruned_loss=0.0141, audio_tagging_loss=0.009136, over 3057701.88 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:44:02,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2312106.6666666665, ans=0.2
2023-11-23 08:44:16,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2312173.3333333335, ans=0.1
2023-11-23 08:44:23,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2312240.0, ans=0.0
2023-11-23 08:44:25,173 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:44:36,957 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346850
2023-11-23 08:44:39,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2312306.6666666665, ans=0.125
2023-11-23 08:44:39,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2312306.6666666665, ans=0.2
2023-11-23 08:44:44,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2312306.6666666665, ans=0.125
2023-11-23 08:44:46,054 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.47 vs. limit=22.5
2023-11-23 08:45:00,227 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10200, loss[loss=0.09675, simple_loss=0.106, pruned_loss=0.03166, audio_tagging_loss=0.01207, over 14863.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09259, pruned_loss=0.01422, audio_tagging_loss=0.009234, over 3058432.54 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:45:04,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2312440.0, ans=0.025
2023-11-23 08:45:19,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2312506.6666666665, ans=0.0
2023-11-23 08:45:20,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2312506.6666666665, ans=0.1
2023-11-23 08:45:23,103 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 08:45:23,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2312506.6666666665, ans=0.025
2023-11-23 08:45:39,833 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.19 vs. limit=15.0
2023-11-23 08:45:40,923 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346900
2023-11-23 08:45:49,345 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.56 vs. limit=8.0
2023-11-23 08:45:54,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2312706.6666666665, ans=0.125
2023-11-23 08:45:54,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2312706.6666666665, ans=0.125
2023-11-23 08:46:01,771 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.830e+01 8.340e+01 8.896e+01 9.645e+01 1.207e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 08:46:04,303 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10250, loss[loss=0.07467, simple_loss=0.09591, pruned_loss=0.01712, audio_tagging_loss=0.0096, over 15538.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09208, pruned_loss=0.01424, audio_tagging_loss=0.009208, over 3053718.12 frames. ], batch size: 58, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:46:07,495 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.75 vs. limit=22.5
2023-11-23 08:46:08,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.50 vs. limit=15.0
2023-11-23 08:46:27,493 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.54 vs. limit=10.0
2023-11-23 08:46:28,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2312906.6666666665, ans=0.125
2023-11-23 08:46:31,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2312906.6666666665, ans=0.125
2023-11-23 08:46:36,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2312906.6666666665, ans=0.125
2023-11-23 08:46:45,684 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 346950
2023-11-23 08:46:50,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2312973.3333333335, ans=0.1
2023-11-23 08:46:54,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2313040.0, ans=0.0
2023-11-23 08:47:01,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2313040.0, ans=0.07
2023-11-23 08:47:08,015 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10300, loss[loss=0.0634, simple_loss=0.0912, pruned_loss=0.00992, audio_tagging_loss=0.007883, over 14004.00 frames. ], tot_loss[loss=0.06982, simple_loss=0.09244, pruned_loss=0.01438, audio_tagging_loss=0.00922, over 3055854.49 frames. ], batch size: 53, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:47:48,545 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347000
2023-11-23 08:47:53,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2313306.6666666665, ans=0.125
2023-11-23 08:47:53,976 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.62 vs. limit=15.0
2023-11-23 08:48:10,248 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.826e+01 8.491e+01 9.106e+01 9.880e+01 1.580e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-23 08:48:12,729 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10350, loss[loss=0.04271, simple_loss=0.04888, pruned_loss=0.006791, audio_tagging_loss=0.01148, over 16771.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.09248, pruned_loss=0.01434, audio_tagging_loss=0.009334, over 3056585.94 frames. ], batch size: 67, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:48:22,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2313440.0, ans=0.125
2023-11-23 08:48:28,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2313506.6666666665, ans=0.2
2023-11-23 08:48:32,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2313506.6666666665, ans=10.0
2023-11-23 08:48:52,781 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347050
2023-11-23 08:49:04,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2313706.6666666665, ans=0.125
2023-11-23 08:49:16,490 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10400, loss[loss=0.09265, simple_loss=0.1254, pruned_loss=0.01951, audio_tagging_loss=0.01046, over 15281.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09128, pruned_loss=0.01412, audio_tagging_loss=0.009529, over 3053399.82 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:49:22,155 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.75 vs. limit=15.0
2023-11-23 08:49:45,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2313906.6666666665, ans=0.125
2023-11-23 08:49:54,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2313973.3333333335, ans=0.125
2023-11-23 08:49:58,387 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347100
2023-11-23 08:50:01,059 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:50:03,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2313973.3333333335, ans=0.0
2023-11-23 08:50:05,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2313973.3333333335, ans=0.0
2023-11-23 08:50:12,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2314040.0, ans=0.125
2023-11-23 08:50:15,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.83 vs. limit=15.0
2023-11-23 08:50:18,925 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.096e+01 8.361e+01 8.771e+01 9.632e+01 1.204e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-23 08:50:20,801 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10450, loss[loss=0.08811, simple_loss=0.1191, pruned_loss=0.02114, audio_tagging_loss=0.007407, over 15541.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09196, pruned_loss=0.01422, audio_tagging_loss=0.009413, over 3050966.22 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:51:02,143 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347150
2023-11-23 08:51:03,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2314306.6666666665, ans=0.0
2023-11-23 08:51:07,295 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:51:08,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2314306.6666666665, ans=0.125
2023-11-23 08:51:24,853 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:51:26,488 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10500, loss[loss=0.05522, simple_loss=0.07122, pruned_loss=0.01152, audio_tagging_loss=0.008087, over 15236.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09165, pruned_loss=0.01409, audio_tagging_loss=0.009278, over 3051867.97 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:51:36,734 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.42 vs. limit=22.5
2023-11-23 08:51:37,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2314506.6666666665, ans=0.04949747468305833
2023-11-23 08:51:40,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2314506.6666666665, ans=0.0
2023-11-23 08:51:46,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2314506.6666666665, ans=0.125
2023-11-23 08:51:51,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2314573.3333333335, ans=0.0
2023-11-23 08:51:56,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2314573.3333333335, ans=0.1
2023-11-23 08:52:06,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347200
2023-11-23 08:52:29,015 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.754e+01 8.399e+01 8.762e+01 9.493e+01 1.186e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 08:52:30,292 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10550, loss[loss=0.06453, simple_loss=0.09602, pruned_loss=0.01127, audio_tagging_loss=0.005253, over 14612.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09169, pruned_loss=0.01388, audio_tagging_loss=0.009161, over 3053387.69 frames. ], batch size: 54, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:52:59,718 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.89 vs. limit=15.0
2023-11-23 08:53:11,426 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.76 vs. limit=15.0
2023-11-23 08:53:12,208 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347250
2023-11-23 08:53:14,149 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=7.99 vs. limit=10.0
2023-11-23 08:53:34,005 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10600, loss[loss=0.0564, simple_loss=0.06969, pruned_loss=0.01018, audio_tagging_loss=0.01137, over 14472.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09154, pruned_loss=0.01392, audio_tagging_loss=0.00915, over 3050593.44 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:53:37,905 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:54:00,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2315240.0, ans=0.0
2023-11-23 08:54:05,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2315240.0, ans=0.125
2023-11-23 08:54:15,986 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347300
2023-11-23 08:54:29,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2315373.3333333335, ans=0.125
2023-11-23 08:54:30,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2315373.3333333335, ans=0.125
2023-11-23 08:54:37,208 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 8.247e+01 8.943e+01 9.697e+01 1.268e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-23 08:54:37,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2315440.0, ans=0.125
2023-11-23 08:54:39,152 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10650, loss[loss=0.05257, simple_loss=0.06311, pruned_loss=0.01048, audio_tagging_loss=0.01053, over 15218.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.0922, pruned_loss=0.01404, audio_tagging_loss=0.009101, over 3046945.77 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:54:44,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2315440.0, ans=0.125
2023-11-23 08:54:50,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2315440.0, ans=0.0
2023-11-23 08:54:51,559 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 08:54:59,565 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.78 vs. limit=15.0
2023-11-23 08:55:17,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2315640.0, ans=0.0
2023-11-23 08:55:19,792 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347350
2023-11-23 08:55:27,011 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.65 vs. limit=15.0
2023-11-23 08:55:38,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2315706.6666666665, ans=0.125
2023-11-23 08:55:41,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2315706.6666666665, ans=0.125
2023-11-23 08:55:44,092 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10700, loss[loss=0.1029, simple_loss=0.1461, pruned_loss=0.02488, audio_tagging_loss=0.004953, over 15743.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09217, pruned_loss=0.01401, audio_tagging_loss=0.009063, over 3048059.24 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:55:45,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2315773.3333333335, ans=0.125
2023-11-23 08:55:54,512 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.77 vs. limit=15.0
2023-11-23 08:56:02,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2315840.0, ans=0.2
2023-11-23 08:56:11,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2315906.6666666665, ans=0.5
2023-11-23 08:56:15,993 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.93 vs. limit=10.0
2023-11-23 08:56:25,724 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347400
2023-11-23 08:56:29,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2315973.3333333335, ans=0.0
2023-11-23 08:56:46,530 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.151e+01 8.772e+01 9.535e+01 1.207e+02, threshold=1.754e+02, percent-clipped=0.0
2023-11-23 08:56:47,781 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10750, loss[loss=0.07851, simple_loss=0.1004, pruned_loss=0.0171, audio_tagging_loss=0.01123, over 14479.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09208, pruned_loss=0.01399, audio_tagging_loss=0.00906, over 3052561.15 frames. ], batch size: 52, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 08:56:51,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2316106.6666666665, ans=0.1
2023-11-23 08:57:16,259 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.87 vs. limit=15.0
2023-11-23 08:57:19,612 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.70 vs. limit=15.0
2023-11-23 08:57:22,116 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.52 vs. limit=15.0
2023-11-23 08:57:28,751 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347450
2023-11-23 08:57:31,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2316306.6666666665, ans=10.0
2023-11-23 08:57:37,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2316373.3333333335, ans=0.07
2023-11-23 08:57:43,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2316373.3333333335, ans=0.0
2023-11-23 08:57:47,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2316373.3333333335, ans=0.2
2023-11-23 08:57:50,641 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10800, loss[loss=0.07588, simple_loss=0.08992, pruned_loss=0.01882, audio_tagging_loss=0.0121, over 14589.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09198, pruned_loss=0.01394, audio_tagging_loss=0.009108, over 3052895.37 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:57:52,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2316440.0, ans=0.125
2023-11-23 08:57:57,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2316440.0, ans=0.125
2023-11-23 08:58:03,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2316506.6666666665, ans=0.125
2023-11-23 08:58:20,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.81 vs. limit=22.5
2023-11-23 08:58:23,871 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.32 vs. limit=6.0
2023-11-23 08:58:24,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2316573.3333333335, ans=0.125
2023-11-23 08:58:25,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2316573.3333333335, ans=0.125
2023-11-23 08:58:31,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347500
2023-11-23 08:58:42,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2316706.6666666665, ans=0.1
2023-11-23 08:58:42,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2316706.6666666665, ans=0.125
2023-11-23 08:58:54,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2316706.6666666665, ans=0.0
2023-11-23 08:58:55,090 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.908e+01 8.196e+01 8.848e+01 9.514e+01 1.233e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 08:58:56,343 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10850, loss[loss=0.07088, simple_loss=0.1007, pruned_loss=0.013, audio_tagging_loss=0.007539, over 15855.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09176, pruned_loss=0.01396, audio_tagging_loss=0.009041, over 3047446.99 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 08:59:01,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2316773.3333333335, ans=0.0
2023-11-23 08:59:01,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2316773.3333333335, ans=0.125
2023-11-23 08:59:15,275 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.80 vs. limit=22.5
2023-11-23 08:59:21,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.38 vs. limit=15.0
2023-11-23 08:59:37,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2316973.3333333335, ans=0.125
2023-11-23 08:59:38,546 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347550
2023-11-23 08:59:46,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2316973.3333333335, ans=0.2
2023-11-23 08:59:47,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2317040.0, ans=0.125
2023-11-23 08:59:57,032 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:00:00,729 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10900, loss[loss=0.0712, simple_loss=0.1046, pruned_loss=0.01204, audio_tagging_loss=0.006869, over 15093.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09153, pruned_loss=0.01388, audio_tagging_loss=0.009085, over 3050479.73 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 09:00:03,757 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=22.5
2023-11-23 09:00:31,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2317240.0, ans=0.1
2023-11-23 09:00:37,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2317240.0, ans=0.1
2023-11-23 09:00:43,180 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347600
2023-11-23 09:01:04,195 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.420e+01 8.915e+01 9.673e+01 1.270e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 09:01:05,478 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 10950, loss[loss=0.07319, simple_loss=0.1003, pruned_loss=0.01694, audio_tagging_loss=0.00612, over 15404.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09165, pruned_loss=0.01404, audio_tagging_loss=0.009126, over 3049196.49 frames. ], batch size: 55, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 09:01:32,130 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.61 vs. limit=22.5
2023-11-23 09:01:47,394 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347650
2023-11-23 09:01:59,196 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.22 vs. limit=15.0
2023-11-23 09:02:04,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2317706.6666666665, ans=0.2
2023-11-23 09:02:11,976 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11000, loss[loss=0.08611, simple_loss=0.1172, pruned_loss=0.01773, audio_tagging_loss=0.009791, over 16024.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09185, pruned_loss=0.01415, audio_tagging_loss=0.00922, over 3050672.75 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:02:18,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2317773.3333333335, ans=0.125
2023-11-23 09:02:19,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.19 vs. limit=15.0
2023-11-23 09:02:21,753 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:02:34,819 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.61 vs. limit=15.0
2023-11-23 09:02:53,192 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347700
2023-11-23 09:03:06,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2318040.0, ans=0.125
2023-11-23 09:03:10,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2318040.0, ans=0.05
2023-11-23 09:03:15,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2318106.6666666665, ans=0.0
2023-11-23 09:03:16,064 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.843e+01 8.585e+01 9.370e+01 1.001e+02 1.222e+02, threshold=1.874e+02, percent-clipped=0.0
2023-11-23 09:03:16,130 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11050, loss[loss=0.07974, simple_loss=0.09894, pruned_loss=0.01954, audio_tagging_loss=0.01073, over 15739.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09183, pruned_loss=0.01416, audio_tagging_loss=0.009251, over 3054334.81 frames. ], batch size: 61, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:03:57,669 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347750
2023-11-23 09:04:19,450 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11100, loss[loss=0.08161, simple_loss=0.1086, pruned_loss=0.02031, audio_tagging_loss=0.007012, over 15868.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.0918, pruned_loss=0.01411, audio_tagging_loss=0.009344, over 3055297.34 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:04:37,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2318506.6666666665, ans=0.0
2023-11-23 09:04:56,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2318573.3333333335, ans=0.2
2023-11-23 09:05:01,257 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347800
2023-11-23 09:05:11,008 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.51 vs. limit=10.0
2023-11-23 09:05:14,826 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.74 vs. limit=15.0
2023-11-23 09:05:15,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2318706.6666666665, ans=0.125
2023-11-23 09:05:20,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2318706.6666666665, ans=0.0
2023-11-23 09:05:25,370 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.253e+01 8.535e+01 9.196e+01 1.008e+02 1.260e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 09:05:25,418 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11150, loss[loss=0.06173, simple_loss=0.0835, pruned_loss=0.008689, audio_tagging_loss=0.0113, over 16298.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09263, pruned_loss=0.01422, audio_tagging_loss=0.009433, over 3057349.25 frames. ], batch size: 62, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:05:27,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2318773.3333333335, ans=0.1
2023-11-23 09:05:33,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2318773.3333333335, ans=0.0
2023-11-23 09:05:46,092 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.18 vs. limit=15.0
2023-11-23 09:05:48,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2318840.0, ans=0.0
2023-11-23 09:06:05,629 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347850
2023-11-23 09:06:07,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2318973.3333333335, ans=0.1
2023-11-23 09:06:29,289 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11200, loss[loss=0.06556, simple_loss=0.08931, pruned_loss=0.01011, audio_tagging_loss=0.01079, over 17250.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09163, pruned_loss=0.01393, audio_tagging_loss=0.009578, over 3067220.44 frames. ], batch size: 65, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 09:06:29,825 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.39 vs. limit=15.0
2023-11-23 09:06:37,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten.whitening_limit, batch_count=2319106.6666666665, ans=15.0
2023-11-23 09:06:45,869 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.57 vs. limit=12.0
2023-11-23 09:07:05,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2319240.0, ans=0.125
2023-11-23 09:07:05,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2319240.0, ans=0.1
2023-11-23 09:07:09,991 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347900
2023-11-23 09:07:32,465 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.786e+01 8.283e+01 8.930e+01 9.840e+01 1.502e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 09:07:32,513 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11250, loss[loss=0.0501, simple_loss=0.06691, pruned_loss=0.007054, audio_tagging_loss=0.009593, over 15858.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09121, pruned_loss=0.01392, audio_tagging_loss=0.00946, over 3053939.41 frames. ], batch size: 59, lr: 2.33e-03, grad_scale: 32.0
2023-11-23 09:07:38,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2319440.0, ans=0.2
2023-11-23 09:07:49,603 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:08:13,119 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 347950
2023-11-23 09:08:26,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2319706.6666666665, ans=0.0
2023-11-23 09:08:36,519 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11300, loss[loss=0.06981, simple_loss=0.08956, pruned_loss=0.01558, audio_tagging_loss=0.009453, over 15758.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09184, pruned_loss=0.01405, audio_tagging_loss=0.00931, over 3055713.63 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:08:38,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2319773.3333333335, ans=0.0
2023-11-23 09:08:44,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2319773.3333333335, ans=0.0
2023-11-23 09:09:01,153 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0
2023-11-23 09:09:09,611 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.46 vs. limit=22.5
2023-11-23 09:09:16,253 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348000
2023-11-23 09:09:17,798 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-348000.pt
2023-11-23 09:09:27,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2319973.3333333335, ans=0.0
2023-11-23 09:09:42,697 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11350, loss[loss=0.06623, simple_loss=0.08749, pruned_loss=0.01345, audio_tagging_loss=0.009029, over 14606.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09265, pruned_loss=0.01433, audio_tagging_loss=0.009139, over 3055925.41 frames. ], batch size: 54, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:09:43,917 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.379e+01 8.172e+01 9.052e+01 9.730e+01 1.154e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 09:10:08,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2320240.0, ans=0.125
2023-11-23 09:10:18,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2320240.0, ans=0.0
2023-11-23 09:10:23,138 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348050
2023-11-23 09:10:45,530 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11400, loss[loss=0.0516, simple_loss=0.07526, pruned_loss=0.006974, audio_tagging_loss=0.006994, over 14959.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09204, pruned_loss=0.01408, audio_tagging_loss=0.009064, over 3040798.74 frames. ], batch size: 60, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:10:55,676 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:11:20,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2320573.3333333335, ans=0.2
2023-11-23 09:11:25,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2320640.0, ans=0.1
2023-11-23 09:11:26,667 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348100
2023-11-23 09:11:49,206 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11450, loss[loss=0.079, simple_loss=0.1096, pruned_loss=0.01598, audio_tagging_loss=0.00822, over 16027.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09231, pruned_loss=0.01404, audio_tagging_loss=0.009048, over 3042783.67 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 8.0
2023-11-23 09:11:52,273 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.268e+01 8.173e+01 8.697e+01 9.540e+01 1.271e+02, threshold=1.739e+02, percent-clipped=0.0
2023-11-23 09:12:09,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2320840.0, ans=0.125
2023-11-23 09:12:11,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=2320840.0, ans=0.5
2023-11-23 09:12:24,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.whiten.whitening_limit, batch_count=2320906.6666666665, ans=12.0
2023-11-23 09:12:29,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348150
2023-11-23 09:12:52,884 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11500, loss[loss=0.06752, simple_loss=0.08801, pruned_loss=0.01209, audio_tagging_loss=0.01143, over 15281.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09189, pruned_loss=0.01411, audio_tagging_loss=0.009134, over 3051797.15 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 8.0
2023-11-23 09:12:54,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2321106.6666666665, ans=0.125
2023-11-23 09:13:03,039 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.21 vs. limit=6.0
2023-11-23 09:13:10,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2321173.3333333335, ans=0.125
2023-11-23 09:13:12,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2321173.3333333335, ans=0.125
2023-11-23 09:13:12,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2321173.3333333335, ans=0.2
2023-11-23 09:13:14,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2321173.3333333335, ans=0.125
2023-11-23 09:13:25,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2321240.0, ans=0.0
2023-11-23 09:13:26,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2321240.0, ans=0.125
2023-11-23 09:13:26,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2321240.0, ans=0.0
2023-11-23 09:13:29,765 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.93 vs. limit=15.0
2023-11-23 09:13:31,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2321306.6666666665, ans=0.0
2023-11-23 09:13:33,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348200
2023-11-23 09:13:46,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2321373.3333333335, ans=0.0
2023-11-23 09:13:56,843 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11550, loss[loss=0.06067, simple_loss=0.08065, pruned_loss=0.01188, audio_tagging_loss=0.008463, over 14439.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09223, pruned_loss=0.01422, audio_tagging_loss=0.009135, over 3053143.25 frames. ], batch size: 56, lr: 2.33e-03, grad_scale: 8.0
2023-11-23 09:13:59,199 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.873e+01 8.220e+01 8.786e+01 9.512e+01 1.197e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-23 09:14:22,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2321573.3333333335, ans=0.1
2023-11-23 09:14:27,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2321573.3333333335, ans=0.0
2023-11-23 09:14:34,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2321640.0, ans=0.1
2023-11-23 09:14:35,095 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:14:36,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2321640.0, ans=0.0
2023-11-23 09:14:37,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348250
2023-11-23 09:14:46,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2321706.6666666665, ans=0.125
2023-11-23 09:15:00,562 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11600, loss[loss=0.05416, simple_loss=0.07214, pruned_loss=0.009477, audio_tagging_loss=0.00861, over 15440.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09309, pruned_loss=0.01422, audio_tagging_loss=0.009092, over 3052630.52 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:15:02,343 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.31 vs. limit=15.0
2023-11-23 09:15:04,957 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.68 vs. limit=15.0
2023-11-23 09:15:25,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2321906.6666666665, ans=0.0
2023-11-23 09:15:32,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2321906.6666666665, ans=0.125
2023-11-23 09:15:34,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2321906.6666666665, ans=0.0
2023-11-23 09:15:35,768 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.58 vs. limit=22.5
2023-11-23 09:15:41,410 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348300
2023-11-23 09:16:04,506 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11650, loss[loss=0.05455, simple_loss=0.07654, pruned_loss=0.008369, audio_tagging_loss=0.007908, over 15347.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09292, pruned_loss=0.01412, audio_tagging_loss=0.009059, over 3052458.80 frames. ], batch size: 57, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:16:06,817 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.255e+01 8.853e+01 9.730e+01 1.242e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 09:16:46,111 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348350
2023-11-23 09:17:02,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2322373.3333333335, ans=0.1
2023-11-23 09:17:05,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2322373.3333333335, ans=0.125
2023-11-23 09:17:08,291 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11700, loss[loss=0.06957, simple_loss=0.09153, pruned_loss=0.01363, audio_tagging_loss=0.01017, over 16407.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09257, pruned_loss=0.01407, audio_tagging_loss=0.009083, over 3055091.54 frames. ], batch size: 63, lr: 2.33e-03, grad_scale: 16.0
2023-11-23 09:17:50,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348400
2023-11-23 09:18:00,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2322706.6666666665, ans=0.125
2023-11-23 09:18:08,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2322706.6666666665, ans=0.0
2023-11-23 09:18:13,162 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11750, loss[loss=0.06502, simple_loss=0.08556, pruned_loss=0.01044, audio_tagging_loss=0.0118, over 15305.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09187, pruned_loss=0.01406, audio_tagging_loss=0.00916, over 3052811.66 frames. ], batch size: 56, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:18:16,769 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.187e+01 8.216e+01 8.845e+01 9.440e+01 1.082e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 09:18:19,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2322773.3333333335, ans=0.125
2023-11-23 09:18:20,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2322773.3333333335, ans=0.2
2023-11-23 09:18:55,055 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348450
2023-11-23 09:19:01,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2322973.3333333335, ans=0.125
2023-11-23 09:19:09,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2323040.0, ans=0.0
2023-11-23 09:19:18,718 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11800, loss[loss=0.07263, simple_loss=0.1033, pruned_loss=0.01262, audio_tagging_loss=0.008382, over 15530.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09151, pruned_loss=0.01406, audio_tagging_loss=0.00923, over 3049491.46 frames. ], batch size: 57, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:19:36,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2323173.3333333335, ans=0.0
2023-11-23 09:19:38,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2323173.3333333335, ans=0.125
2023-11-23 09:19:40,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2323173.3333333335, ans=0.125
2023-11-23 09:19:44,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2323240.0, ans=0.0
2023-11-23 09:19:59,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348500
2023-11-23 09:20:01,968 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.48 vs. limit=15.0
2023-11-23 09:20:22,060 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11850, loss[loss=0.06495, simple_loss=0.09462, pruned_loss=0.01134, audio_tagging_loss=0.006295, over 14696.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09155, pruned_loss=0.01406, audio_tagging_loss=0.009304, over 3036604.93 frames. ], batch size: 56, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:20:24,375 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.461e+01 9.109e+01 9.788e+01 1.263e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-23 09:20:37,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2323506.6666666665, ans=0.0
2023-11-23 09:20:53,427 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.37 vs. limit=12.0
2023-11-23 09:20:55,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2323573.3333333335, ans=0.2
2023-11-23 09:20:57,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2323573.3333333335, ans=0.2
2023-11-23 09:21:03,622 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348550
2023-11-23 09:21:11,561 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.31 vs. limit=15.0
2023-11-23 09:21:18,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2323706.6666666665, ans=0.125
2023-11-23 09:21:20,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2323706.6666666665, ans=0.125
2023-11-23 09:21:25,837 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11900, loss[loss=0.06296, simple_loss=0.08182, pruned_loss=0.009597, audio_tagging_loss=0.01246, over 16382.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.092, pruned_loss=0.014, audio_tagging_loss=0.00935, over 3045651.30 frames. ], batch size: 61, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:21:38,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2323840.0, ans=0.125
2023-11-23 09:22:05,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2323973.3333333335, ans=0.2
2023-11-23 09:22:06,678 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348600
2023-11-23 09:22:11,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2323973.3333333335, ans=0.2
2023-11-23 09:22:15,326 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.05 vs. limit=10.0
2023-11-23 09:22:31,778 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 11950, loss[loss=0.0893, simple_loss=0.1258, pruned_loss=0.01989, audio_tagging_loss=0.006491, over 16445.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09125, pruned_loss=0.01394, audio_tagging_loss=0.009526, over 3049686.02 frames. ], batch size: 58, lr: 2.32e-03, grad_scale: 16.0
2023-11-23 09:22:34,236 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.393e+01 8.390e+01 8.982e+01 9.534e+01 1.652e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 09:22:38,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2324106.6666666665, ans=0.125
2023-11-23 09:23:07,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2324306.6666666665, ans=0.125
2023-11-23 09:23:08,798 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.21 vs. limit=15.0
2023-11-23 09:23:11,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348650
2023-11-23 09:23:13,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2324306.6666666665, ans=0.125
2023-11-23 09:23:14,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2324306.6666666665, ans=0.125
2023-11-23 09:23:30,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2324373.3333333335, ans=0.125
2023-11-23 09:23:33,671 INFO [train_asr.py:1221] (0/4) Epoch 29, batch 12000, loss[loss=0.06521, simple_loss=0.0853, pruned_loss=0.01231, audio_tagging_loss=0.01025, over 16188.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09139, pruned_loss=0.01397, audio_tagging_loss=0.009643, over 3050942.78 frames. ], batch size: 62, lr: 2.32e-03, grad_scale: 32.0
2023-11-23 09:23:33,674 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 09:23:59,896 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([2.9344, 2.5525, 2.2452, 2.7094, 2.4327, 2.5198, 2.4026, 2.4574],
       device='cuda:0')
2023-11-23 09:24:15,955 INFO [train_asr.py:1253] (0/4) Epoch 29, validation: loss=0.05844, simple_loss=0.05118, pruned_loss=0.005114, audio_tagging_loss=0.02774, over 4681554.00 frames. 
2023-11-23 09:24:15,956 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 09:24:39,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2324573.3333333335, ans=0.0
2023-11-23 09:24:44,683 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-29.pt
2023-11-23 09:25:23,514 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 0, loss[loss=0.09197, simple_loss=0.1134, pruned_loss=0.01618, audio_tagging_loss=0.01911, over 15523.00 frames. ], tot_loss[loss=0.09197, simple_loss=0.1134, pruned_loss=0.01618, audio_tagging_loss=0.01911, over 15523.00 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:25:23,516 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 09:25:46,292 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.7749, 5.8570, 5.8909, 5.8788], device='cuda:0')
2023-11-23 09:26:02,088 INFO [train_asr.py:1253] (0/4) Epoch 30, validation: loss=0.05824, simple_loss=0.05113, pruned_loss=0.005061, audio_tagging_loss=0.02761, over 4681554.00 frames. 
2023-11-23 09:26:02,089 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 09:26:09,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2324600.0, ans=0.0
2023-11-23 09:26:11,940 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348700
2023-11-23 09:26:30,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2324733.3333333335, ans=0.025
2023-11-23 09:26:35,745 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 8.954e+01 9.660e+01 1.053e+02 1.291e+02, threshold=1.932e+02, percent-clipped=0.0
2023-11-23 09:27:02,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2324866.6666666665, ans=0.0
2023-11-23 09:27:03,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.57 vs. limit=15.0
2023-11-23 09:27:05,460 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 50, loss[loss=0.09733, simple_loss=0.1223, pruned_loss=0.02419, audio_tagging_loss=0.01197, over 15235.00 frames. ], tot_loss[loss=0.08026, simple_loss=0.09599, pruned_loss=0.0148, audio_tagging_loss=0.01746, over 683894.64 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:27:15,520 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348750
2023-11-23 09:27:35,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2325066.6666666665, ans=0.125
2023-11-23 09:27:37,453 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.57 vs. limit=12.0
2023-11-23 09:27:42,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2325066.6666666665, ans=0.125
2023-11-23 09:27:59,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2325200.0, ans=0.125
2023-11-23 09:28:02,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2325200.0, ans=0.125
2023-11-23 09:28:07,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2325266.6666666665, ans=0.0
2023-11-23 09:28:08,354 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 100, loss[loss=0.0743, simple_loss=0.08855, pruned_loss=0.01487, audio_tagging_loss=0.01516, over 14302.00 frames. ], tot_loss[loss=0.07813, simple_loss=0.09336, pruned_loss=0.01463, audio_tagging_loss=0.01682, over 1206192.96 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:28:10,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2325266.6666666665, ans=0.125
2023-11-23 09:28:18,710 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348800
2023-11-23 09:28:42,103 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.02 vs. limit=22.5
2023-11-23 09:28:44,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2325400.0, ans=0.125
2023-11-23 09:28:46,352 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.439e+01 8.985e+01 9.565e+01 1.028e+02 2.272e+02, threshold=1.913e+02, percent-clipped=1.0
2023-11-23 09:28:47,943 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:28:52,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2325466.6666666665, ans=0.125
2023-11-23 09:29:06,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2325533.3333333335, ans=0.125
2023-11-23 09:29:08,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2325533.3333333335, ans=0.025
2023-11-23 09:29:13,429 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 150, loss[loss=0.08048, simple_loss=0.112, pruned_loss=0.01473, audio_tagging_loss=0.009772, over 15710.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09562, pruned_loss=0.01441, audio_tagging_loss=0.01491, over 1618052.46 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:29:15,521 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:29:24,405 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348850
2023-11-23 09:29:37,314 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.40 vs. limit=15.0
2023-11-23 09:29:39,695 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.41 vs. limit=15.0
2023-11-23 09:29:56,400 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.11 vs. limit=22.5
2023-11-23 09:30:06,268 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.03 vs. limit=22.5
2023-11-23 09:30:10,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2325866.6666666665, ans=0.025
2023-11-23 09:30:18,419 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 200, loss[loss=0.06157, simple_loss=0.07625, pruned_loss=0.01053, audio_tagging_loss=0.01291, over 14620.00 frames. ], tot_loss[loss=0.0747, simple_loss=0.09427, pruned_loss=0.01431, audio_tagging_loss=0.01325, over 1935796.56 frames. ], batch size: 53, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:30:28,218 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348900
2023-11-23 09:30:55,305 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.392e+01 9.007e+01 9.652e+01 1.196e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 09:31:04,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2326133.3333333335, ans=0.125
2023-11-23 09:31:16,136 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.88 vs. limit=22.5
2023-11-23 09:31:20,013 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.40 vs. limit=10.0
2023-11-23 09:31:21,710 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 250, loss[loss=0.08044, simple_loss=0.1048, pruned_loss=0.01657, audio_tagging_loss=0.01146, over 15348.00 frames. ], tot_loss[loss=0.07339, simple_loss=0.09405, pruned_loss=0.01425, audio_tagging_loss=0.01211, over 2181419.20 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:31:23,718 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.31 vs. limit=22.5
2023-11-23 09:31:31,579 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 348950
2023-11-23 09:31:31,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2326266.6666666665, ans=0.1
2023-11-23 09:31:34,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.56 vs. limit=15.0
2023-11-23 09:31:51,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2326400.0, ans=0.125
2023-11-23 09:32:20,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2326533.3333333335, ans=0.0
2023-11-23 09:32:21,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2326533.3333333335, ans=0.0
2023-11-23 09:32:25,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2326600.0, ans=10.0
2023-11-23 09:32:25,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2326600.0, ans=0.0
2023-11-23 09:32:26,045 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 300, loss[loss=0.06091, simple_loss=0.0813, pruned_loss=0.01232, audio_tagging_loss=0.00794, over 14934.00 frames. ], tot_loss[loss=0.07289, simple_loss=0.09478, pruned_loss=0.01428, audio_tagging_loss=0.01122, over 2369152.37 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:32:36,464 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349000
2023-11-23 09:32:38,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2326666.6666666665, ans=0.0
2023-11-23 09:32:42,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2326666.6666666665, ans=0.125
2023-11-23 09:32:52,952 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.49 vs. limit=15.0
2023-11-23 09:32:56,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_na.min_abs, batch_count=2326733.3333333335, ans=0.02
2023-11-23 09:32:57,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2326733.3333333335, ans=0.2
2023-11-23 09:33:02,992 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.407e+01 8.907e+01 9.577e+01 1.241e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 09:33:09,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2326800.0, ans=0.0
2023-11-23 09:33:25,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2326866.6666666665, ans=0.125
2023-11-23 09:33:30,928 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 350, loss[loss=0.06548, simple_loss=0.09246, pruned_loss=0.01102, audio_tagging_loss=0.008225, over 15519.00 frames. ], tot_loss[loss=0.07208, simple_loss=0.09451, pruned_loss=0.01424, audio_tagging_loss=0.01058, over 2518135.93 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 09:33:40,795 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349050
2023-11-23 09:33:53,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2327000.0, ans=0.0
2023-11-23 09:33:54,648 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.55 vs. limit=15.0
2023-11-23 09:34:15,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2327133.3333333335, ans=0.125
2023-11-23 09:34:34,500 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 400, loss[loss=0.04952, simple_loss=0.05686, pruned_loss=0.008516, audio_tagging_loss=0.01257, over 14813.00 frames. ], tot_loss[loss=0.07121, simple_loss=0.09378, pruned_loss=0.01418, audio_tagging_loss=0.01014, over 2637634.68 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:34:44,556 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349100
2023-11-23 09:34:55,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2327333.3333333335, ans=0.0
2023-11-23 09:35:11,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2327400.0, ans=0.2
2023-11-23 09:35:13,632 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.097e+01 8.415e+01 9.145e+01 9.820e+01 1.379e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-23 09:35:26,759 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.14 vs. limit=10.0
2023-11-23 09:35:36,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2327533.3333333335, ans=0.0
2023-11-23 09:35:39,517 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 450, loss[loss=0.07335, simple_loss=0.1057, pruned_loss=0.01283, audio_tagging_loss=0.00769, over 16218.00 frames. ], tot_loss[loss=0.07021, simple_loss=0.09265, pruned_loss=0.01397, audio_tagging_loss=0.009914, over 2730101.24 frames. ], batch size: 62, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:35:40,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2327600.0, ans=0.125
2023-11-23 09:35:41,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2327600.0, ans=0.125
2023-11-23 09:35:50,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349150
2023-11-23 09:36:29,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2327866.6666666665, ans=0.0
2023-11-23 09:36:43,355 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 500, loss[loss=0.08344, simple_loss=0.1143, pruned_loss=0.01949, audio_tagging_loss=0.00679, over 16119.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09248, pruned_loss=0.01398, audio_tagging_loss=0.009802, over 2806672.06 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:36:48,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2327933.3333333335, ans=0.125
2023-11-23 09:36:53,771 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349200
2023-11-23 09:37:14,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2328066.6666666665, ans=0.2
2023-11-23 09:37:22,405 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.714e+01 8.359e+01 8.809e+01 9.526e+01 1.262e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 09:37:48,460 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 550, loss[loss=0.0528, simple_loss=0.07024, pruned_loss=0.009233, audio_tagging_loss=0.008444, over 14303.00 frames. ], tot_loss[loss=0.07006, simple_loss=0.09252, pruned_loss=0.01408, audio_tagging_loss=0.009719, over 2855583.21 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:37:58,378 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349250
2023-11-23 09:38:23,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2328400.0, ans=0.125
2023-11-23 09:38:25,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2328466.6666666665, ans=0.0
2023-11-23 09:38:28,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2328466.6666666665, ans=0.0
2023-11-23 09:38:31,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2328466.6666666665, ans=0.125
2023-11-23 09:38:35,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.40 vs. limit=15.0
2023-11-23 09:38:39,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2328533.3333333335, ans=0.1
2023-11-23 09:38:52,669 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 600, loss[loss=0.07208, simple_loss=0.09483, pruned_loss=0.01482, audio_tagging_loss=0.00984, over 14832.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09286, pruned_loss=0.01409, audio_tagging_loss=0.009618, over 2898017.61 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:39:02,769 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.29 vs. limit=15.0
2023-11-23 09:39:03,351 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349300
2023-11-23 09:39:11,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2328666.6666666665, ans=0.125
2023-11-23 09:39:16,973 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.98 vs. limit=6.0
2023-11-23 09:39:30,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2328800.0, ans=0.125
2023-11-23 09:39:31,015 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.406e+01 8.142e+01 8.700e+01 9.475e+01 1.259e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-23 09:39:57,388 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 650, loss[loss=0.07543, simple_loss=0.0962, pruned_loss=0.01842, audio_tagging_loss=0.008905, over 15298.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09269, pruned_loss=0.01426, audio_tagging_loss=0.009582, over 2931432.11 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:40:07,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349350
2023-11-23 09:40:10,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2329000.0, ans=0.0
2023-11-23 09:40:13,082 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:40:14,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2329000.0, ans=0.0
2023-11-23 09:40:18,212 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.12 vs. limit=22.5
2023-11-23 09:40:19,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2329000.0, ans=0.2
2023-11-23 09:40:20,311 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:40:25,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2329066.6666666665, ans=0.1
2023-11-23 09:40:36,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2329133.3333333335, ans=0.125
2023-11-23 09:40:40,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2329133.3333333335, ans=0.2
2023-11-23 09:40:49,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2329200.0, ans=0.2
2023-11-23 09:41:02,196 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 700, loss[loss=0.07691, simple_loss=0.1017, pruned_loss=0.01612, audio_tagging_loss=0.009927, over 14628.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09274, pruned_loss=0.01419, audio_tagging_loss=0.009456, over 2956169.60 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:41:03,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2329266.6666666665, ans=0.1
2023-11-23 09:41:06,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2329266.6666666665, ans=0.0
2023-11-23 09:41:07,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2329266.6666666665, ans=0.1
2023-11-23 09:41:12,059 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349400
2023-11-23 09:41:13,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2329333.3333333335, ans=0.125
2023-11-23 09:41:18,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2329333.3333333335, ans=0.1
2023-11-23 09:41:29,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2329400.0, ans=0.0
2023-11-23 09:41:39,336 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.78 vs. limit=22.5
2023-11-23 09:41:41,175 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.961e+01 8.086e+01 8.675e+01 9.792e+01 1.198e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-23 09:41:42,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2329466.6666666665, ans=0.025
2023-11-23 09:41:44,221 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.10 vs. limit=15.0
2023-11-23 09:41:48,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2329466.6666666665, ans=0.1
2023-11-23 09:41:55,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2329533.3333333335, ans=0.0
2023-11-23 09:42:06,212 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 750, loss[loss=0.06548, simple_loss=0.08228, pruned_loss=0.01461, audio_tagging_loss=0.009733, over 15455.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09224, pruned_loss=0.0141, audio_tagging_loss=0.009394, over 2975775.89 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:42:10,150 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.02 vs. limit=10.0
2023-11-23 09:42:17,150 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349450
2023-11-23 09:42:21,511 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.47 vs. limit=6.0
2023-11-23 09:42:36,898 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.28 vs. limit=22.5
2023-11-23 09:43:12,044 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 800, loss[loss=0.06442, simple_loss=0.07906, pruned_loss=0.01528, audio_tagging_loss=0.009608, over 14646.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09294, pruned_loss=0.01423, audio_tagging_loss=0.009442, over 2997527.20 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:43:17,130 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 09:43:20,001 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.42 vs. limit=15.0
2023-11-23 09:43:21,738 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349500
2023-11-23 09:43:26,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2330000.0, ans=0.125
2023-11-23 09:43:46,463 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.66 vs. limit=15.0
2023-11-23 09:43:50,470 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.139e+01 8.858e+01 9.489e+01 1.259e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 09:44:01,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2330133.3333333335, ans=15.0
2023-11-23 09:44:15,724 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 850, loss[loss=0.05573, simple_loss=0.07265, pruned_loss=0.008453, audio_tagging_loss=0.01096, over 14883.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09341, pruned_loss=0.01417, audio_tagging_loss=0.009395, over 3006823.53 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:44:15,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2330266.6666666665, ans=0.125
2023-11-23 09:44:26,113 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349550
2023-11-23 09:44:48,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2330400.0, ans=0.125
2023-11-23 09:44:51,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2330400.0, ans=0.125
2023-11-23 09:45:06,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.96 vs. limit=15.0
2023-11-23 09:45:11,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2330533.3333333335, ans=0.0
2023-11-23 09:45:18,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2330600.0, ans=0.1
2023-11-23 09:45:19,618 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 900, loss[loss=0.0854, simple_loss=0.1089, pruned_loss=0.02123, audio_tagging_loss=0.009743, over 14525.00 frames. ], tot_loss[loss=0.07057, simple_loss=0.09365, pruned_loss=0.01427, audio_tagging_loss=0.009477, over 3004939.56 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:45:30,747 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349600
2023-11-23 09:45:58,085 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.328e+01 8.818e+01 9.334e+01 1.944e+02, threshold=1.764e+02, percent-clipped=1.0
2023-11-23 09:45:59,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2330800.0, ans=0.125
2023-11-23 09:46:11,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2330866.6666666665, ans=0.0
2023-11-23 09:46:18,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2330866.6666666665, ans=0.0
2023-11-23 09:46:19,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2330866.6666666665, ans=0.125
2023-11-23 09:46:24,204 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 950, loss[loss=0.07348, simple_loss=0.08986, pruned_loss=0.01789, audio_tagging_loss=0.01065, over 15655.00 frames. ], tot_loss[loss=0.07045, simple_loss=0.09361, pruned_loss=0.01429, audio_tagging_loss=0.009356, over 3018437.20 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:46:34,448 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349650
2023-11-23 09:46:37,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2331000.0, ans=0.125
2023-11-23 09:46:44,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2331000.0, ans=0.125
2023-11-23 09:46:54,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2331066.6666666665, ans=0.0
2023-11-23 09:47:05,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2331133.3333333335, ans=0.125
2023-11-23 09:47:11,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2331133.3333333335, ans=0.1
2023-11-23 09:47:28,206 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1000, loss[loss=0.05318, simple_loss=0.06649, pruned_loss=0.008565, audio_tagging_loss=0.01137, over 15531.00 frames. ], tot_loss[loss=0.07024, simple_loss=0.09364, pruned_loss=0.01432, audio_tagging_loss=0.0091, over 3025511.07 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:47:38,041 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349700
2023-11-23 09:47:57,297 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:48:07,586 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.508e+01 8.244e+01 9.173e+01 9.794e+01 1.161e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-23 09:48:25,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2331533.3333333335, ans=0.125
2023-11-23 09:48:32,115 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1050, loss[loss=0.05463, simple_loss=0.0711, pruned_loss=0.01168, audio_tagging_loss=0.007401, over 14134.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09263, pruned_loss=0.0141, audio_tagging_loss=0.009085, over 3025002.95 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:48:43,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349750
2023-11-23 09:49:09,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2331733.3333333335, ans=0.0
2023-11-23 09:49:18,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2331800.0, ans=0.125
2023-11-23 09:49:36,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2331933.3333333335, ans=0.125
2023-11-23 09:49:37,736 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1100, loss[loss=0.08019, simple_loss=0.1041, pruned_loss=0.01955, audio_tagging_loss=0.008615, over 14993.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09248, pruned_loss=0.01399, audio_tagging_loss=0.009088, over 3028872.37 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:49:40,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2331933.3333333335, ans=0.125
2023-11-23 09:49:41,463 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:49:48,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349800
2023-11-23 09:50:15,123 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 8.049e+01 8.629e+01 9.475e+01 1.161e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-23 09:50:24,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2332133.3333333335, ans=0.125
2023-11-23 09:50:33,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2332200.0, ans=0.125
2023-11-23 09:50:36,401 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.84 vs. limit=22.5
2023-11-23 09:50:42,044 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1150, loss[loss=0.08566, simple_loss=0.1229, pruned_loss=0.01822, audio_tagging_loss=0.006011, over 14911.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09273, pruned_loss=0.01394, audio_tagging_loss=0.008892, over 3032547.17 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:50:52,118 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349850
2023-11-23 09:51:03,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2332333.3333333335, ans=0.1
2023-11-23 09:51:11,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2332400.0, ans=0.125
2023-11-23 09:51:45,112 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1200, loss[loss=0.05768, simple_loss=0.07333, pruned_loss=0.0109, audio_tagging_loss=0.01011, over 14618.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09273, pruned_loss=0.01393, audio_tagging_loss=0.008894, over 3038024.96 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:51:51,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.32 vs. limit=22.5
2023-11-23 09:51:54,966 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349900
2023-11-23 09:52:19,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2332733.3333333335, ans=0.0
2023-11-23 09:52:20,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2332733.3333333335, ans=0.125
2023-11-23 09:52:23,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2332800.0, ans=0.0
2023-11-23 09:52:24,715 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.318e+01 8.340e+01 9.039e+01 9.535e+01 1.288e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-23 09:52:40,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2332866.6666666665, ans=0.2
2023-11-23 09:52:49,382 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1250, loss[loss=0.06266, simple_loss=0.07821, pruned_loss=0.0101, audio_tagging_loss=0.01346, over 16240.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09191, pruned_loss=0.01388, audio_tagging_loss=0.008968, over 3042613.71 frames. ], batch size: 63, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:52:59,694 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 349950
2023-11-23 09:53:42,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2333200.0, ans=0.125
2023-11-23 09:53:52,859 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1300, loss[loss=0.08691, simple_loss=0.1135, pruned_loss=0.02156, audio_tagging_loss=0.008615, over 14973.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09179, pruned_loss=0.01391, audio_tagging_loss=0.008938, over 3042454.74 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:53:54,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2333266.6666666665, ans=0.125
2023-11-23 09:54:03,069 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350000
2023-11-23 09:54:08,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2333333.3333333335, ans=0.0
2023-11-23 09:54:13,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2333333.3333333335, ans=0.125
2023-11-23 09:54:13,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.80 vs. limit=15.0
2023-11-23 09:54:17,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2333400.0, ans=0.0
2023-11-23 09:54:27,683 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.28 vs. limit=15.0
2023-11-23 09:54:32,748 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.017e+01 7.999e+01 8.654e+01 9.564e+01 1.101e+02, threshold=1.731e+02, percent-clipped=0.0
2023-11-23 09:54:43,870 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.52 vs. limit=15.0
2023-11-23 09:54:49,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2333533.3333333335, ans=0.95
2023-11-23 09:54:49,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2333533.3333333335, ans=0.125
2023-11-23 09:54:56,653 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1350, loss[loss=0.07352, simple_loss=0.09428, pruned_loss=0.01532, audio_tagging_loss=0.01106, over 13680.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09189, pruned_loss=0.01389, audio_tagging_loss=0.008925, over 3042575.05 frames. ], batch size: 53, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:55:06,718 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350050
2023-11-23 09:55:39,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2333800.0, ans=0.125
2023-11-23 09:55:41,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2333800.0, ans=0.2
2023-11-23 09:55:43,883 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 09:56:00,519 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1400, loss[loss=0.08358, simple_loss=0.1172, pruned_loss=0.01884, audio_tagging_loss=0.006149, over 15962.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09156, pruned_loss=0.01388, audio_tagging_loss=0.009078, over 3039418.97 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:56:04,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2333933.3333333335, ans=0.0
2023-11-23 09:56:11,068 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350100
2023-11-23 09:56:22,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2334000.0, ans=0.0
2023-11-23 09:56:24,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2334000.0, ans=0.0
2023-11-23 09:56:26,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2334066.6666666665, ans=0.125
2023-11-23 09:56:26,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2334066.6666666665, ans=0.125
2023-11-23 09:56:31,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2334066.6666666665, ans=0.0
2023-11-23 09:56:33,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2334066.6666666665, ans=10.0
2023-11-23 09:56:39,726 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.829e+01 8.084e+01 8.982e+01 9.535e+01 1.222e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 09:56:51,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2334200.0, ans=0.07
2023-11-23 09:57:04,693 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1450, loss[loss=0.07772, simple_loss=0.1095, pruned_loss=0.01712, audio_tagging_loss=0.005871, over 15156.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.0925, pruned_loss=0.01397, audio_tagging_loss=0.00902, over 3035361.33 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 09:57:14,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350150
2023-11-23 09:57:19,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2334333.3333333335, ans=0.0
2023-11-23 09:57:19,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2334333.3333333335, ans=0.125
2023-11-23 09:57:23,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2334333.3333333335, ans=0.0
2023-11-23 09:57:30,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.56 vs. limit=22.5
2023-11-23 09:57:38,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.03 vs. limit=22.5
2023-11-23 09:57:46,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2334466.6666666665, ans=0.125
2023-11-23 09:57:53,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2334533.3333333335, ans=0.125
2023-11-23 09:58:06,392 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1500, loss[loss=0.07653, simple_loss=0.09318, pruned_loss=0.02073, audio_tagging_loss=0.009212, over 16007.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09319, pruned_loss=0.01411, audio_tagging_loss=0.009155, over 3034342.79 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 09:58:16,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350200
2023-11-23 09:58:33,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2334733.3333333335, ans=0.125
2023-11-23 09:58:49,007 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.294e+01 8.399e+01 9.056e+01 9.468e+01 1.678e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 09:58:59,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2334866.6666666665, ans=0.1
2023-11-23 09:59:00,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2334866.6666666665, ans=0.125
2023-11-23 09:59:10,065 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1550, loss[loss=0.05696, simple_loss=0.06732, pruned_loss=0.01257, audio_tagging_loss=0.01072, over 15795.00 frames. ], tot_loss[loss=0.06974, simple_loss=0.09307, pruned_loss=0.01413, audio_tagging_loss=0.009078, over 3040276.54 frames. ], batch size: 62, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 09:59:10,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2334933.3333333335, ans=0.0
2023-11-23 09:59:19,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2334933.3333333335, ans=0.125
2023-11-23 09:59:20,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350250
2023-11-23 09:59:23,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2335000.0, ans=0.125
2023-11-23 09:59:32,022 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.69 vs. limit=15.0
2023-11-23 09:59:41,737 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.17 vs. limit=15.0
2023-11-23 10:00:04,729 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.23 vs. limit=12.0
2023-11-23 10:00:14,438 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1600, loss[loss=0.07628, simple_loss=0.0973, pruned_loss=0.01678, audio_tagging_loss=0.01085, over 15344.00 frames. ], tot_loss[loss=0.06989, simple_loss=0.09309, pruned_loss=0.01417, audio_tagging_loss=0.009178, over 3037449.44 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:00:17,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2335266.6666666665, ans=0.125
2023-11-23 10:00:24,792 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350300
2023-11-23 10:00:33,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2335333.3333333335, ans=0.0
2023-11-23 10:00:40,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.07 vs. limit=15.0
2023-11-23 10:00:44,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2335400.0, ans=0.125
2023-11-23 10:00:54,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2335466.6666666665, ans=0.95
2023-11-23 10:00:55,560 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.176e+01 8.486e+01 9.012e+01 9.974e+01 1.250e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 10:01:16,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2335600.0, ans=0.125
2023-11-23 10:01:17,409 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1650, loss[loss=0.05788, simple_loss=0.07012, pruned_loss=0.0107, audio_tagging_loss=0.01212, over 15497.00 frames. ], tot_loss[loss=0.06947, simple_loss=0.09235, pruned_loss=0.01396, audio_tagging_loss=0.00933, over 3038820.80 frames. ], batch size: 60, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:01:27,182 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350350
2023-11-23 10:01:27,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2335600.0, ans=0.0
2023-11-23 10:02:05,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2335800.0, ans=0.125
2023-11-23 10:02:21,308 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1700, loss[loss=0.06413, simple_loss=0.08149, pruned_loss=0.01353, audio_tagging_loss=0.00986, over 14502.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09248, pruned_loss=0.01416, audio_tagging_loss=0.009315, over 3042495.09 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:02:31,875 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350400
2023-11-23 10:03:04,968 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.998e+01 8.246e+01 8.760e+01 9.410e+01 1.135e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 10:03:16,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2336200.0, ans=0.0
2023-11-23 10:03:25,711 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1750, loss[loss=0.06985, simple_loss=0.08968, pruned_loss=0.01422, audio_tagging_loss=0.01079, over 14204.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09242, pruned_loss=0.01404, audio_tagging_loss=0.009309, over 3046056.79 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:03:30,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2336266.6666666665, ans=0.0
2023-11-23 10:03:36,516 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350450
2023-11-23 10:03:51,405 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.51 vs. limit=6.0
2023-11-23 10:04:07,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2336466.6666666665, ans=0.125
2023-11-23 10:04:24,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2336533.3333333335, ans=0.0
2023-11-23 10:04:31,151 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1800, loss[loss=0.0747, simple_loss=0.09697, pruned_loss=0.01444, audio_tagging_loss=0.01177, over 15882.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09316, pruned_loss=0.01398, audio_tagging_loss=0.009173, over 3052590.10 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:04:37,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2336600.0, ans=0.125
2023-11-23 10:04:37,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2336600.0, ans=0.0
2023-11-23 10:04:41,018 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350500
2023-11-23 10:04:43,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2336666.6666666665, ans=0.1
2023-11-23 10:05:05,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2336733.3333333335, ans=0.1
2023-11-23 10:05:14,791 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.56 vs. limit=15.0
2023-11-23 10:05:15,199 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.757e+01 8.590e+01 9.171e+01 9.752e+01 1.397e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 10:05:27,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2336866.6666666665, ans=0.125
2023-11-23 10:05:31,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2336866.6666666665, ans=0.035
2023-11-23 10:05:32,074 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:05:35,402 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1850, loss[loss=0.06652, simple_loss=0.09356, pruned_loss=0.01082, audio_tagging_loss=0.008913, over 15345.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09217, pruned_loss=0.01386, audio_tagging_loss=0.009198, over 3051644.70 frames. ], batch size: 59, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:05:46,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350550
2023-11-23 10:05:46,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2336933.3333333335, ans=0.125
2023-11-23 10:05:49,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.01 vs. limit=15.0
2023-11-23 10:05:58,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2337000.0, ans=0.0
2023-11-23 10:06:03,021 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:06:05,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2337066.6666666665, ans=0.025
2023-11-23 10:06:12,322 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:06:16,226 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.12 vs. limit=12.0
2023-11-23 10:06:31,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2337200.0, ans=0.125
2023-11-23 10:06:37,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2337200.0, ans=0.125
2023-11-23 10:06:38,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2337266.6666666665, ans=0.125
2023-11-23 10:06:39,847 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1900, loss[loss=0.1013, simple_loss=0.1391, pruned_loss=0.02452, audio_tagging_loss=0.007209, over 14821.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09223, pruned_loss=0.01389, audio_tagging_loss=0.009123, over 3061231.17 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:06:50,379 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350600
2023-11-23 10:06:53,262 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:06:54,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2337333.3333333335, ans=0.125
2023-11-23 10:07:17,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2337466.6666666665, ans=0.125
2023-11-23 10:07:24,832 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.717e+01 8.255e+01 9.028e+01 9.866e+01 1.250e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 10:07:33,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2337533.3333333335, ans=0.125
2023-11-23 10:07:45,220 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 1950, loss[loss=0.07771, simple_loss=0.1055, pruned_loss=0.01672, audio_tagging_loss=0.008241, over 15128.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09135, pruned_loss=0.01375, audio_tagging_loss=0.009044, over 3052941.11 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 8.0
2023-11-23 10:07:45,774 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.24 vs. limit=22.5
2023-11-23 10:07:54,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2337600.0, ans=0.1
2023-11-23 10:07:55,249 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.72 vs. limit=15.0
2023-11-23 10:07:55,976 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350650
2023-11-23 10:08:04,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2337666.6666666665, ans=0.125
2023-11-23 10:08:16,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2337733.3333333335, ans=0.04949747468305833
2023-11-23 10:08:23,333 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.25 vs. limit=22.5
2023-11-23 10:08:25,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2337800.0, ans=0.125
2023-11-23 10:08:28,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2337800.0, ans=0.125
2023-11-23 10:08:51,221 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2000, loss[loss=0.05512, simple_loss=0.06194, pruned_loss=0.01207, audio_tagging_loss=0.01208, over 14604.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09037, pruned_loss=0.01375, audio_tagging_loss=0.009099, over 3043046.09 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:09:02,134 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350700
2023-11-23 10:09:06,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2338000.0, ans=0.2
2023-11-23 10:09:18,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=12.0
2023-11-23 10:09:35,805 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.268e+01 8.767e+01 9.455e+01 1.201e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-23 10:09:38,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2338133.3333333335, ans=0.0
2023-11-23 10:09:57,055 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2050, loss[loss=0.06287, simple_loss=0.0847, pruned_loss=0.01358, audio_tagging_loss=0.006941, over 15170.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09097, pruned_loss=0.01374, audio_tagging_loss=0.009104, over 3043203.18 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:09:57,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.81 vs. limit=10.0
2023-11-23 10:10:07,742 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350750
2023-11-23 10:10:07,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2338266.6666666665, ans=0.5
2023-11-23 10:10:15,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2338333.3333333335, ans=0.125
2023-11-23 10:10:41,631 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.84 vs. limit=6.0
2023-11-23 10:10:52,227 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.70 vs. limit=22.5
2023-11-23 10:11:01,824 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2100, loss[loss=0.07765, simple_loss=0.121, pruned_loss=0.01117, audio_tagging_loss=0.005963, over 14890.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09119, pruned_loss=0.01381, audio_tagging_loss=0.008966, over 3040066.23 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:11:03,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2338600.0, ans=0.1
2023-11-23 10:11:11,702 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350800
2023-11-23 10:11:24,066 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:11:46,496 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.972e+01 8.232e+01 8.810e+01 9.514e+01 1.231e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 10:11:53,403 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.35 vs. limit=15.0
2023-11-23 10:12:06,511 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2150, loss[loss=0.09028, simple_loss=0.1198, pruned_loss=0.02236, audio_tagging_loss=0.007998, over 15287.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09225, pruned_loss=0.014, audio_tagging_loss=0.008929, over 3038516.97 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:12:09,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2338933.3333333335, ans=0.0
2023-11-23 10:12:15,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2338933.3333333335, ans=0.0
2023-11-23 10:12:17,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350850
2023-11-23 10:12:36,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2339066.6666666665, ans=0.125
2023-11-23 10:12:41,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2339066.6666666665, ans=0.0
2023-11-23 10:12:47,149 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:13:12,304 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2200, loss[loss=0.0665, simple_loss=0.09468, pruned_loss=0.01201, audio_tagging_loss=0.007144, over 15069.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09124, pruned_loss=0.01394, audio_tagging_loss=0.008975, over 3030628.68 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:13:22,669 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350900
2023-11-23 10:13:22,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2339266.6666666665, ans=0.0
2023-11-23 10:13:36,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2339400.0, ans=0.04949747468305833
2023-11-23 10:13:48,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2339466.6666666665, ans=0.0
2023-11-23 10:13:55,238 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.425e+01 8.264e+01 8.958e+01 9.575e+01 1.152e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 10:14:12,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2339533.3333333335, ans=0.125
2023-11-23 10:14:17,029 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2250, loss[loss=0.06671, simple_loss=0.08262, pruned_loss=0.01419, audio_tagging_loss=0.01121, over 15099.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09097, pruned_loss=0.0138, audio_tagging_loss=0.009042, over 3037276.69 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:14:18,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2339600.0, ans=0.1
2023-11-23 10:14:21,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2339600.0, ans=0.125
2023-11-23 10:14:26,988 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 350950
2023-11-23 10:14:39,878 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.33 vs. limit=15.0
2023-11-23 10:14:42,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.25 vs. limit=15.0
2023-11-23 10:14:47,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2339733.3333333335, ans=0.1
2023-11-23 10:14:55,279 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.65 vs. limit=15.0
2023-11-23 10:14:56,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2339800.0, ans=0.125
2023-11-23 10:15:14,661 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.72 vs. limit=6.0
2023-11-23 10:15:17,033 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.34 vs. limit=15.0
2023-11-23 10:15:21,427 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2300, loss[loss=0.0589, simple_loss=0.08661, pruned_loss=0.008825, audio_tagging_loss=0.006769, over 14284.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09138, pruned_loss=0.01377, audio_tagging_loss=0.009018, over 3042541.43 frames. ], batch size: 54, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:15:31,195 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351000
2023-11-23 10:15:37,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2340000.0, ans=0.125
2023-11-23 10:15:52,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2340066.6666666665, ans=0.0
2023-11-23 10:16:01,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2340133.3333333335, ans=0.125
2023-11-23 10:16:03,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2340133.3333333335, ans=0.125
2023-11-23 10:16:06,089 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.543e+01 9.090e+01 9.740e+01 1.795e+02, threshold=1.818e+02, percent-clipped=1.0
2023-11-23 10:16:08,853 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:16:11,610 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.47 vs. limit=15.0
2023-11-23 10:16:12,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2340200.0, ans=0.125
2023-11-23 10:16:19,856 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:16:20,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2340200.0, ans=0.1
2023-11-23 10:16:26,864 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2350, loss[loss=0.05965, simple_loss=0.07857, pruned_loss=0.01102, audio_tagging_loss=0.009343, over 15433.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09197, pruned_loss=0.01377, audio_tagging_loss=0.009121, over 3040405.34 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:16:32,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2340266.6666666665, ans=0.125
2023-11-23 10:16:38,041 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351050
2023-11-23 10:16:57,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2340400.0, ans=0.1
2023-11-23 10:16:57,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2340400.0, ans=0.0
2023-11-23 10:17:19,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2340533.3333333335, ans=0.0
2023-11-23 10:17:33,015 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2400, loss[loss=0.03664, simple_loss=0.04081, pruned_loss=0.002174, audio_tagging_loss=0.01406, over 14209.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09256, pruned_loss=0.01393, audio_tagging_loss=0.009176, over 3036365.64 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:17:42,687 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351100
2023-11-23 10:17:52,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2340666.6666666665, ans=0.0
2023-11-23 10:18:04,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2340733.3333333335, ans=0.1
2023-11-23 10:18:11,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2340800.0, ans=0.125
2023-11-23 10:18:11,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2340800.0, ans=0.0
2023-11-23 10:18:15,896 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.630e+01 8.266e+01 8.725e+01 9.443e+01 1.197e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-23 10:18:20,517 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:18:25,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2340866.6666666665, ans=0.0
2023-11-23 10:18:26,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2340866.6666666665, ans=0.04949747468305833
2023-11-23 10:18:34,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2340866.6666666665, ans=0.1
2023-11-23 10:18:36,487 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2450, loss[loss=0.06796, simple_loss=0.08371, pruned_loss=0.01593, audio_tagging_loss=0.01017, over 13907.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09175, pruned_loss=0.01381, audio_tagging_loss=0.009238, over 3036334.06 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:18:46,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351150
2023-11-23 10:18:50,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2341000.0, ans=0.1
2023-11-23 10:19:05,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2341066.6666666665, ans=0.125
2023-11-23 10:19:41,855 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2500, loss[loss=0.06705, simple_loss=0.08165, pruned_loss=0.01496, audio_tagging_loss=0.01126, over 15264.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.0917, pruned_loss=0.01388, audio_tagging_loss=0.009253, over 3039345.86 frames. ], batch size: 58, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:19:52,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2341266.6666666665, ans=0.1
2023-11-23 10:19:53,980 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351200
2023-11-23 10:20:05,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.28 vs. limit=22.5
2023-11-23 10:20:27,207 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.94 vs. limit=15.0
2023-11-23 10:20:27,908 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.304e+01 8.850e+01 9.688e+01 1.424e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 10:20:51,098 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2550, loss[loss=0.06381, simple_loss=0.08265, pruned_loss=0.0145, audio_tagging_loss=0.007979, over 13816.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09247, pruned_loss=0.01419, audio_tagging_loss=0.009189, over 3041640.15 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:20:56,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2341600.0, ans=0.125
2023-11-23 10:21:01,173 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351250
2023-11-23 10:21:12,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2341666.6666666665, ans=0.05
2023-11-23 10:21:35,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2341800.0, ans=0.0
2023-11-23 10:21:36,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2341800.0, ans=0.125
2023-11-23 10:21:45,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2341866.6666666665, ans=0.0
2023-11-23 10:21:47,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2341866.6666666665, ans=0.125
2023-11-23 10:21:56,978 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2600, loss[loss=0.06992, simple_loss=0.0928, pruned_loss=0.01471, audio_tagging_loss=0.008811, over 15341.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09179, pruned_loss=0.01399, audio_tagging_loss=0.009162, over 3036305.21 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:22:06,954 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351300
2023-11-23 10:22:14,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2342000.0, ans=0.125
2023-11-23 10:22:42,344 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.380e+01 9.027e+01 9.591e+01 1.211e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 10:22:47,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2342133.3333333335, ans=0.0
2023-11-23 10:23:02,641 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2650, loss[loss=0.08228, simple_loss=0.1101, pruned_loss=0.02025, audio_tagging_loss=0.006967, over 14132.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09165, pruned_loss=0.0139, audio_tagging_loss=0.009103, over 3032576.67 frames. ], batch size: 55, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:23:08,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2342266.6666666665, ans=0.125
2023-11-23 10:23:11,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2342266.6666666665, ans=0.0
2023-11-23 10:23:14,173 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351350
2023-11-23 10:23:55,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2342533.3333333335, ans=0.0
2023-11-23 10:24:09,562 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2700, loss[loss=0.05745, simple_loss=0.06529, pruned_loss=0.01274, audio_tagging_loss=0.01207, over 14824.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.0914, pruned_loss=0.01395, audio_tagging_loss=0.009048, over 3041819.87 frames. ], batch size: 57, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:24:20,921 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351400
2023-11-23 10:24:32,849 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.41 vs. limit=12.0
2023-11-23 10:24:35,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2342733.3333333335, ans=0.0
2023-11-23 10:24:44,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2342733.3333333335, ans=0.125
2023-11-23 10:24:46,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2342733.3333333335, ans=0.125
2023-11-23 10:24:55,400 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.154e+01 8.358e+01 8.964e+01 9.971e+01 1.230e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 10:25:15,074 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2750, loss[loss=0.06826, simple_loss=0.09395, pruned_loss=0.01207, audio_tagging_loss=0.009211, over 14876.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09259, pruned_loss=0.01427, audio_tagging_loss=0.008949, over 3040547.37 frames. ], batch size: 56, lr: 2.28e-03, grad_scale: 16.0
2023-11-23 10:25:18,240 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.65 vs. limit=15.0
2023-11-23 10:25:24,986 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351450
2023-11-23 10:25:41,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2343066.6666666665, ans=0.0
2023-11-23 10:26:12,105 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:26:19,259 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2800, loss[loss=0.06572, simple_loss=0.08171, pruned_loss=0.01381, audio_tagging_loss=0.01106, over 16017.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09201, pruned_loss=0.01424, audio_tagging_loss=0.008909, over 3043294.29 frames. ], batch size: 63, lr: 2.28e-03, grad_scale: 32.0
2023-11-23 10:26:30,653 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351500
2023-11-23 10:26:57,134 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.08 vs. limit=15.0
2023-11-23 10:27:05,085 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.185e+01 8.850e+01 9.511e+01 1.118e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 10:27:13,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.27 vs. limit=10.0
2023-11-23 10:27:24,911 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2850, loss[loss=0.06921, simple_loss=0.09628, pruned_loss=0.01199, audio_tagging_loss=0.009074, over 15960.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09135, pruned_loss=0.01403, audio_tagging_loss=0.008859, over 3037607.51 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:27:30,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2343600.0, ans=0.0
2023-11-23 10:27:35,435 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351550
2023-11-23 10:27:37,162 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.79 vs. limit=15.0
2023-11-23 10:27:44,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2343666.6666666665, ans=0.125
2023-11-23 10:27:49,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2343733.3333333335, ans=0.125
2023-11-23 10:28:01,464 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.66 vs. limit=15.0
2023-11-23 10:28:05,563 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0
2023-11-23 10:28:26,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2343866.6666666665, ans=0.1
2023-11-23 10:28:27,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2343866.6666666665, ans=0.0
2023-11-23 10:28:29,911 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2900, loss[loss=0.07475, simple_loss=0.09542, pruned_loss=0.01557, audio_tagging_loss=0.01147, over 14633.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09237, pruned_loss=0.01418, audio_tagging_loss=0.008935, over 3040262.41 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:28:33,887 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.12 vs. limit=6.0
2023-11-23 10:28:40,793 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351600
2023-11-23 10:29:00,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.58 vs. limit=6.0
2023-11-23 10:29:06,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2344066.6666666665, ans=0.125
2023-11-23 10:29:07,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2344066.6666666665, ans=0.2
2023-11-23 10:29:17,048 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.267e+01 8.374e+01 8.898e+01 9.788e+01 1.211e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 10:29:18,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2344133.3333333335, ans=0.2
2023-11-23 10:29:30,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2344200.0, ans=0.125
2023-11-23 10:29:30,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.47 vs. limit=15.0
2023-11-23 10:29:36,346 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 2950, loss[loss=0.05881, simple_loss=0.0757, pruned_loss=0.01144, audio_tagging_loss=0.00952, over 15141.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09144, pruned_loss=0.01387, audio_tagging_loss=0.009049, over 3039015.27 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:29:47,256 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351650
2023-11-23 10:30:18,350 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.36 vs. limit=15.0
2023-11-23 10:30:36,019 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.90 vs. limit=15.0
2023-11-23 10:30:42,045 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3000, loss[loss=0.05815, simple_loss=0.07813, pruned_loss=0.00891, audio_tagging_loss=0.01017, over 14195.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09257, pruned_loss=0.01405, audio_tagging_loss=0.009081, over 3034351.83 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:30:42,048 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 10:31:09,927 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([4.4541, 3.7886, 4.3631, 3.3426], device='cuda:0')
2023-11-23 10:31:20,386 INFO [train_asr.py:1253] (0/4) Epoch 30, validation: loss=0.05789, simple_loss=0.05111, pruned_loss=0.005034, audio_tagging_loss=0.0273, over 4681554.00 frames. 
2023-11-23 10:31:20,387 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 10:31:24,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2344600.0, ans=0.0
2023-11-23 10:31:31,350 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351700
2023-11-23 10:31:36,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2344666.6666666665, ans=0.2
2023-11-23 10:32:08,869 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.435e+01 9.100e+01 1.011e+02 1.193e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-23 10:32:25,960 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3050, loss[loss=0.05025, simple_loss=0.05852, pruned_loss=0.00977, audio_tagging_loss=0.01122, over 16078.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09346, pruned_loss=0.01407, audio_tagging_loss=0.0091, over 3037087.77 frames. ], batch size: 61, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:32:32,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2344933.3333333335, ans=0.05
2023-11-23 10:32:37,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351750
2023-11-23 10:32:40,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2345000.0, ans=0.125
2023-11-23 10:32:42,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2345000.0, ans=0.125
2023-11-23 10:32:54,725 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.31 vs. limit=15.0
2023-11-23 10:33:04,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2345133.3333333335, ans=0.125
2023-11-23 10:33:06,570 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:33:22,923 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.57 vs. limit=22.5
2023-11-23 10:33:27,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2345200.0, ans=0.125
2023-11-23 10:33:32,030 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3100, loss[loss=0.09789, simple_loss=0.1278, pruned_loss=0.02521, audio_tagging_loss=0.008758, over 14702.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09358, pruned_loss=0.01412, audio_tagging_loss=0.009144, over 3041524.62 frames. ], batch size: 53, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:33:42,766 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351800
2023-11-23 10:34:21,255 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.167e+01 8.412e+01 9.012e+01 9.624e+01 1.358e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 10:34:26,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2345533.3333333335, ans=0.125
2023-11-23 10:34:26,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2345533.3333333335, ans=0.2
2023-11-23 10:34:37,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2345600.0, ans=0.1
2023-11-23 10:34:38,153 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3150, loss[loss=0.05938, simple_loss=0.07728, pruned_loss=0.0131, audio_tagging_loss=0.007638, over 14912.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.09256, pruned_loss=0.01409, audio_tagging_loss=0.009239, over 3034812.81 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:34:46,383 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.78 vs. limit=22.5
2023-11-23 10:34:48,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351850
2023-11-23 10:35:04,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2345733.3333333335, ans=0.125
2023-11-23 10:35:10,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2345733.3333333335, ans=0.0
2023-11-23 10:35:33,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2345866.6666666665, ans=0.07
2023-11-23 10:35:34,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2345866.6666666665, ans=0.2
2023-11-23 10:35:43,488 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3200, loss[loss=0.08506, simple_loss=0.1155, pruned_loss=0.02066, audio_tagging_loss=0.006656, over 14724.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09277, pruned_loss=0.01405, audio_tagging_loss=0.009375, over 3038153.41 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:35:49,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2345933.3333333335, ans=0.0
2023-11-23 10:35:54,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351900
2023-11-23 10:36:14,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2346066.6666666665, ans=0.0
2023-11-23 10:36:16,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2346066.6666666665, ans=0.04949747468305833
2023-11-23 10:36:31,824 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.291e+01 8.179e+01 8.758e+01 9.576e+01 1.199e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 10:36:35,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2346200.0, ans=0.125
2023-11-23 10:36:38,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.42 vs. limit=15.0
2023-11-23 10:36:48,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2346266.6666666665, ans=0.125
2023-11-23 10:36:49,766 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3250, loss[loss=0.08561, simple_loss=0.1009, pruned_loss=0.02145, audio_tagging_loss=0.01372, over 14073.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09185, pruned_loss=0.01386, audio_tagging_loss=0.009472, over 3040520.14 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:37:00,317 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 351950
2023-11-23 10:37:13,197 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.97 vs. limit=15.0
2023-11-23 10:37:17,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2346400.0, ans=0.0
2023-11-23 10:37:20,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2346400.0, ans=0.1
2023-11-23 10:37:41,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2346533.3333333335, ans=0.0
2023-11-23 10:37:44,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.27 vs. limit=15.0
2023-11-23 10:37:54,539 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3300, loss[loss=0.0927, simple_loss=0.1254, pruned_loss=0.02054, audio_tagging_loss=0.009468, over 14774.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09259, pruned_loss=0.01405, audio_tagging_loss=0.009513, over 3046548.20 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:38:02,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2346600.0, ans=0.125
2023-11-23 10:38:04,393 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352000
2023-11-23 10:38:05,852 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-352000.pt
2023-11-23 10:38:14,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2346666.6666666665, ans=0.1
2023-11-23 10:38:31,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2346733.3333333335, ans=0.125
2023-11-23 10:38:46,039 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.425e+01 8.301e+01 8.918e+01 9.607e+01 1.152e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 10:38:47,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2346800.0, ans=0.125
2023-11-23 10:38:50,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2346866.6666666665, ans=0.0
2023-11-23 10:38:55,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2346866.6666666665, ans=0.1
2023-11-23 10:39:02,153 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3350, loss[loss=0.05663, simple_loss=0.07588, pruned_loss=0.009553, audio_tagging_loss=0.009138, over 14743.00 frames. ], tot_loss[loss=0.07035, simple_loss=0.09348, pruned_loss=0.01421, audio_tagging_loss=0.009406, over 3045510.56 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:39:12,930 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352050
2023-11-23 10:39:19,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2347000.0, ans=0.2
2023-11-23 10:39:19,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.15 vs. limit=15.0
2023-11-23 10:39:24,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2347000.0, ans=0.1
2023-11-23 10:39:29,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2347066.6666666665, ans=0.0
2023-11-23 10:39:33,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2347066.6666666665, ans=0.0
2023-11-23 10:39:38,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2347066.6666666665, ans=0.125
2023-11-23 10:39:49,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2347133.3333333335, ans=0.0
2023-11-23 10:39:50,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2347133.3333333335, ans=0.125
2023-11-23 10:39:52,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2347133.3333333335, ans=0.125
2023-11-23 10:40:08,314 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3400, loss[loss=0.06902, simple_loss=0.1009, pruned_loss=0.01163, audio_tagging_loss=0.006921, over 15546.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09299, pruned_loss=0.0141, audio_tagging_loss=0.009345, over 3041364.79 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:40:19,069 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352100
2023-11-23 10:40:39,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2347400.0, ans=0.0
2023-11-23 10:40:42,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2347400.0, ans=0.125
2023-11-23 10:40:42,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2347400.0, ans=0.125
2023-11-23 10:40:43,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2347400.0, ans=0.125
2023-11-23 10:40:47,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2347466.6666666665, ans=0.125
2023-11-23 10:40:48,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2347466.6666666665, ans=0.0
2023-11-23 10:40:56,362 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.131e+01 8.875e+01 9.563e+01 1.133e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 10:40:59,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2347533.3333333335, ans=0.0
2023-11-23 10:41:12,946 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3450, loss[loss=0.06385, simple_loss=0.09201, pruned_loss=0.0113, audio_tagging_loss=0.006549, over 14588.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09298, pruned_loss=0.01394, audio_tagging_loss=0.009195, over 3044058.42 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:41:15,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2347600.0, ans=0.125
2023-11-23 10:41:21,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2347600.0, ans=0.5
2023-11-23 10:41:23,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352150
2023-11-23 10:41:26,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2347666.6666666665, ans=0.0
2023-11-23 10:41:55,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2347800.0, ans=0.1
2023-11-23 10:42:16,601 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3500, loss[loss=0.07702, simple_loss=0.1109, pruned_loss=0.0152, audio_tagging_loss=0.006378, over 15023.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09266, pruned_loss=0.01392, audio_tagging_loss=0.009158, over 3046084.41 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:42:26,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352200
2023-11-23 10:42:52,034 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:42:52,762 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.30 vs. limit=22.5
2023-11-23 10:43:04,276 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.439e+01 8.164e+01 8.763e+01 9.572e+01 1.144e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-23 10:43:09,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2348200.0, ans=0.125
2023-11-23 10:43:10,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2348200.0, ans=0.1
2023-11-23 10:43:14,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2348200.0, ans=0.125
2023-11-23 10:43:20,957 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3550, loss[loss=0.07044, simple_loss=0.09966, pruned_loss=0.01378, audio_tagging_loss=0.006829, over 15047.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09202, pruned_loss=0.0137, audio_tagging_loss=0.009141, over 3047442.39 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:43:31,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352250
2023-11-23 10:43:32,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2348266.6666666665, ans=0.125
2023-11-23 10:43:58,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2348466.6666666665, ans=0.0
2023-11-23 10:44:02,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2348466.6666666665, ans=0.025
2023-11-23 10:44:05,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2348466.6666666665, ans=0.125
2023-11-23 10:44:09,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2348466.6666666665, ans=0.1
2023-11-23 10:44:14,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2348533.3333333335, ans=0.1
2023-11-23 10:44:23,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2348533.3333333335, ans=0.2
2023-11-23 10:44:25,957 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3600, loss[loss=0.07515, simple_loss=0.09371, pruned_loss=0.01762, audio_tagging_loss=0.01068, over 14804.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09217, pruned_loss=0.01387, audio_tagging_loss=0.009089, over 3039990.24 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:44:27,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2348600.0, ans=0.1
2023-11-23 10:44:35,767 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352300
2023-11-23 10:44:38,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2348666.6666666665, ans=0.2
2023-11-23 10:44:44,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2348666.6666666665, ans=0.125
2023-11-23 10:44:46,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2348666.6666666665, ans=0.125
2023-11-23 10:44:49,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2348733.3333333335, ans=0.5
2023-11-23 10:45:13,762 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.915e+01 8.138e+01 8.784e+01 9.716e+01 1.349e+02, threshold=1.757e+02, percent-clipped=0.0
2023-11-23 10:45:19,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2348866.6666666665, ans=0.125
2023-11-23 10:45:19,485 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.34 vs. limit=10.0
2023-11-23 10:45:20,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2348866.6666666665, ans=0.025
2023-11-23 10:45:23,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.77 vs. limit=6.0
2023-11-23 10:45:29,832 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3650, loss[loss=0.06266, simple_loss=0.08513, pruned_loss=0.01138, audio_tagging_loss=0.008718, over 16112.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09263, pruned_loss=0.01406, audio_tagging_loss=0.009086, over 3049042.85 frames. ], batch size: 63, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:45:32,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2348933.3333333335, ans=0.0
2023-11-23 10:45:38,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2348933.3333333335, ans=0.5
2023-11-23 10:45:39,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0
2023-11-23 10:45:39,972 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352350
2023-11-23 10:45:41,768 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.45 vs. limit=15.0
2023-11-23 10:45:58,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2349066.6666666665, ans=0.125
2023-11-23 10:46:00,053 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.63 vs. limit=12.0
2023-11-23 10:46:11,608 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.89 vs. limit=15.0
2023-11-23 10:46:13,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2349133.3333333335, ans=0.0
2023-11-23 10:46:34,677 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3700, loss[loss=0.07413, simple_loss=0.0941, pruned_loss=0.01771, audio_tagging_loss=0.009374, over 15826.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09326, pruned_loss=0.01424, audio_tagging_loss=0.009001, over 3048207.42 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:46:42,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2349266.6666666665, ans=0.125
2023-11-23 10:46:46,496 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352400
2023-11-23 10:47:11,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2349400.0, ans=0.1
2023-11-23 10:47:23,651 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.475e+01 8.984e+01 9.756e+01 1.281e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 10:47:34,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2349533.3333333335, ans=0.0
2023-11-23 10:47:42,662 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3750, loss[loss=0.09061, simple_loss=0.1252, pruned_loss=0.01945, audio_tagging_loss=0.008568, over 15046.00 frames. ], tot_loss[loss=0.07015, simple_loss=0.09394, pruned_loss=0.01419, audio_tagging_loss=0.008993, over 3049359.55 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:47:44,335 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:47:48,867 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.38 vs. limit=6.0
2023-11-23 10:47:53,034 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352450
2023-11-23 10:47:54,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2349666.6666666665, ans=0.0
2023-11-23 10:47:58,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2349666.6666666665, ans=0.1
2023-11-23 10:47:58,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2349666.6666666665, ans=0.05
2023-11-23 10:48:09,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2349733.3333333335, ans=0.2
2023-11-23 10:48:10,490 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.23 vs. limit=12.0
2023-11-23 10:48:31,262 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:48:43,929 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-23 10:48:44,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2349866.6666666665, ans=0.07
2023-11-23 10:48:49,450 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3800, loss[loss=0.08057, simple_loss=0.1146, pruned_loss=0.01568, audio_tagging_loss=0.007615, over 14943.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09327, pruned_loss=0.01399, audio_tagging_loss=0.009076, over 3043998.36 frames. ], batch size: 53, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:48:53,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2349933.3333333335, ans=0.0
2023-11-23 10:48:56,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2349933.3333333335, ans=0.025
2023-11-23 10:48:59,653 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352500
2023-11-23 10:49:03,601 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:49:06,680 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.72 vs. limit=6.0
2023-11-23 10:49:10,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2350000.0, ans=0.0
2023-11-23 10:49:30,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2350133.3333333335, ans=0.0
2023-11-23 10:49:38,114 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.155e+01 8.465e+01 8.919e+01 9.665e+01 1.243e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 10:49:51,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2350200.0, ans=0.125
2023-11-23 10:49:55,003 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3850, loss[loss=0.05078, simple_loss=0.06249, pruned_loss=0.009851, audio_tagging_loss=0.009686, over 16088.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.0931, pruned_loss=0.01412, audio_tagging_loss=0.009132, over 3039665.25 frames. ], batch size: 63, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 10:49:56,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2350266.6666666665, ans=0.0
2023-11-23 10:49:57,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2350266.6666666665, ans=0.125
2023-11-23 10:49:57,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2350266.6666666665, ans=0.125
2023-11-23 10:50:04,498 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.48 vs. limit=12.0
2023-11-23 10:50:06,556 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352550
2023-11-23 10:50:06,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2350266.6666666665, ans=0.0
2023-11-23 10:51:02,867 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3900, loss[loss=0.07565, simple_loss=0.1081, pruned_loss=0.01376, audio_tagging_loss=0.007815, over 15988.00 frames. ], tot_loss[loss=0.06988, simple_loss=0.09334, pruned_loss=0.01406, audio_tagging_loss=0.009143, over 3039401.54 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:51:09,604 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:51:13,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2350600.0, ans=0.125
2023-11-23 10:51:14,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352600
2023-11-23 10:51:39,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2350733.3333333335, ans=0.1
2023-11-23 10:51:39,529 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.87 vs. limit=15.0
2023-11-23 10:51:55,079 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.057e+01 8.310e+01 8.820e+01 9.647e+01 1.440e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 10:52:10,444 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 3950, loss[loss=0.0659, simple_loss=0.0799, pruned_loss=0.01459, audio_tagging_loss=0.01136, over 13780.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09247, pruned_loss=0.01403, audio_tagging_loss=0.009377, over 3045572.65 frames. ], batch size: 53, lr: 2.27e-03, grad_scale: 8.0
2023-11-23 10:52:14,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2350933.3333333335, ans=0.125
2023-11-23 10:52:20,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352650
2023-11-23 10:53:09,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.75 vs. limit=22.5
2023-11-23 10:53:16,338 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4000, loss[loss=0.0881, simple_loss=0.1191, pruned_loss=0.01819, audio_tagging_loss=0.01036, over 15216.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09195, pruned_loss=0.01393, audio_tagging_loss=0.00945, over 3042789.50 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:53:17,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2351266.6666666665, ans=0.125
2023-11-23 10:53:20,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2351266.6666666665, ans=0.1
2023-11-23 10:53:27,933 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352700
2023-11-23 10:53:42,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2351400.0, ans=0.0
2023-11-23 10:53:53,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.16 vs. limit=12.0
2023-11-23 10:53:57,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2351466.6666666665, ans=0.125
2023-11-23 10:53:57,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2351466.6666666665, ans=0.0
2023-11-23 10:54:08,346 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.731e+01 8.440e+01 9.076e+01 9.924e+01 2.102e+02, threshold=1.815e+02, percent-clipped=1.0
2023-11-23 10:54:09,189 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.29 vs. limit=15.0
2023-11-23 10:54:23,989 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4050, loss[loss=0.05138, simple_loss=0.06468, pruned_loss=0.005974, audio_tagging_loss=0.01307, over 14979.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09143, pruned_loss=0.01365, audio_tagging_loss=0.009466, over 3041692.96 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:54:26,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2351600.0, ans=0.0
2023-11-23 10:54:27,767 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:54:31,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2351600.0, ans=0.125
2023-11-23 10:54:34,877 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352750
2023-11-23 10:54:42,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2351666.6666666665, ans=0.2
2023-11-23 10:54:42,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2351666.6666666665, ans=0.1
2023-11-23 10:55:12,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2351800.0, ans=0.125
2023-11-23 10:55:24,081 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.82 vs. limit=15.0
2023-11-23 10:55:30,784 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4100, loss[loss=0.06755, simple_loss=0.08288, pruned_loss=0.01618, audio_tagging_loss=0.009928, over 14400.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09133, pruned_loss=0.01362, audio_tagging_loss=0.009337, over 3042585.85 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:55:41,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352800
2023-11-23 10:55:45,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2352000.0, ans=0.125
2023-11-23 10:55:51,060 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.90 vs. limit=12.0
2023-11-23 10:55:54,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2352000.0, ans=0.2
2023-11-23 10:56:03,157 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.70 vs. limit=15.0
2023-11-23 10:56:22,981 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.387e+01 8.886e+01 9.725e+01 1.263e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 10:56:37,361 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4150, loss[loss=0.07845, simple_loss=0.1053, pruned_loss=0.01392, audio_tagging_loss=0.01186, over 14891.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.0915, pruned_loss=0.01366, audio_tagging_loss=0.009233, over 3043832.28 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:56:37,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2352266.6666666665, ans=0.125
2023-11-23 10:56:41,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2352266.6666666665, ans=0.2
2023-11-23 10:56:47,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352850
2023-11-23 10:56:51,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2352333.3333333335, ans=0.0
2023-11-23 10:56:55,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2352333.3333333335, ans=0.2
2023-11-23 10:57:25,916 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 10:57:38,077 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 10:57:39,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2352533.3333333335, ans=0.125
2023-11-23 10:57:42,842 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.23 vs. limit=22.5
2023-11-23 10:57:43,202 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4200, loss[loss=0.07071, simple_loss=0.09785, pruned_loss=0.01559, audio_tagging_loss=0.006194, over 15825.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09178, pruned_loss=0.0137, audio_tagging_loss=0.009129, over 3048468.12 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:57:44,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2352600.0, ans=0.125
2023-11-23 10:57:51,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2352600.0, ans=0.125
2023-11-23 10:57:53,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352900
2023-11-23 10:58:01,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2352666.6666666665, ans=0.125
2023-11-23 10:58:27,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2352800.0, ans=0.2
2023-11-23 10:58:34,393 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.262e+01 8.280e+01 9.153e+01 9.878e+01 1.174e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-23 10:58:48,837 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4250, loss[loss=0.07871, simple_loss=0.1017, pruned_loss=0.02004, audio_tagging_loss=0.007815, over 14289.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09274, pruned_loss=0.0139, audio_tagging_loss=0.009063, over 3049394.71 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 10:58:59,488 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 352950
2023-11-23 10:59:13,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.68 vs. limit=22.5
2023-11-23 10:59:25,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2353066.6666666665, ans=0.125
2023-11-23 10:59:54,227 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4300, loss[loss=0.08851, simple_loss=0.1274, pruned_loss=0.01794, audio_tagging_loss=0.006858, over 17373.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09293, pruned_loss=0.01384, audio_tagging_loss=0.009014, over 3054386.82 frames. ], batch size: 62, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:00:04,390 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353000
2023-11-23 11:00:04,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2353266.6666666665, ans=0.07
2023-11-23 11:00:15,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2353333.3333333335, ans=0.2
2023-11-23 11:00:25,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2353400.0, ans=0.1
2023-11-23 11:00:45,969 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.099e+01 8.412e+01 9.137e+01 9.657e+01 1.208e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-23 11:01:00,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2353600.0, ans=0.0
2023-11-23 11:01:00,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2353600.0, ans=0.125
2023-11-23 11:01:01,197 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4350, loss[loss=0.09628, simple_loss=0.1279, pruned_loss=0.02582, audio_tagging_loss=0.006499, over 14200.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09291, pruned_loss=0.01382, audio_tagging_loss=0.009017, over 3048011.94 frames. ], batch size: 53, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:01:07,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2353600.0, ans=0.125
2023-11-23 11:01:11,773 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353050
2023-11-23 11:01:12,334 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.74 vs. limit=15.0
2023-11-23 11:01:13,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2353666.6666666665, ans=0.1
2023-11-23 11:01:55,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2353866.6666666665, ans=0.125
2023-11-23 11:02:07,037 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4400, loss[loss=0.06971, simple_loss=0.08914, pruned_loss=0.01509, audio_tagging_loss=0.01005, over 15793.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09269, pruned_loss=0.01385, audio_tagging_loss=0.009061, over 3043046.79 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:02:10,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2353933.3333333335, ans=0.0
2023-11-23 11:02:17,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353100
2023-11-23 11:02:22,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2354000.0, ans=0.125
2023-11-23 11:02:28,774 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.88 vs. limit=15.0
2023-11-23 11:02:40,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2354066.6666666665, ans=0.2
2023-11-23 11:02:41,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2354066.6666666665, ans=0.2
2023-11-23 11:02:47,609 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.13 vs. limit=15.0
2023-11-23 11:02:49,547 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:02:58,599 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.258e+01 8.734e+01 9.523e+01 1.170e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-23 11:03:04,511 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0
2023-11-23 11:03:12,648 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4450, loss[loss=0.08909, simple_loss=0.1124, pruned_loss=0.02516, audio_tagging_loss=0.007731, over 14384.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09349, pruned_loss=0.01398, audio_tagging_loss=0.008922, over 3051228.81 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:03:23,295 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353150
2023-11-23 11:03:25,303 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.17 vs. limit=10.0
2023-11-23 11:03:54,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2354466.6666666665, ans=0.09899494936611666
2023-11-23 11:03:54,364 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.30 vs. limit=6.0
2023-11-23 11:03:59,365 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.14 vs. limit=15.0
2023-11-23 11:04:01,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2354466.6666666665, ans=0.0
2023-11-23 11:04:04,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2354533.3333333335, ans=0.2
2023-11-23 11:04:04,462 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.90 vs. limit=15.0
2023-11-23 11:04:06,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2354533.3333333335, ans=0.125
2023-11-23 11:04:10,274 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.82 vs. limit=15.0
2023-11-23 11:04:18,934 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4500, loss[loss=0.06196, simple_loss=0.08036, pruned_loss=0.01187, audio_tagging_loss=0.009908, over 14476.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09358, pruned_loss=0.01388, audio_tagging_loss=0.008841, over 3055461.54 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:04:23,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2354600.0, ans=0.125
2023-11-23 11:04:29,738 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353200
2023-11-23 11:04:51,947 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.18 vs. limit=15.0
2023-11-23 11:05:03,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2354800.0, ans=0.1
2023-11-23 11:05:08,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2354800.0, ans=0.05
2023-11-23 11:05:10,689 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.746e+01 8.361e+01 8.894e+01 9.678e+01 1.285e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 11:05:11,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2354866.6666666665, ans=0.1
2023-11-23 11:05:25,340 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4550, loss[loss=0.07024, simple_loss=0.08547, pruned_loss=0.01334, audio_tagging_loss=0.01416, over 13755.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.0939, pruned_loss=0.01401, audio_tagging_loss=0.008852, over 3049471.04 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:05:32,446 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0
2023-11-23 11:05:35,597 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353250
2023-11-23 11:05:58,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2355066.6666666665, ans=0.0
2023-11-23 11:06:06,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2355133.3333333335, ans=0.0
2023-11-23 11:06:16,978 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 11:06:30,726 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4600, loss[loss=0.07322, simple_loss=0.08963, pruned_loss=0.01881, audio_tagging_loss=0.009596, over 14930.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09316, pruned_loss=0.01391, audio_tagging_loss=0.009037, over 3055093.60 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:06:40,942 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353300
2023-11-23 11:06:45,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2355333.3333333335, ans=0.1
2023-11-23 11:06:46,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2355333.3333333335, ans=0.125
2023-11-23 11:06:49,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.37 vs. limit=6.0
2023-11-23 11:06:52,245 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.35 vs. limit=5.0
2023-11-23 11:07:10,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2355466.6666666665, ans=0.125
2023-11-23 11:07:17,177 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.99 vs. limit=6.0
2023-11-23 11:07:22,705 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.086e+01 8.374e+01 9.113e+01 9.738e+01 1.636e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-23 11:07:29,383 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:07:35,299 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4650, loss[loss=0.08581, simple_loss=0.1114, pruned_loss=0.02136, audio_tagging_loss=0.008728, over 14985.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09225, pruned_loss=0.01378, audio_tagging_loss=0.009182, over 3052136.13 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:07:45,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2355600.0, ans=10.0
2023-11-23 11:07:47,192 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353350
2023-11-23 11:07:54,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2355666.6666666665, ans=0.125
2023-11-23 11:07:55,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2355666.6666666665, ans=0.0
2023-11-23 11:08:42,522 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4700, loss[loss=0.07263, simple_loss=0.1008, pruned_loss=0.01407, audio_tagging_loss=0.008177, over 14666.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09271, pruned_loss=0.01397, audio_tagging_loss=0.009309, over 3057497.04 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:08:52,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353400
2023-11-23 11:08:52,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2355933.3333333335, ans=0.0
2023-11-23 11:08:55,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2356000.0, ans=0.125
2023-11-23 11:08:55,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2356000.0, ans=10.0
2023-11-23 11:08:55,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2356000.0, ans=0.0
2023-11-23 11:09:09,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2356066.6666666665, ans=0.0
2023-11-23 11:09:16,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2356066.6666666665, ans=0.0
2023-11-23 11:09:17,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.23 vs. limit=15.0
2023-11-23 11:09:24,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2356133.3333333335, ans=0.1
2023-11-23 11:09:35,051 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.787e+01 8.186e+01 8.702e+01 9.591e+01 1.216e+02, threshold=1.740e+02, percent-clipped=0.0
2023-11-23 11:09:42,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2356200.0, ans=0.125
2023-11-23 11:09:45,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2356200.0, ans=0.125
2023-11-23 11:09:47,648 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4750, loss[loss=0.0577, simple_loss=0.07256, pruned_loss=0.01061, audio_tagging_loss=0.01081, over 15865.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09218, pruned_loss=0.014, audio_tagging_loss=0.009341, over 3051239.45 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:09:57,670 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353450
2023-11-23 11:10:29,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2356466.6666666665, ans=0.0
2023-11-23 11:10:52,099 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4800, loss[loss=0.08233, simple_loss=0.1088, pruned_loss=0.01761, audio_tagging_loss=0.0103, over 14783.00 frames. ], tot_loss[loss=0.06984, simple_loss=0.0927, pruned_loss=0.01414, audio_tagging_loss=0.009351, over 3042767.64 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:11:04,010 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353500
2023-11-23 11:11:09,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2356666.6666666665, ans=0.125
2023-11-23 11:11:16,653 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.69 vs. limit=15.0
2023-11-23 11:11:44,951 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.702e+01 8.167e+01 8.824e+01 9.550e+01 1.189e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 11:11:59,424 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4850, loss[loss=0.068, simple_loss=0.09579, pruned_loss=0.01013, audio_tagging_loss=0.009973, over 14792.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09233, pruned_loss=0.01399, audio_tagging_loss=0.009393, over 3042308.75 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:12:07,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2356933.3333333335, ans=0.125
2023-11-23 11:12:10,081 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353550
2023-11-23 11:12:21,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2357000.0, ans=0.125
2023-11-23 11:12:23,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2357066.6666666665, ans=0.2
2023-11-23 11:12:40,025 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.07 vs. limit=6.0
2023-11-23 11:12:48,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2357133.3333333335, ans=0.125
2023-11-23 11:12:49,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2357133.3333333335, ans=0.125
2023-11-23 11:12:57,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2357200.0, ans=0.1
2023-11-23 11:13:04,976 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4900, loss[loss=0.06956, simple_loss=0.1031, pruned_loss=0.01161, audio_tagging_loss=0.006424, over 15096.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09228, pruned_loss=0.01396, audio_tagging_loss=0.009385, over 3044334.86 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:13:15,130 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353600
2023-11-23 11:13:18,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2357333.3333333335, ans=0.125
2023-11-23 11:13:27,169 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.95 vs. limit=15.0
2023-11-23 11:13:57,835 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.869e+01 8.207e+01 8.666e+01 9.499e+01 1.245e+02, threshold=1.733e+02, percent-clipped=0.0
2023-11-23 11:14:06,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2357533.3333333335, ans=0.2
2023-11-23 11:14:10,310 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 4950, loss[loss=0.06578, simple_loss=0.08775, pruned_loss=0.01104, audio_tagging_loss=0.01087, over 17974.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09183, pruned_loss=0.01385, audio_tagging_loss=0.009288, over 3050561.26 frames. ], batch size: 69, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:14:21,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353650
2023-11-23 11:14:44,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2357733.3333333335, ans=0.2
2023-11-23 11:15:09,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2357866.6666666665, ans=0.125
2023-11-23 11:15:17,625 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5000, loss[loss=0.0554, simple_loss=0.06975, pruned_loss=0.009845, audio_tagging_loss=0.01069, over 15205.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.0908, pruned_loss=0.01369, audio_tagging_loss=0.009231, over 3042356.56 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:15:29,531 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353700
2023-11-23 11:15:44,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2358066.6666666665, ans=0.07
2023-11-23 11:15:52,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2358066.6666666665, ans=0.0
2023-11-23 11:15:54,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2358066.6666666665, ans=0.125
2023-11-23 11:16:09,737 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.27 vs. limit=12.0
2023-11-23 11:16:10,294 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.172e+01 8.851e+01 9.559e+01 1.178e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 11:16:24,089 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5050, loss[loss=0.05955, simple_loss=0.07708, pruned_loss=0.009565, audio_tagging_loss=0.01144, over 15152.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09131, pruned_loss=0.0137, audio_tagging_loss=0.009129, over 3042894.03 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:16:24,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2358266.6666666665, ans=0.0
2023-11-23 11:16:26,995 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:16:33,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2358266.6666666665, ans=0.125
2023-11-23 11:16:34,185 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353750
2023-11-23 11:16:36,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2358333.3333333335, ans=0.125
2023-11-23 11:16:38,811 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.21 vs. limit=22.5
2023-11-23 11:16:44,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2358333.3333333335, ans=0.2
2023-11-23 11:17:08,903 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.72 vs. limit=15.0
2023-11-23 11:17:09,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2358466.6666666665, ans=0.0
2023-11-23 11:17:16,200 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:17:29,521 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5100, loss[loss=0.07621, simple_loss=0.1042, pruned_loss=0.01202, audio_tagging_loss=0.0121, over 14332.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09184, pruned_loss=0.01379, audio_tagging_loss=0.009021, over 3041549.70 frames. ], batch size: 53, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:17:40,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353800
2023-11-23 11:17:40,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2358600.0, ans=0.125
2023-11-23 11:17:45,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2358666.6666666665, ans=0.2
2023-11-23 11:17:49,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2358666.6666666665, ans=0.125
2023-11-23 11:18:23,225 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.214e+01 8.798e+01 9.658e+01 1.127e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-23 11:18:24,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2358866.6666666665, ans=0.0
2023-11-23 11:18:35,637 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5150, loss[loss=0.06944, simple_loss=0.09231, pruned_loss=0.01452, audio_tagging_loss=0.008763, over 17067.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09195, pruned_loss=0.01381, audio_tagging_loss=0.008924, over 3047846.71 frames. ], batch size: 67, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:18:36,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2358933.3333333335, ans=0.0
2023-11-23 11:18:46,402 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353850
2023-11-23 11:19:17,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2359133.3333333335, ans=0.0
2023-11-23 11:19:19,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2359133.3333333335, ans=0.125
2023-11-23 11:19:26,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2359133.3333333335, ans=0.1
2023-11-23 11:19:30,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2359200.0, ans=0.125
2023-11-23 11:19:30,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2359200.0, ans=0.0
2023-11-23 11:19:38,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2359200.0, ans=0.0
2023-11-23 11:19:42,045 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5200, loss[loss=0.0621, simple_loss=0.08669, pruned_loss=0.009006, audio_tagging_loss=0.009749, over 15417.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09224, pruned_loss=0.014, audio_tagging_loss=0.008896, over 3048530.96 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:19:52,591 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353900
2023-11-23 11:19:52,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2359266.6666666665, ans=0.0
2023-11-23 11:19:57,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2359333.3333333335, ans=0.125
2023-11-23 11:20:36,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=15.41 vs. limit=15.0
2023-11-23 11:20:36,397 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.853e+01 8.445e+01 9.037e+01 9.922e+01 1.226e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 11:20:47,858 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5250, loss[loss=0.08886, simple_loss=0.1183, pruned_loss=0.02052, audio_tagging_loss=0.009171, over 16434.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.0928, pruned_loss=0.01404, audio_tagging_loss=0.008944, over 3052594.63 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:20:50,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2359600.0, ans=0.125
2023-11-23 11:20:57,913 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 353950
2023-11-23 11:21:19,458 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.29 vs. limit=22.5
2023-11-23 11:21:51,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2359866.6666666665, ans=0.0
2023-11-23 11:21:54,210 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5300, loss[loss=0.08204, simple_loss=0.1039, pruned_loss=0.02127, audio_tagging_loss=0.008825, over 13953.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09257, pruned_loss=0.01398, audio_tagging_loss=0.008953, over 3047143.79 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:22:02,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2359933.3333333335, ans=0.07
2023-11-23 11:22:04,417 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354000
2023-11-23 11:22:16,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2360000.0, ans=0.125
2023-11-23 11:22:18,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2360000.0, ans=0.0
2023-11-23 11:22:23,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.81 vs. limit=15.0
2023-11-23 11:22:24,944 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0
2023-11-23 11:22:49,361 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.002e+01 8.452e+01 9.049e+01 9.694e+01 1.252e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 11:23:00,042 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5350, loss[loss=0.07283, simple_loss=0.1037, pruned_loss=0.0124, audio_tagging_loss=0.008595, over 15318.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09261, pruned_loss=0.01394, audio_tagging_loss=0.009047, over 3044882.37 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:23:10,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354050
2023-11-23 11:23:14,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2360333.3333333335, ans=0.2
2023-11-23 11:23:16,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2360333.3333333335, ans=0.2
2023-11-23 11:23:18,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2360333.3333333335, ans=0.1
2023-11-23 11:23:32,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2360400.0, ans=0.125
2023-11-23 11:23:52,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2360533.3333333335, ans=0.125
2023-11-23 11:24:06,737 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5400, loss[loss=0.08167, simple_loss=0.1133, pruned_loss=0.01713, audio_tagging_loss=0.00789, over 15897.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09307, pruned_loss=0.01396, audio_tagging_loss=0.008991, over 3053303.80 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:24:12,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2360600.0, ans=0.125
2023-11-23 11:24:16,910 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354100
2023-11-23 11:24:28,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2360666.6666666665, ans=0.0
2023-11-23 11:24:42,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2360733.3333333335, ans=0.125
2023-11-23 11:24:55,247 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:25:02,418 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.248e+01 8.837e+01 9.737e+01 1.214e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 11:25:03,074 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.60 vs. limit=15.0
2023-11-23 11:25:07,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2360866.6666666665, ans=0.1
2023-11-23 11:25:12,630 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5450, loss[loss=0.07051, simple_loss=0.1026, pruned_loss=0.009745, audio_tagging_loss=0.009458, over 15381.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.0928, pruned_loss=0.01397, audio_tagging_loss=0.009019, over 3048584.49 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:25:24,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354150
2023-11-23 11:25:46,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2361066.6666666665, ans=0.125
2023-11-23 11:25:46,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2361066.6666666665, ans=0.0
2023-11-23 11:25:49,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2361066.6666666665, ans=0.125
2023-11-23 11:26:11,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2361200.0, ans=0.04949747468305833
2023-11-23 11:26:19,514 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5500, loss[loss=0.07516, simple_loss=0.09871, pruned_loss=0.01486, audio_tagging_loss=0.01095, over 15866.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09386, pruned_loss=0.01406, audio_tagging_loss=0.008961, over 3046681.42 frames. ], batch size: 59, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:26:30,334 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354200
2023-11-23 11:26:37,670 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0
2023-11-23 11:26:40,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2361333.3333333335, ans=0.125
2023-11-23 11:26:43,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2361333.3333333335, ans=0.0
2023-11-23 11:26:43,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2361333.3333333335, ans=0.0
2023-11-23 11:26:51,947 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:26:59,455 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.82 vs. limit=22.5
2023-11-23 11:27:02,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2361466.6666666665, ans=0.125
2023-11-23 11:27:16,268 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.480e+01 8.474e+01 9.083e+01 9.895e+01 1.258e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 11:27:21,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2361533.3333333335, ans=0.0
2023-11-23 11:27:22,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2361533.3333333335, ans=0.2
2023-11-23 11:27:26,425 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5550, loss[loss=0.04444, simple_loss=0.05187, pruned_loss=0.006033, audio_tagging_loss=0.01247, over 15806.00 frames. ], tot_loss[loss=0.07027, simple_loss=0.09408, pruned_loss=0.01421, audio_tagging_loss=0.009018, over 3048874.19 frames. ], batch size: 61, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:27:28,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2361600.0, ans=0.125
2023-11-23 11:27:37,404 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354250
2023-11-23 11:27:57,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2361733.3333333335, ans=0.125
2023-11-23 11:28:32,586 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5600, loss[loss=0.0593, simple_loss=0.07651, pruned_loss=0.01132, audio_tagging_loss=0.009718, over 14153.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09278, pruned_loss=0.01387, audio_tagging_loss=0.009093, over 3045119.25 frames. ], batch size: 54, lr: 2.27e-03, grad_scale: 32.0
2023-11-23 11:28:39,112 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.20 vs. limit=15.0
2023-11-23 11:28:40,327 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.21 vs. limit=15.0
2023-11-23 11:28:43,744 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354300
2023-11-23 11:28:57,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2362000.0, ans=0.125
2023-11-23 11:28:59,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2362066.6666666665, ans=0.125
2023-11-23 11:29:03,869 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.59 vs. limit=15.0
2023-11-23 11:29:20,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2362133.3333333335, ans=0.1
2023-11-23 11:29:21,282 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 11:29:27,450 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.19 vs. limit=15.0
2023-11-23 11:29:28,818 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.99 vs. limit=6.0
2023-11-23 11:29:29,333 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.942e+01 8.297e+01 9.078e+01 9.642e+01 1.395e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 11:29:32,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2362200.0, ans=0.125
2023-11-23 11:29:32,970 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.07 vs. limit=15.0
2023-11-23 11:29:36,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2362200.0, ans=0.125
2023-11-23 11:29:38,677 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5650, loss[loss=0.0731, simple_loss=0.1117, pruned_loss=0.009596, audio_tagging_loss=0.007649, over 15170.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.0918, pruned_loss=0.01358, audio_tagging_loss=0.00925, over 3047782.14 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:29:49,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354350
2023-11-23 11:30:12,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2362400.0, ans=0.125
2023-11-23 11:30:15,656 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.22 vs. limit=22.5
2023-11-23 11:30:44,129 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5700, loss[loss=0.0651, simple_loss=0.09105, pruned_loss=0.01231, audio_tagging_loss=0.007259, over 15442.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09199, pruned_loss=0.01367, audio_tagging_loss=0.009262, over 3044552.17 frames. ], batch size: 56, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:30:47,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0
2023-11-23 11:30:48,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2362600.0, ans=0.125
2023-11-23 11:30:51,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2362600.0, ans=0.125
2023-11-23 11:30:54,334 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354400
2023-11-23 11:30:59,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2362666.6666666665, ans=0.1
2023-11-23 11:31:00,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2362666.6666666665, ans=0.1
2023-11-23 11:31:09,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2362733.3333333335, ans=0.125
2023-11-23 11:31:15,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2362733.3333333335, ans=0.0
2023-11-23 11:31:40,361 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.224e+01 8.276e+01 8.914e+01 9.745e+01 1.403e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 11:31:49,784 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5750, loss[loss=0.04115, simple_loss=0.05308, pruned_loss=0.005451, audio_tagging_loss=0.009159, over 13535.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09108, pruned_loss=0.01356, audio_tagging_loss=0.009164, over 3047544.71 frames. ], batch size: 52, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:31:50,064 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:31:53,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.28 vs. limit=15.0
2023-11-23 11:32:01,150 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354450
2023-11-23 11:32:17,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2363066.6666666665, ans=0.0
2023-11-23 11:32:32,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2363133.3333333335, ans=0.95
2023-11-23 11:32:37,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2363133.3333333335, ans=0.125
2023-11-23 11:32:56,530 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5800, loss[loss=0.05178, simple_loss=0.06794, pruned_loss=0.007855, audio_tagging_loss=0.009954, over 14718.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09144, pruned_loss=0.01373, audio_tagging_loss=0.008934, over 3048025.09 frames. ], batch size: 58, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:33:07,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354500
2023-11-23 11:33:08,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff3.min_abs, batch_count=2363333.3333333335, ans=0.2
2023-11-23 11:33:32,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2363400.0, ans=0.05
2023-11-23 11:33:50,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2363533.3333333335, ans=0.125
2023-11-23 11:33:53,514 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.821e+01 8.322e+01 8.914e+01 9.568e+01 1.180e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 11:34:02,462 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5850, loss[loss=0.06823, simple_loss=0.09275, pruned_loss=0.01584, audio_tagging_loss=0.006015, over 14329.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09126, pruned_loss=0.01369, audio_tagging_loss=0.008958, over 3039670.40 frames. ], batch size: 57, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:34:08,213 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.81 vs. limit=6.0
2023-11-23 11:34:10,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2363600.0, ans=0.0
2023-11-23 11:34:12,391 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354550
2023-11-23 11:34:12,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2363600.0, ans=0.125
2023-11-23 11:34:14,095 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.90 vs. limit=6.0
2023-11-23 11:34:20,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2363666.6666666665, ans=0.0
2023-11-23 11:34:32,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2363733.3333333335, ans=0.0
2023-11-23 11:34:36,759 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.02 vs. limit=10.0
2023-11-23 11:34:38,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2363733.3333333335, ans=0.125
2023-11-23 11:35:06,296 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5900, loss[loss=0.04508, simple_loss=0.05923, pruned_loss=0.00603, audio_tagging_loss=0.009431, over 14262.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09133, pruned_loss=0.01359, audio_tagging_loss=0.008843, over 3038334.77 frames. ], batch size: 55, lr: 2.27e-03, grad_scale: 16.0
2023-11-23 11:35:17,690 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354600
2023-11-23 11:35:25,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2364000.0, ans=0.125
2023-11-23 11:35:29,855 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.92 vs. limit=15.0
2023-11-23 11:35:37,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2364066.6666666665, ans=0.125
2023-11-23 11:35:45,621 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.54 vs. limit=15.0
2023-11-23 11:35:48,348 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=7.05 vs. limit=10.0
2023-11-23 11:36:02,559 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.354e+01 8.917e+01 9.513e+01 1.247e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 11:36:12,773 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 5950, loss[loss=0.05835, simple_loss=0.07938, pruned_loss=0.008521, audio_tagging_loss=0.01014, over 14649.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09174, pruned_loss=0.01374, audio_tagging_loss=0.008814, over 3043107.67 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 11:36:19,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2364266.6666666665, ans=0.0
2023-11-23 11:36:20,782 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.82 vs. limit=15.0
2023-11-23 11:36:23,417 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354650
2023-11-23 11:36:23,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2364266.6666666665, ans=0.125
2023-11-23 11:36:23,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2364266.6666666665, ans=10.0
2023-11-23 11:36:30,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.47 vs. limit=15.0
2023-11-23 11:36:31,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2364333.3333333335, ans=0.2
2023-11-23 11:36:42,466 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.88 vs. limit=6.0
2023-11-23 11:37:02,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2364466.6666666665, ans=0.1
2023-11-23 11:37:17,289 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6000, loss[loss=0.07913, simple_loss=0.1106, pruned_loss=0.0164, audio_tagging_loss=0.007414, over 15958.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09198, pruned_loss=0.01391, audio_tagging_loss=0.008931, over 3035145.12 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:37:17,292 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 11:37:57,936 INFO [train_asr.py:1253] (0/4) Epoch 30, validation: loss=0.05791, simple_loss=0.05108, pruned_loss=0.005053, audio_tagging_loss=0.02732, over 4681554.00 frames. 
2023-11-23 11:37:57,937 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 11:38:08,580 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354700
2023-11-23 11:38:22,332 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:38:29,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2364733.3333333335, ans=0.125
2023-11-23 11:38:35,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2364800.0, ans=0.125
2023-11-23 11:38:44,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2364800.0, ans=0.125
2023-11-23 11:38:45,077 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 11:38:52,919 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.693e+01 8.297e+01 9.002e+01 9.719e+01 1.265e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 11:38:54,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2364866.6666666665, ans=0.125
2023-11-23 11:39:02,747 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6050, loss[loss=0.06881, simple_loss=0.09456, pruned_loss=0.01445, audio_tagging_loss=0.00708, over 15348.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09208, pruned_loss=0.01387, audio_tagging_loss=0.00897, over 3037928.14 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:39:13,265 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354750
2023-11-23 11:39:18,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2365000.0, ans=0.125
2023-11-23 11:39:24,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2365000.0, ans=6.0
2023-11-23 11:39:43,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2365133.3333333335, ans=0.0
2023-11-23 11:40:07,157 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6100, loss[loss=0.07701, simple_loss=0.106, pruned_loss=0.01776, audio_tagging_loss=0.006255, over 15842.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09124, pruned_loss=0.01363, audio_tagging_loss=0.009073, over 3043460.34 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:40:17,113 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354800
2023-11-23 11:40:34,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2365400.0, ans=0.125
2023-11-23 11:40:35,756 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.18 vs. limit=12.0
2023-11-23 11:40:49,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2365466.6666666665, ans=0.125
2023-11-23 11:41:02,775 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.175e+01 8.198e+01 8.863e+01 9.563e+01 1.175e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 11:41:03,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.62 vs. limit=10.0
2023-11-23 11:41:11,408 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6150, loss[loss=0.06539, simple_loss=0.08388, pruned_loss=0.01033, audio_tagging_loss=0.01312, over 14763.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09127, pruned_loss=0.01369, audio_tagging_loss=0.009081, over 3039111.92 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:41:20,123 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.51 vs. limit=22.5
2023-11-23 11:41:21,773 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354850
2023-11-23 11:41:52,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2365800.0, ans=0.1
2023-11-23 11:42:09,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2365866.6666666665, ans=0.125
2023-11-23 11:42:12,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2365866.6666666665, ans=0.125
2023-11-23 11:42:16,183 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6200, loss[loss=0.06762, simple_loss=0.08556, pruned_loss=0.01592, audio_tagging_loss=0.00892, over 15210.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09148, pruned_loss=0.01379, audio_tagging_loss=0.009108, over 3034397.59 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:42:17,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2365933.3333333335, ans=0.125
2023-11-23 11:42:25,098 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 11:42:27,290 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354900
2023-11-23 11:42:32,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2366000.0, ans=0.125
2023-11-23 11:42:37,696 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.83 vs. limit=10.0
2023-11-23 11:43:04,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2366133.3333333335, ans=0.0
2023-11-23 11:43:07,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2366200.0, ans=0.07
2023-11-23 11:43:13,598 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.682e+01 8.235e+01 8.845e+01 9.606e+01 1.266e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 11:43:17,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2366200.0, ans=0.125
2023-11-23 11:43:21,064 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6250, loss[loss=0.05843, simple_loss=0.08313, pruned_loss=0.008188, audio_tagging_loss=0.008676, over 14778.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09164, pruned_loss=0.01392, audio_tagging_loss=0.009199, over 3034313.70 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 11:43:26,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2366266.6666666665, ans=0.125
2023-11-23 11:43:31,041 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 354950
2023-11-23 11:43:31,432 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.42 vs. limit=22.5
2023-11-23 11:43:36,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2366333.3333333335, ans=0.0
2023-11-23 11:44:02,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2366466.6666666665, ans=0.125
2023-11-23 11:44:24,654 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6300, loss[loss=0.09177, simple_loss=0.1264, pruned_loss=0.02047, audio_tagging_loss=0.008077, over 15031.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09201, pruned_loss=0.01385, audio_tagging_loss=0.009302, over 3036456.81 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 11:44:34,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355000
2023-11-23 11:45:01,149 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.96 vs. limit=6.0
2023-11-23 11:45:16,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2366866.6666666665, ans=0.1
2023-11-23 11:45:17,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2366866.6666666665, ans=0.0
2023-11-23 11:45:20,943 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.278e+01 8.805e+01 9.594e+01 1.253e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-23 11:45:28,970 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6350, loss[loss=0.07065, simple_loss=0.0865, pruned_loss=0.0165, audio_tagging_loss=0.01089, over 15192.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09098, pruned_loss=0.01371, audio_tagging_loss=0.009471, over 3037384.69 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 11:45:34,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2366933.3333333335, ans=0.0
2023-11-23 11:45:38,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2366933.3333333335, ans=0.2
2023-11-23 11:45:39,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355050
2023-11-23 11:45:52,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.56 vs. limit=15.0
2023-11-23 11:45:53,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2367000.0, ans=0.125
2023-11-23 11:45:53,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2367000.0, ans=0.0
2023-11-23 11:46:05,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2367066.6666666665, ans=0.0
2023-11-23 11:46:28,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2367200.0, ans=0.125
2023-11-23 11:46:34,447 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6400, loss[loss=0.08241, simple_loss=0.1081, pruned_loss=0.01913, audio_tagging_loss=0.009238, over 15162.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09055, pruned_loss=0.01357, audio_tagging_loss=0.009538, over 3036502.32 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:46:44,365 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355100
2023-11-23 11:46:46,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.51 vs. limit=12.0
2023-11-23 11:46:50,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2367333.3333333335, ans=0.125
2023-11-23 11:46:53,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2367333.3333333335, ans=0.0
2023-11-23 11:47:26,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.49 vs. limit=15.0
2023-11-23 11:47:27,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2367533.3333333335, ans=0.0
2023-11-23 11:47:27,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2367533.3333333335, ans=0.125
2023-11-23 11:47:30,741 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.626e+01 8.031e+01 8.724e+01 9.486e+01 1.143e+02, threshold=1.745e+02, percent-clipped=0.0
2023-11-23 11:47:32,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2367533.3333333335, ans=0.2
2023-11-23 11:47:38,274 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6450, loss[loss=0.07246, simple_loss=0.09189, pruned_loss=0.01786, audio_tagging_loss=0.008648, over 15245.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09059, pruned_loss=0.01355, audio_tagging_loss=0.009612, over 3035979.61 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:47:48,503 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355150
2023-11-23 11:48:00,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2367666.6666666665, ans=0.125
2023-11-23 11:48:07,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2367733.3333333335, ans=0.125
2023-11-23 11:48:23,400 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.75 vs. limit=22.5
2023-11-23 11:48:33,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2367866.6666666665, ans=0.125
2023-11-23 11:48:36,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2367866.6666666665, ans=0.0
2023-11-23 11:48:43,085 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6500, loss[loss=0.08756, simple_loss=0.1196, pruned_loss=0.02089, audio_tagging_loss=0.006854, over 16064.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09039, pruned_loss=0.0135, audio_tagging_loss=0.009568, over 3043349.29 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:48:53,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355200
2023-11-23 11:48:59,643 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.46 vs. limit=15.0
2023-11-23 11:49:02,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2368000.0, ans=0.1
2023-11-23 11:49:40,007 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.91 vs. limit=22.5
2023-11-23 11:49:40,453 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.870e+01 8.456e+01 9.159e+01 9.950e+01 1.283e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-23 11:49:44,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2368200.0, ans=0.125
2023-11-23 11:49:46,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2368200.0, ans=0.125
2023-11-23 11:49:48,697 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6550, loss[loss=0.06956, simple_loss=0.09584, pruned_loss=0.01712, audio_tagging_loss=0.004521, over 16321.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09078, pruned_loss=0.01351, audio_tagging_loss=0.009306, over 3044820.92 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:49:59,201 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355250
2023-11-23 11:50:29,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2368466.6666666665, ans=0.125
2023-11-23 11:50:43,139 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.22 vs. limit=10.0
2023-11-23 11:50:47,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.83 vs. limit=15.0
2023-11-23 11:50:53,282 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6600, loss[loss=0.05705, simple_loss=0.06395, pruned_loss=0.01107, audio_tagging_loss=0.01401, over 16008.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09162, pruned_loss=0.01361, audio_tagging_loss=0.009238, over 3043385.05 frames. ], batch size: 63, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:50:54,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2368600.0, ans=0.1
2023-11-23 11:51:03,317 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355300
2023-11-23 11:51:11,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2368666.6666666665, ans=0.2
2023-11-23 11:51:23,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2368733.3333333335, ans=0.125
2023-11-23 11:51:35,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2368800.0, ans=0.0
2023-11-23 11:51:50,575 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.423e+01 8.216e+01 8.892e+01 9.603e+01 1.376e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-23 11:51:54,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2368866.6666666665, ans=0.125
2023-11-23 11:51:58,420 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6650, loss[loss=0.06593, simple_loss=0.08661, pruned_loss=0.01191, audio_tagging_loss=0.01071, over 15734.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09047, pruned_loss=0.01335, audio_tagging_loss=0.009282, over 3047766.66 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:52:00,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2368933.3333333335, ans=0.05
2023-11-23 11:52:08,224 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.37 vs. limit=15.0
2023-11-23 11:52:08,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355350
2023-11-23 11:52:08,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2368933.3333333335, ans=0.125
2023-11-23 11:52:14,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2369000.0, ans=0.1
2023-11-23 11:52:24,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2369066.6666666665, ans=0.125
2023-11-23 11:52:28,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2369066.6666666665, ans=0.0
2023-11-23 11:52:48,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2369133.3333333335, ans=0.125
2023-11-23 11:53:03,820 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6700, loss[loss=0.07385, simple_loss=0.1049, pruned_loss=0.01553, audio_tagging_loss=0.005845, over 16393.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09075, pruned_loss=0.01358, audio_tagging_loss=0.009256, over 3050108.05 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:53:13,927 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355400
2023-11-23 11:53:24,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.75 vs. limit=15.0
2023-11-23 11:54:01,072 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.640e+01 8.203e+01 8.988e+01 9.539e+01 1.363e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 11:54:03,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2369533.3333333335, ans=0.125
2023-11-23 11:54:08,668 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6750, loss[loss=0.06683, simple_loss=0.09735, pruned_loss=0.009598, audio_tagging_loss=0.008558, over 14152.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09212, pruned_loss=0.01373, audio_tagging_loss=0.009145, over 3049733.21 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:54:13,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2369600.0, ans=0.125
2023-11-23 11:54:19,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355450
2023-11-23 11:54:23,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2369666.6666666665, ans=0.0
2023-11-23 11:54:30,342 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0
2023-11-23 11:54:36,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2369733.3333333335, ans=0.0
2023-11-23 11:54:38,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2369733.3333333335, ans=0.09899494936611666
2023-11-23 11:54:40,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2369733.3333333335, ans=0.0
2023-11-23 11:55:05,560 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.32 vs. limit=10.0
2023-11-23 11:55:11,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2369866.6666666665, ans=0.0
2023-11-23 11:55:13,505 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6800, loss[loss=0.06019, simple_loss=0.08396, pruned_loss=0.01123, audio_tagging_loss=0.006983, over 14332.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09186, pruned_loss=0.01362, audio_tagging_loss=0.009113, over 3045298.03 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:55:18,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2369933.3333333335, ans=0.125
2023-11-23 11:55:24,556 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355500
2023-11-23 11:55:38,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2370066.6666666665, ans=0.125
2023-11-23 11:55:40,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2370066.6666666665, ans=0.125
2023-11-23 11:55:41,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2370066.6666666665, ans=0.09899494936611666
2023-11-23 11:56:11,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2370200.0, ans=0.125
2023-11-23 11:56:12,194 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.201e+01 8.354e+01 8.978e+01 9.690e+01 1.235e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 11:56:19,034 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6850, loss[loss=0.05305, simple_loss=0.07441, pruned_loss=0.006175, audio_tagging_loss=0.009671, over 16720.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09253, pruned_loss=0.01375, audio_tagging_loss=0.00906, over 3048742.79 frames. ], batch size: 64, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:56:26,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2370266.6666666665, ans=0.125
2023-11-23 11:56:29,498 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355550
2023-11-23 11:56:35,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2370333.3333333335, ans=0.95
2023-11-23 11:56:50,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2370400.0, ans=0.025
2023-11-23 11:57:07,017 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.32 vs. limit=10.0
2023-11-23 11:57:24,310 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6900, loss[loss=0.05823, simple_loss=0.08121, pruned_loss=0.009357, audio_tagging_loss=0.008271, over 14703.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09276, pruned_loss=0.01379, audio_tagging_loss=0.00911, over 3042767.54 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:57:34,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355600
2023-11-23 11:58:11,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2370800.0, ans=0.0
2023-11-23 11:58:12,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.17 vs. limit=15.0
2023-11-23 11:58:15,743 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 11:58:23,527 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.385e+01 8.957e+01 9.598e+01 1.316e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 11:58:25,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2370866.6666666665, ans=0.2
2023-11-23 11:58:28,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2370933.3333333335, ans=0.125
2023-11-23 11:58:29,851 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 6950, loss[loss=0.07323, simple_loss=0.1023, pruned_loss=0.01361, audio_tagging_loss=0.008459, over 15247.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09277, pruned_loss=0.01395, audio_tagging_loss=0.009155, over 3043258.69 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:58:41,257 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355650
2023-11-23 11:58:48,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2371000.0, ans=0.1
2023-11-23 11:59:12,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2371133.3333333335, ans=0.1
2023-11-23 11:59:36,276 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7000, loss[loss=0.08752, simple_loss=0.1144, pruned_loss=0.02048, audio_tagging_loss=0.009861, over 15778.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09259, pruned_loss=0.01403, audio_tagging_loss=0.009184, over 3041877.59 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 11:59:46,813 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355700
2023-11-23 12:00:06,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2371400.0, ans=0.125
2023-11-23 12:00:16,210 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.79 vs. limit=15.0
2023-11-23 12:00:18,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2371466.6666666665, ans=0.1
2023-11-23 12:00:34,909 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.294e+01 8.338e+01 8.854e+01 9.577e+01 1.141e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 12:00:36,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2371533.3333333335, ans=0.1
2023-11-23 12:00:39,886 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7050, loss[loss=0.06126, simple_loss=0.07668, pruned_loss=0.01086, audio_tagging_loss=0.01207, over 15890.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09243, pruned_loss=0.01392, audio_tagging_loss=0.009161, over 3043190.53 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:00:49,727 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355750
2023-11-23 12:01:01,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.87 vs. limit=15.0
2023-11-23 12:01:08,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2371733.3333333335, ans=0.125
2023-11-23 12:01:12,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2371733.3333333335, ans=0.0
2023-11-23 12:01:12,983 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.29 vs. limit=15.0
2023-11-23 12:01:20,807 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.59 vs. limit=15.0
2023-11-23 12:01:43,833 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7100, loss[loss=0.06255, simple_loss=0.09255, pruned_loss=0.008961, audio_tagging_loss=0.007311, over 15665.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09226, pruned_loss=0.01403, audio_tagging_loss=0.00915, over 3043708.04 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:01:53,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2371933.3333333335, ans=0.0
2023-11-23 12:01:54,865 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355800
2023-11-23 12:02:01,458 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.67 vs. limit=22.5
2023-11-23 12:02:18,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2372066.6666666665, ans=0.125
2023-11-23 12:02:43,357 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.907e+01 8.520e+01 9.184e+01 9.861e+01 1.233e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 12:02:49,737 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7150, loss[loss=0.06011, simple_loss=0.08161, pruned_loss=0.008685, audio_tagging_loss=0.01062, over 14925.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09185, pruned_loss=0.01397, audio_tagging_loss=0.009294, over 3040997.49 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:02:52,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2372266.6666666665, ans=0.2
2023-11-23 12:03:00,218 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355850
2023-11-23 12:03:04,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2372333.3333333335, ans=0.125
2023-11-23 12:03:08,026 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.37 vs. limit=15.0
2023-11-23 12:03:11,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2372333.3333333335, ans=0.2
2023-11-23 12:03:19,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2372400.0, ans=0.0
2023-11-23 12:03:22,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2372400.0, ans=0.125
2023-11-23 12:03:23,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2372400.0, ans=0.125
2023-11-23 12:03:24,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2372400.0, ans=0.125
2023-11-23 12:03:42,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2372533.3333333335, ans=0.035
2023-11-23 12:03:53,736 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7200, loss[loss=0.07151, simple_loss=0.09365, pruned_loss=0.01581, audio_tagging_loss=0.008873, over 15831.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09176, pruned_loss=0.01394, audio_tagging_loss=0.009348, over 3038076.79 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:04:01,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2372600.0, ans=0.0
2023-11-23 12:04:03,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355900
2023-11-23 12:04:15,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2372666.6666666665, ans=0.025
2023-11-23 12:04:27,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2372733.3333333335, ans=0.1
2023-11-23 12:04:28,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2372733.3333333335, ans=0.125
2023-11-23 12:04:29,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2372733.3333333335, ans=0.125
2023-11-23 12:04:29,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2372733.3333333335, ans=0.125
2023-11-23 12:04:29,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2372733.3333333335, ans=0.125
2023-11-23 12:04:29,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2372733.3333333335, ans=0.5
2023-11-23 12:04:52,180 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.220e+01 8.499e+01 9.077e+01 9.977e+01 1.549e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 12:04:57,126 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7250, loss[loss=0.0538, simple_loss=0.06623, pruned_loss=0.009255, audio_tagging_loss=0.01143, over 15178.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.0913, pruned_loss=0.01392, audio_tagging_loss=0.00938, over 3034677.84 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:05:07,505 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 355950
2023-11-23 12:05:31,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2373066.6666666665, ans=0.05
2023-11-23 12:05:37,015 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.21 vs. limit=15.0
2023-11-23 12:05:39,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2373133.3333333335, ans=0.125
2023-11-23 12:05:41,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2373133.3333333335, ans=0.015
2023-11-23 12:05:46,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2373133.3333333335, ans=0.125
2023-11-23 12:05:51,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2373200.0, ans=0.125
2023-11-23 12:05:59,943 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.55 vs. limit=22.5
2023-11-23 12:06:01,449 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7300, loss[loss=0.05436, simple_loss=0.0754, pruned_loss=0.007672, audio_tagging_loss=0.008988, over 15462.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09127, pruned_loss=0.01388, audio_tagging_loss=0.009262, over 3034130.19 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:06:12,745 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356000
2023-11-23 12:06:14,295 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-356000.pt
2023-11-23 12:06:43,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2373466.6666666665, ans=0.1
2023-11-23 12:07:02,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2373533.3333333335, ans=0.0
2023-11-23 12:07:05,006 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.072e+01 8.323e+01 8.907e+01 9.427e+01 1.100e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 12:07:05,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2373533.3333333335, ans=0.125
2023-11-23 12:07:06,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2373533.3333333335, ans=0.125
2023-11-23 12:07:10,565 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7350, loss[loss=0.09559, simple_loss=0.1375, pruned_loss=0.0218, audio_tagging_loss=0.005034, over 15395.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09136, pruned_loss=0.01397, audio_tagging_loss=0.009082, over 3032175.66 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:07:14,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2373600.0, ans=0.1
2023-11-23 12:07:20,534 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356050
2023-11-23 12:07:45,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2373733.3333333335, ans=0.2
2023-11-23 12:07:45,820 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.51 vs. limit=15.0
2023-11-23 12:07:49,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2373800.0, ans=0.125
2023-11-23 12:07:54,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2373800.0, ans=0.2
2023-11-23 12:08:14,465 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7400, loss[loss=0.06665, simple_loss=0.08887, pruned_loss=0.01599, audio_tagging_loss=0.006226, over 15048.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09108, pruned_loss=0.01397, audio_tagging_loss=0.009039, over 3032981.81 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:08:24,221 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356100
2023-11-23 12:08:25,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2374000.0, ans=0.2
2023-11-23 12:09:04,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2374200.0, ans=0.015
2023-11-23 12:09:13,006 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.433e+01 8.940e+01 9.689e+01 1.513e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 12:09:18,589 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7450, loss[loss=0.07988, simple_loss=0.1039, pruned_loss=0.01994, audio_tagging_loss=0.007999, over 15733.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09198, pruned_loss=0.01407, audio_tagging_loss=0.009035, over 3030854.06 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:09:28,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2374266.6666666665, ans=0.0
2023-11-23 12:09:29,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356150
2023-11-23 12:09:39,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2374333.3333333335, ans=0.035
2023-11-23 12:09:47,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2374400.0, ans=0.0
2023-11-23 12:09:50,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2374400.0, ans=0.2
2023-11-23 12:09:50,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2374400.0, ans=0.125
2023-11-23 12:10:01,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2374466.6666666665, ans=0.05
2023-11-23 12:10:06,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2374466.6666666665, ans=0.125
2023-11-23 12:10:10,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2374533.3333333335, ans=0.125
2023-11-23 12:10:14,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.55 vs. limit=15.0
2023-11-23 12:10:18,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2374533.3333333335, ans=0.2
2023-11-23 12:10:21,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2374533.3333333335, ans=0.2
2023-11-23 12:10:23,198 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.68 vs. limit=15.0
2023-11-23 12:10:24,409 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7500, loss[loss=0.0561, simple_loss=0.06432, pruned_loss=0.01333, audio_tagging_loss=0.0106, over 14941.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09096, pruned_loss=0.01398, audio_tagging_loss=0.009055, over 3030564.52 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:10:24,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2374600.0, ans=0.125
2023-11-23 12:10:26,640 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.78 vs. limit=10.0
2023-11-23 12:10:29,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2374600.0, ans=10.0
2023-11-23 12:10:32,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2374600.0, ans=0.0
2023-11-23 12:10:34,378 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356200
2023-11-23 12:10:56,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2374733.3333333335, ans=0.0
2023-11-23 12:11:14,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2374800.0, ans=0.2
2023-11-23 12:11:23,841 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.927e+01 8.224e+01 8.821e+01 9.416e+01 1.230e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 12:11:28,803 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7550, loss[loss=0.05197, simple_loss=0.07162, pruned_loss=0.007143, audio_tagging_loss=0.009013, over 15522.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09194, pruned_loss=0.01416, audio_tagging_loss=0.009044, over 3036155.97 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:11:30,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2374933.3333333335, ans=0.1
2023-11-23 12:11:31,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2374933.3333333335, ans=0.0
2023-11-23 12:11:38,790 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356250
2023-11-23 12:11:38,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2374933.3333333335, ans=0.1
2023-11-23 12:11:42,080 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.69 vs. limit=15.0
2023-11-23 12:11:57,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2375066.6666666665, ans=0.125
2023-11-23 12:12:05,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2375066.6666666665, ans=0.025
2023-11-23 12:12:17,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.72 vs. limit=10.0
2023-11-23 12:12:24,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2375200.0, ans=0.2
2023-11-23 12:12:24,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2375200.0, ans=0.125
2023-11-23 12:12:28,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2375200.0, ans=0.1
2023-11-23 12:12:30,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2375200.0, ans=0.1
2023-11-23 12:12:34,271 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7600, loss[loss=0.08044, simple_loss=0.1107, pruned_loss=0.0174, audio_tagging_loss=0.007669, over 16166.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.0916, pruned_loss=0.01397, audio_tagging_loss=0.009029, over 3041485.12 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:12:44,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.11 vs. limit=15.0
2023-11-23 12:12:44,687 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356300
2023-11-23 12:13:25,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2375533.3333333335, ans=0.05
2023-11-23 12:13:25,474 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.00 vs. limit=15.0
2023-11-23 12:13:35,218 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.003e+01 8.205e+01 8.929e+01 9.831e+01 1.216e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 12:13:39,067 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7650, loss[loss=0.06633, simple_loss=0.09616, pruned_loss=0.01168, audio_tagging_loss=0.006573, over 15696.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.0925, pruned_loss=0.01419, audio_tagging_loss=0.008958, over 3044330.73 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:13:39,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2375600.0, ans=0.1
2023-11-23 12:13:50,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356350
2023-11-23 12:14:09,068 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:14:11,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2375733.3333333335, ans=0.1
2023-11-23 12:14:16,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2375800.0, ans=0.035
2023-11-23 12:14:18,756 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.74 vs. limit=15.0
2023-11-23 12:14:28,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2375800.0, ans=0.125
2023-11-23 12:14:29,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.36 vs. limit=15.0
2023-11-23 12:14:38,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2375866.6666666665, ans=0.125
2023-11-23 12:14:44,746 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7700, loss[loss=0.06441, simple_loss=0.07744, pruned_loss=0.01394, audio_tagging_loss=0.01175, over 16341.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09233, pruned_loss=0.01411, audio_tagging_loss=0.009005, over 3047073.50 frames. ], batch size: 62, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:14:54,598 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356400
2023-11-23 12:14:54,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2375933.3333333335, ans=0.1
2023-11-23 12:14:57,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2376000.0, ans=0.0
2023-11-23 12:15:20,073 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.06 vs. limit=15.0
2023-11-23 12:15:33,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2376133.3333333335, ans=0.125
2023-11-23 12:15:37,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2376200.0, ans=0.125
2023-11-23 12:15:38,595 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.76 vs. limit=15.0
2023-11-23 12:15:44,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2376200.0, ans=0.1
2023-11-23 12:15:45,413 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.176e+01 8.917e+01 9.503e+01 1.166e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 12:15:49,148 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7750, loss[loss=0.08568, simple_loss=0.1098, pruned_loss=0.02172, audio_tagging_loss=0.009048, over 14920.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09215, pruned_loss=0.01422, audio_tagging_loss=0.009024, over 3043373.41 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:16:00,193 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356450
2023-11-23 12:16:13,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2376333.3333333335, ans=0.125
2023-11-23 12:16:14,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2376400.0, ans=0.0
2023-11-23 12:16:22,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2376400.0, ans=0.125
2023-11-23 12:16:53,612 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.47 vs. limit=15.0
2023-11-23 12:16:54,282 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7800, loss[loss=0.09678, simple_loss=0.1252, pruned_loss=0.02245, audio_tagging_loss=0.01172, over 14043.00 frames. ], tot_loss[loss=0.0702, simple_loss=0.09362, pruned_loss=0.01437, audio_tagging_loss=0.009016, over 3043177.41 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:17:04,740 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356500
2023-11-23 12:17:38,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2376800.0, ans=0.05
2023-11-23 12:17:47,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2376866.6666666665, ans=0.09899494936611666
2023-11-23 12:17:53,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2376866.6666666665, ans=0.1
2023-11-23 12:17:54,733 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.181e+01 8.920e+01 9.526e+01 1.528e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 12:17:58,458 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7850, loss[loss=0.08243, simple_loss=0.1058, pruned_loss=0.02059, audio_tagging_loss=0.008924, over 15554.00 frames. ], tot_loss[loss=0.0708, simple_loss=0.0944, pruned_loss=0.01458, audio_tagging_loss=0.009021, over 3047291.73 frames. ], batch size: 58, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:18:01,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.38 vs. limit=22.5
2023-11-23 12:18:09,053 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356550
2023-11-23 12:18:26,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2377066.6666666665, ans=0.0
2023-11-23 12:18:53,163 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:19:02,651 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7900, loss[loss=0.0633, simple_loss=0.08182, pruned_loss=0.01246, audio_tagging_loss=0.009938, over 13862.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09266, pruned_loss=0.01425, audio_tagging_loss=0.009188, over 3049794.29 frames. ], batch size: 53, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:19:05,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2377266.6666666665, ans=0.2
2023-11-23 12:19:08,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2377266.6666666665, ans=10.0
2023-11-23 12:19:10,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2377266.6666666665, ans=0.0
2023-11-23 12:19:13,392 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356600
2023-11-23 12:19:15,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2377333.3333333335, ans=0.5
2023-11-23 12:19:30,372 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.78 vs. limit=10.0
2023-11-23 12:19:59,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2377533.3333333335, ans=0.0
2023-11-23 12:20:04,577 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.810e+01 8.633e+01 9.246e+01 1.018e+02 1.529e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-23 12:20:08,299 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 7950, loss[loss=0.05727, simple_loss=0.07972, pruned_loss=0.008026, audio_tagging_loss=0.009381, over 15776.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09191, pruned_loss=0.01409, audio_tagging_loss=0.009376, over 3050993.39 frames. ], batch size: 59, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:20:18,771 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356650
2023-11-23 12:20:26,027 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 12:20:32,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=2377666.6666666665, ans=15.0
2023-11-23 12:21:07,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2377866.6666666665, ans=0.125
2023-11-23 12:21:12,869 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8000, loss[loss=0.08293, simple_loss=0.1171, pruned_loss=0.01641, audio_tagging_loss=0.007944, over 15185.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09107, pruned_loss=0.01397, audio_tagging_loss=0.009507, over 3046004.37 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:21:23,804 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356700
2023-11-23 12:21:35,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2378000.0, ans=0.125
2023-11-23 12:21:44,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2378066.6666666665, ans=0.1
2023-11-23 12:21:55,923 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.57 vs. limit=6.0
2023-11-23 12:22:05,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2378200.0, ans=0.1
2023-11-23 12:22:15,959 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.021e+01 8.250e+01 8.793e+01 9.379e+01 1.192e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 12:22:18,481 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8050, loss[loss=0.05215, simple_loss=0.07012, pruned_loss=0.007918, audio_tagging_loss=0.009169, over 14439.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09167, pruned_loss=0.01413, audio_tagging_loss=0.009514, over 3046342.80 frames. ], batch size: 53, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:22:28,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356750
2023-11-23 12:22:30,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2378333.3333333335, ans=0.1
2023-11-23 12:22:35,212 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.12 vs. limit=15.0
2023-11-23 12:22:46,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2378400.0, ans=0.125
2023-11-23 12:22:48,144 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.22 vs. limit=22.5
2023-11-23 12:22:55,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2378400.0, ans=0.125
2023-11-23 12:23:05,412 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=12.0
2023-11-23 12:23:07,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2378466.6666666665, ans=0.125
2023-11-23 12:23:08,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2378466.6666666665, ans=10.0
2023-11-23 12:23:19,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2378533.3333333335, ans=0.125
2023-11-23 12:23:24,492 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8100, loss[loss=0.05546, simple_loss=0.07258, pruned_loss=0.01194, audio_tagging_loss=0.007234, over 16243.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09215, pruned_loss=0.01433, audio_tagging_loss=0.009438, over 3037278.56 frames. ], batch size: 62, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:23:34,694 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356800
2023-11-23 12:24:00,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2378733.3333333335, ans=0.1
2023-11-23 12:24:04,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2378800.0, ans=0.1
2023-11-23 12:24:07,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.55 vs. limit=10.0
2023-11-23 12:24:21,548 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:24:27,905 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.660e+01 9.350e+01 9.941e+01 1.281e+02, threshold=1.870e+02, percent-clipped=0.0
2023-11-23 12:24:30,495 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8150, loss[loss=0.08098, simple_loss=0.1185, pruned_loss=0.01624, audio_tagging_loss=0.005486, over 14698.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09251, pruned_loss=0.01438, audio_tagging_loss=0.009231, over 3038105.95 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:24:40,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356850
2023-11-23 12:24:50,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2379000.0, ans=0.125
2023-11-23 12:25:08,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2379133.3333333335, ans=0.125
2023-11-23 12:25:16,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2379133.3333333335, ans=0.0
2023-11-23 12:25:34,970 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8200, loss[loss=0.0702, simple_loss=0.09945, pruned_loss=0.01339, audio_tagging_loss=0.00709, over 16593.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09293, pruned_loss=0.01425, audio_tagging_loss=0.00913, over 3040189.45 frames. ], batch size: 62, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:25:37,472 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 12:25:37,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2379266.6666666665, ans=0.0
2023-11-23 12:25:46,483 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356900
2023-11-23 12:26:38,260 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.899e+01 8.430e+01 9.137e+01 9.990e+01 1.706e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-23 12:26:40,878 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8250, loss[loss=0.05446, simple_loss=0.07032, pruned_loss=0.008771, audio_tagging_loss=0.01053, over 15542.00 frames. ], tot_loss[loss=0.06981, simple_loss=0.09313, pruned_loss=0.01428, audio_tagging_loss=0.008964, over 3046338.56 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:26:47,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2379600.0, ans=0.1
2023-11-23 12:26:51,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 356950
2023-11-23 12:27:03,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2379666.6666666665, ans=0.05
2023-11-23 12:27:19,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.57 vs. limit=22.5
2023-11-23 12:27:44,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2379933.3333333335, ans=0.0
2023-11-23 12:27:45,815 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8300, loss[loss=0.06006, simple_loss=0.08813, pruned_loss=0.01017, audio_tagging_loss=0.005831, over 15482.00 frames. ], tot_loss[loss=0.06961, simple_loss=0.0929, pruned_loss=0.01421, audio_tagging_loss=0.008951, over 3051556.74 frames. ], batch size: 60, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:27:46,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2379933.3333333335, ans=0.2
2023-11-23 12:27:49,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2379933.3333333335, ans=0.125
2023-11-23 12:27:51,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2379933.3333333335, ans=0.125
2023-11-23 12:27:55,865 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357000
2023-11-23 12:28:03,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2380000.0, ans=0.2
2023-11-23 12:28:04,157 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.34 vs. limit=15.0
2023-11-23 12:28:41,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2380200.0, ans=0.125
2023-11-23 12:28:44,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2380200.0, ans=0.0
2023-11-23 12:28:47,625 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.282e+01 8.543e+01 9.003e+01 9.782e+01 1.205e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 12:28:50,049 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8350, loss[loss=0.05948, simple_loss=0.07968, pruned_loss=0.01155, audio_tagging_loss=0.00809, over 15109.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09244, pruned_loss=0.01405, audio_tagging_loss=0.008909, over 3052242.24 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:29:00,486 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357050
2023-11-23 12:29:28,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2380466.6666666665, ans=0.125
2023-11-23 12:29:37,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2380466.6666666665, ans=0.2
2023-11-23 12:29:54,905 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8400, loss[loss=0.09564, simple_loss=0.1375, pruned_loss=0.02075, audio_tagging_loss=0.006123, over 16846.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09205, pruned_loss=0.01393, audio_tagging_loss=0.00898, over 3051199.21 frames. ], batch size: 61, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:30:05,198 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357100
2023-11-23 12:30:17,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2380666.6666666665, ans=0.0
2023-11-23 12:30:27,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2380733.3333333335, ans=0.125
2023-11-23 12:30:35,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2380800.0, ans=0.125
2023-11-23 12:30:48,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2380866.6666666665, ans=0.0
2023-11-23 12:30:57,938 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.912e+01 8.179e+01 8.935e+01 9.716e+01 1.287e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 12:30:59,217 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8450, loss[loss=0.07042, simple_loss=0.09506, pruned_loss=0.01561, audio_tagging_loss=0.00728, over 15184.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09187, pruned_loss=0.01386, audio_tagging_loss=0.00899, over 3046206.47 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:31:08,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.33 vs. limit=12.0
2023-11-23 12:31:09,769 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357150
2023-11-23 12:31:09,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2380933.3333333335, ans=0.07
2023-11-23 12:31:14,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2381000.0, ans=0.0
2023-11-23 12:31:31,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2381066.6666666665, ans=0.0
2023-11-23 12:32:03,579 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8500, loss[loss=0.0447, simple_loss=0.05392, pruned_loss=0.008627, audio_tagging_loss=0.009111, over 14330.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09278, pruned_loss=0.01394, audio_tagging_loss=0.008992, over 3047214.48 frames. ], batch size: 55, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:32:08,925 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:32:13,532 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357200
2023-11-23 12:32:13,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2381266.6666666665, ans=0.0
2023-11-23 12:32:36,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2381400.0, ans=0.035
2023-11-23 12:32:41,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2381466.6666666665, ans=0.125
2023-11-23 12:32:52,011 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.76 vs. limit=15.0
2023-11-23 12:33:01,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2381533.3333333335, ans=0.125
2023-11-23 12:33:06,327 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.727e+01 8.293e+01 8.960e+01 9.755e+01 1.208e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 12:33:06,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2381600.0, ans=0.2
2023-11-23 12:33:08,146 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8550, loss[loss=0.04814, simple_loss=0.06385, pruned_loss=0.009001, audio_tagging_loss=0.00721, over 15194.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09351, pruned_loss=0.01402, audio_tagging_loss=0.008995, over 3051835.82 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:33:19,383 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357250
2023-11-23 12:33:32,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2381666.6666666665, ans=0.0
2023-11-23 12:33:33,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2381733.3333333335, ans=0.0
2023-11-23 12:33:33,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2381733.3333333335, ans=0.125
2023-11-23 12:33:54,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2381800.0, ans=0.125
2023-11-23 12:34:13,825 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8600, loss[loss=0.06854, simple_loss=0.09065, pruned_loss=0.01121, audio_tagging_loss=0.01201, over 14682.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09246, pruned_loss=0.01388, audio_tagging_loss=0.009179, over 3045479.47 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:34:20,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2381933.3333333335, ans=0.05
2023-11-23 12:34:24,366 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357300
2023-11-23 12:34:30,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2382000.0, ans=0.125
2023-11-23 12:34:38,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.30 vs. limit=15.0
2023-11-23 12:35:16,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2382200.0, ans=0.0
2023-11-23 12:35:17,576 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.105e+01 8.339e+01 9.203e+01 9.998e+01 1.297e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-23 12:35:18,846 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8650, loss[loss=0.08216, simple_loss=0.1047, pruned_loss=0.02013, audio_tagging_loss=0.009677, over 15045.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09293, pruned_loss=0.01397, audio_tagging_loss=0.009122, over 3050056.83 frames. ], batch size: 53, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:35:22,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2382266.6666666665, ans=0.125
2023-11-23 12:35:28,756 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357350
2023-11-23 12:35:35,712 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0
2023-11-23 12:36:12,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2382533.3333333335, ans=0.125
2023-11-23 12:36:17,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2382533.3333333335, ans=0.125
2023-11-23 12:36:18,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2382533.3333333335, ans=0.125
2023-11-23 12:36:18,567 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.69 vs. limit=22.5
2023-11-23 12:36:20,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2382533.3333333335, ans=0.0
2023-11-23 12:36:22,800 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8700, loss[loss=0.09643, simple_loss=0.1359, pruned_loss=0.02137, audio_tagging_loss=0.007117, over 15033.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09278, pruned_loss=0.01384, audio_tagging_loss=0.009157, over 3045146.18 frames. ], batch size: 52, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:36:31,341 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.90 vs. limit=15.0
2023-11-23 12:36:33,971 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357400
2023-11-23 12:36:34,594 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.57 vs. limit=15.0
2023-11-23 12:36:41,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2382666.6666666665, ans=0.0
2023-11-23 12:37:11,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.26 vs. limit=10.0
2023-11-23 12:37:14,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2382866.6666666665, ans=0.125
2023-11-23 12:37:27,564 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 8.508e+01 9.323e+01 9.917e+01 1.440e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-23 12:37:28,911 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8750, loss[loss=0.07529, simple_loss=0.09973, pruned_loss=0.01709, audio_tagging_loss=0.00834, over 14879.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09286, pruned_loss=0.01395, audio_tagging_loss=0.009159, over 3050846.80 frames. ], batch size: 54, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:37:40,213 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357450
2023-11-23 12:37:52,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2383000.0, ans=0.125
2023-11-23 12:37:58,549 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.57 vs. limit=15.0
2023-11-23 12:38:14,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2383133.3333333335, ans=0.125
2023-11-23 12:38:15,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2383133.3333333335, ans=0.125
2023-11-23 12:38:25,392 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.17 vs. limit=12.0
2023-11-23 12:38:35,196 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8800, loss[loss=0.05678, simple_loss=0.082, pruned_loss=0.009992, audio_tagging_loss=0.005785, over 15128.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09352, pruned_loss=0.01399, audio_tagging_loss=0.009196, over 3055114.57 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 32.0
2023-11-23 12:38:39,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2383266.6666666665, ans=0.125
2023-11-23 12:38:44,982 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357500
2023-11-23 12:39:06,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2383400.0, ans=0.0
2023-11-23 12:39:07,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2383400.0, ans=0.125
2023-11-23 12:39:26,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2383533.3333333335, ans=0.1
2023-11-23 12:39:29,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2383533.3333333335, ans=0.125
2023-11-23 12:39:39,751 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.896e+01 8.582e+01 9.222e+01 9.925e+01 1.869e+02, threshold=1.844e+02, percent-clipped=1.0
2023-11-23 12:39:39,802 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8850, loss[loss=0.05863, simple_loss=0.07828, pruned_loss=0.01193, audio_tagging_loss=0.007564, over 14926.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09311, pruned_loss=0.01396, audio_tagging_loss=0.009241, over 3049152.85 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:39:50,521 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357550
2023-11-23 12:39:54,631 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 12:40:05,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2383733.3333333335, ans=0.125
2023-11-23 12:40:28,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2383800.0, ans=0.125
2023-11-23 12:40:45,545 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8900, loss[loss=0.08487, simple_loss=0.1147, pruned_loss=0.02155, audio_tagging_loss=0.005983, over 15727.00 frames. ], tot_loss[loss=0.07001, simple_loss=0.09324, pruned_loss=0.01416, audio_tagging_loss=0.009223, over 3051687.90 frames. ], batch size: 57, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:40:52,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.52 vs. limit=15.0
2023-11-23 12:40:56,667 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357600
2023-11-23 12:40:58,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2384000.0, ans=0.0
2023-11-23 12:41:46,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2384200.0, ans=0.1
2023-11-23 12:41:51,033 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.775e+01 8.078e+01 8.629e+01 9.554e+01 1.140e+02, threshold=1.726e+02, percent-clipped=0.0
2023-11-23 12:41:51,107 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 8950, loss[loss=0.07172, simple_loss=0.09793, pruned_loss=0.01384, audio_tagging_loss=0.008918, over 14978.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09335, pruned_loss=0.01413, audio_tagging_loss=0.009145, over 3058850.82 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:42:00,997 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357650
2023-11-23 12:42:07,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2384333.3333333335, ans=0.04949747468305833
2023-11-23 12:42:23,910 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.41 vs. limit=22.5
2023-11-23 12:42:28,039 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:42:28,228 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:42:47,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2384533.3333333335, ans=0.125
2023-11-23 12:42:54,540 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9000, loss[loss=0.05416, simple_loss=0.07171, pruned_loss=0.009112, audio_tagging_loss=0.009199, over 15963.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09391, pruned_loss=0.01443, audio_tagging_loss=0.009052, over 3064022.78 frames. ], batch size: 62, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:42:54,544 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 12:43:29,236 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4872, 3.2583, 3.7349, 3.4550], device='cuda:0')
2023-11-23 12:43:36,350 INFO [train_asr.py:1253] (0/4) Epoch 30, validation: loss=0.05877, simple_loss=0.051, pruned_loss=0.005026, audio_tagging_loss=0.02824, over 4681554.00 frames. 
2023-11-23 12:43:36,350 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 12:43:40,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2384600.0, ans=0.125
2023-11-23 12:43:47,540 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357700
2023-11-23 12:43:51,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.whiten.whitening_limit, batch_count=2384666.6666666665, ans=12.0
2023-11-23 12:43:58,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2384666.6666666665, ans=0.0
2023-11-23 12:44:01,387 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.24 vs. limit=12.0
2023-11-23 12:44:04,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2384733.3333333335, ans=0.0
2023-11-23 12:44:06,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2384733.3333333335, ans=0.125
2023-11-23 12:44:10,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2384733.3333333335, ans=0.1
2023-11-23 12:44:16,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2384800.0, ans=0.125
2023-11-23 12:44:17,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2384800.0, ans=0.2
2023-11-23 12:44:27,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2384866.6666666665, ans=0.0
2023-11-23 12:44:41,113 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.551e+01 8.530e+01 8.986e+01 9.928e+01 1.282e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 12:44:41,180 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9050, loss[loss=0.05906, simple_loss=0.07403, pruned_loss=0.01122, audio_tagging_loss=0.01083, over 14804.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09284, pruned_loss=0.01418, audio_tagging_loss=0.009053, over 3059919.58 frames. ], batch size: 56, lr: 2.26e-03, grad_scale: 16.0
2023-11-23 12:44:41,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2384933.3333333335, ans=0.0
2023-11-23 12:44:43,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2384933.3333333335, ans=0.0
2023-11-23 12:44:45,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2384933.3333333335, ans=0.125
2023-11-23 12:44:50,320 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.74 vs. limit=22.5
2023-11-23 12:44:51,064 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357750
2023-11-23 12:45:44,794 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9100, loss[loss=0.08389, simple_loss=0.1112, pruned_loss=0.02075, audio_tagging_loss=0.007525, over 17178.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09298, pruned_loss=0.01416, audio_tagging_loss=0.00892, over 3058677.94 frames. ], batch size: 63, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:45:55,229 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357800
2023-11-23 12:46:01,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2385333.3333333335, ans=0.1
2023-11-23 12:46:28,292 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:46:43,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2385533.3333333335, ans=0.125
2023-11-23 12:46:47,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2385533.3333333335, ans=0.07
2023-11-23 12:46:49,813 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.972e+01 8.272e+01 8.936e+01 9.614e+01 1.250e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 12:46:49,859 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9150, loss[loss=0.07614, simple_loss=0.09943, pruned_loss=0.01673, audio_tagging_loss=0.009703, over 15729.00 frames. ], tot_loss[loss=0.07054, simple_loss=0.09436, pruned_loss=0.01446, audio_tagging_loss=0.008899, over 3060207.03 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:46:56,661 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.64 vs. limit=22.5
2023-11-23 12:46:58,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2385600.0, ans=0.1
2023-11-23 12:47:00,196 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357850
2023-11-23 12:47:10,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2385666.6666666665, ans=0.1
2023-11-23 12:47:21,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2385733.3333333335, ans=0.0
2023-11-23 12:47:47,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2385866.6666666665, ans=0.125
2023-11-23 12:47:47,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2385866.6666666665, ans=0.125
2023-11-23 12:47:53,032 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9200, loss[loss=0.06902, simple_loss=0.09601, pruned_loss=0.01098, audio_tagging_loss=0.01003, over 15756.00 frames. ], tot_loss[loss=0.07005, simple_loss=0.09367, pruned_loss=0.01437, audio_tagging_loss=0.008849, over 3062082.52 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 12:47:57,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2385933.3333333335, ans=0.1
2023-11-23 12:48:02,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2385933.3333333335, ans=0.1
2023-11-23 12:48:03,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357900
2023-11-23 12:48:25,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2386066.6666666665, ans=0.125
2023-11-23 12:48:44,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_na.min_abs, batch_count=2386200.0, ans=0.02
2023-11-23 12:48:56,516 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.572e+01 8.180e+01 8.978e+01 9.494e+01 1.363e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 12:48:56,564 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9250, loss[loss=0.05642, simple_loss=0.07349, pruned_loss=0.009216, audio_tagging_loss=0.01046, over 16428.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09269, pruned_loss=0.01425, audio_tagging_loss=0.008928, over 3064243.26 frames. ], batch size: 61, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 12:49:05,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2386266.6666666665, ans=0.125
2023-11-23 12:49:06,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 357950
2023-11-23 12:49:18,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2386333.3333333335, ans=0.0
2023-11-23 12:49:36,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.45 vs. limit=22.5
2023-11-23 12:49:37,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2386466.6666666665, ans=0.05
2023-11-23 12:49:41,640 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:50:00,439 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9300, loss[loss=0.07626, simple_loss=0.09789, pruned_loss=0.01659, audio_tagging_loss=0.01072, over 17256.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09169, pruned_loss=0.01408, audio_tagging_loss=0.009021, over 3062789.38 frames. ], batch size: 65, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 12:50:02,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2386600.0, ans=0.125
2023-11-23 12:50:04,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2386600.0, ans=0.2
2023-11-23 12:50:10,753 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358000
2023-11-23 12:50:19,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2386666.6666666665, ans=0.125
2023-11-23 12:50:23,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2386666.6666666665, ans=0.125
2023-11-23 12:50:25,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2386733.3333333335, ans=0.2
2023-11-23 12:50:25,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2386733.3333333335, ans=0.0
2023-11-23 12:50:30,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2386733.3333333335, ans=0.0
2023-11-23 12:50:38,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2386800.0, ans=0.0
2023-11-23 12:50:42,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2386800.0, ans=0.0
2023-11-23 12:50:52,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2386866.6666666665, ans=0.0
2023-11-23 12:50:53,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2386866.6666666665, ans=0.125
2023-11-23 12:50:53,368 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.77 vs. limit=15.0
2023-11-23 12:50:54,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2386866.6666666665, ans=0.1
2023-11-23 12:51:04,206 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.960e+01 8.283e+01 8.829e+01 9.752e+01 1.235e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 12:51:04,249 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9350, loss[loss=0.07851, simple_loss=0.1113, pruned_loss=0.0158, audio_tagging_loss=0.00709, over 15905.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09215, pruned_loss=0.01416, audio_tagging_loss=0.00902, over 3065861.60 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 12:51:08,779 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=11.04 vs. limit=15.0
2023-11-23 12:51:13,932 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358050
2023-11-23 12:51:49,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2387133.3333333335, ans=0.1
2023-11-23 12:52:00,409 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:52:02,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2387200.0, ans=0.0
2023-11-23 12:52:07,411 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9400, loss[loss=0.07881, simple_loss=0.1088, pruned_loss=0.01694, audio_tagging_loss=0.007491, over 15103.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09245, pruned_loss=0.01407, audio_tagging_loss=0.009027, over 3061265.01 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 8.0
2023-11-23 12:52:17,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358100
2023-11-23 12:52:22,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2387333.3333333335, ans=0.125
2023-11-23 12:52:58,198 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.26 vs. limit=15.0
2023-11-23 12:53:04,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2387533.3333333335, ans=0.025
2023-11-23 12:53:10,721 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 12:53:11,877 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9450, loss[loss=0.08643, simple_loss=0.1281, pruned_loss=0.01669, audio_tagging_loss=0.005703, over 15395.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09278, pruned_loss=0.01404, audio_tagging_loss=0.009155, over 3058140.37 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 8.0
2023-11-23 12:53:14,839 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.523e+01 9.315e+01 1.049e+02 1.250e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-23 12:53:22,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358150
2023-11-23 12:53:22,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2387600.0, ans=0.125
2023-11-23 12:53:40,359 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.00 vs. limit=12.0
2023-11-23 12:53:47,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2387733.3333333335, ans=0.2
2023-11-23 12:54:00,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2387800.0, ans=0.125
2023-11-23 12:54:16,493 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9500, loss[loss=0.07158, simple_loss=0.1013, pruned_loss=0.0125, audio_tagging_loss=0.008448, over 15242.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09333, pruned_loss=0.01409, audio_tagging_loss=0.009202, over 3063676.81 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 8.0
2023-11-23 12:54:16,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2387933.3333333335, ans=0.025
2023-11-23 12:54:18,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2387933.3333333335, ans=0.125
2023-11-23 12:54:26,354 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358200
2023-11-23 12:54:40,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2388066.6666666665, ans=0.0
2023-11-23 12:55:06,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2388200.0, ans=10.0
2023-11-23 12:55:19,854 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9550, loss[loss=0.06769, simple_loss=0.09339, pruned_loss=0.01365, audio_tagging_loss=0.007345, over 15167.00 frames. ], tot_loss[loss=0.07007, simple_loss=0.09339, pruned_loss=0.01413, audio_tagging_loss=0.009245, over 3058725.66 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 8.0
2023-11-23 12:55:22,287 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.501e+01 9.046e+01 9.743e+01 1.461e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 12:55:24,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.50 vs. limit=15.0
2023-11-23 12:55:30,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358250
2023-11-23 12:55:40,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2388333.3333333335, ans=0.0
2023-11-23 12:55:50,071 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.02 vs. limit=15.0
2023-11-23 12:55:57,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2388466.6666666665, ans=0.1
2023-11-23 12:56:04,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2388466.6666666665, ans=0.09899494936611666
2023-11-23 12:56:10,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2388533.3333333335, ans=0.125
2023-11-23 12:56:24,038 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9600, loss[loss=0.06895, simple_loss=0.09768, pruned_loss=0.01216, audio_tagging_loss=0.007948, over 14781.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09339, pruned_loss=0.01403, audio_tagging_loss=0.009305, over 3051005.27 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:56:24,566 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.92 vs. limit=15.0
2023-11-23 12:56:27,196 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.32 vs. limit=15.0
2023-11-23 12:56:27,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2388600.0, ans=0.2
2023-11-23 12:56:34,229 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358300
2023-11-23 12:56:34,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2388600.0, ans=0.0
2023-11-23 12:56:45,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.31 vs. limit=15.0
2023-11-23 12:56:45,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2388666.6666666665, ans=0.125
2023-11-23 12:57:03,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2388800.0, ans=0.125
2023-11-23 12:57:04,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2388800.0, ans=0.0
2023-11-23 12:57:19,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2388866.6666666665, ans=0.125
2023-11-23 12:57:21,139 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.87 vs. limit=15.0
2023-11-23 12:57:23,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2388866.6666666665, ans=0.125
2023-11-23 12:57:27,869 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9650, loss[loss=0.09292, simple_loss=0.1302, pruned_loss=0.02066, audio_tagging_loss=0.007171, over 15804.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09284, pruned_loss=0.01406, audio_tagging_loss=0.009313, over 3044674.27 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:57:30,912 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.376e+01 8.856e+01 9.540e+01 1.217e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 12:57:38,318 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358350
2023-11-23 12:57:39,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2389000.0, ans=0.125
2023-11-23 12:57:48,557 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.47 vs. limit=15.0
2023-11-23 12:58:27,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2389200.0, ans=0.5
2023-11-23 12:58:31,816 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9700, loss[loss=0.06762, simple_loss=0.08641, pruned_loss=0.01347, audio_tagging_loss=0.01095, over 14786.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09201, pruned_loss=0.01395, audio_tagging_loss=0.009183, over 3039763.35 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:58:41,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358400
2023-11-23 12:58:46,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2389333.3333333335, ans=0.2
2023-11-23 12:59:12,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.23 vs. limit=15.0
2023-11-23 12:59:19,264 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 12:59:31,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2389533.3333333335, ans=0.125
2023-11-23 12:59:36,168 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9750, loss[loss=0.06401, simple_loss=0.09082, pruned_loss=0.01051, audio_tagging_loss=0.008085, over 15443.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09238, pruned_loss=0.0138, audio_tagging_loss=0.009051, over 3039454.55 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 12:59:39,183 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.975e+01 8.327e+01 8.730e+01 9.519e+01 2.872e+02, threshold=1.746e+02, percent-clipped=1.0
2023-11-23 12:59:39,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2389600.0, ans=0.95
2023-11-23 12:59:47,161 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358450
2023-11-23 13:00:40,789 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9800, loss[loss=0.07548, simple_loss=0.1037, pruned_loss=0.01765, audio_tagging_loss=0.005996, over 15538.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.0928, pruned_loss=0.01393, audio_tagging_loss=0.009041, over 3041342.21 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:00:51,346 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358500
2023-11-23 13:01:16,676 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:01:28,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2390133.3333333335, ans=0.0
2023-11-23 13:01:38,907 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:01:39,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2390200.0, ans=0.125
2023-11-23 13:01:40,742 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.42 vs. limit=12.0
2023-11-23 13:01:45,068 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9850, loss[loss=0.05783, simple_loss=0.07416, pruned_loss=0.009181, audio_tagging_loss=0.01158, over 14677.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09174, pruned_loss=0.01363, audio_tagging_loss=0.00903, over 3037525.39 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:01:47,544 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.668e+01 8.339e+01 8.951e+01 9.575e+01 1.283e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 13:01:55,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358550
2023-11-23 13:02:03,106 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.51 vs. limit=22.5
2023-11-23 13:02:05,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2390333.3333333335, ans=0.125
2023-11-23 13:02:05,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2390333.3333333335, ans=0.1
2023-11-23 13:02:32,199 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.98 vs. limit=15.0
2023-11-23 13:02:49,000 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9900, loss[loss=0.05563, simple_loss=0.07155, pruned_loss=0.009777, audio_tagging_loss=0.01008, over 15795.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09204, pruned_loss=0.0137, audio_tagging_loss=0.008946, over 3037652.51 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:02:49,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2390600.0, ans=0.125
2023-11-23 13:02:59,919 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358600
2023-11-23 13:03:01,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2390666.6666666665, ans=0.125
2023-11-23 13:03:07,573 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.84 vs. limit=10.0
2023-11-23 13:03:37,239 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.96 vs. limit=22.5
2023-11-23 13:03:41,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2390866.6666666665, ans=0.2
2023-11-23 13:03:45,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2390866.6666666665, ans=0.125
2023-11-23 13:03:45,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2390866.6666666665, ans=0.125
2023-11-23 13:03:53,916 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 9950, loss[loss=0.04648, simple_loss=0.06619, pruned_loss=0.006261, audio_tagging_loss=0.007124, over 14782.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09236, pruned_loss=0.01389, audio_tagging_loss=0.008907, over 3035648.37 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:03:56,395 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.331e+01 8.358e+01 9.006e+01 9.900e+01 1.170e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 13:04:04,295 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358650
2023-11-23 13:04:12,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2391000.0, ans=0.125
2023-11-23 13:04:15,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2391000.0, ans=0.125
2023-11-23 13:04:19,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2391066.6666666665, ans=0.1
2023-11-23 13:04:29,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2391133.3333333335, ans=0.125
2023-11-23 13:04:46,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2391200.0, ans=0.1
2023-11-23 13:04:54,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2391200.0, ans=0.2
2023-11-23 13:04:57,628 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10000, loss[loss=0.07569, simple_loss=0.1062, pruned_loss=0.01364, audio_tagging_loss=0.008935, over 15442.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09209, pruned_loss=0.01392, audio_tagging_loss=0.008879, over 3038992.83 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:05:07,295 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358700
2023-11-23 13:05:11,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2391333.3333333335, ans=0.125
2023-11-23 13:05:16,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2391333.3333333335, ans=0.1
2023-11-23 13:05:19,734 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:06:01,422 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10050, loss[loss=0.08071, simple_loss=0.101, pruned_loss=0.0194, audio_tagging_loss=0.01079, over 15126.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09208, pruned_loss=0.01397, audio_tagging_loss=0.008951, over 3039799.50 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:06:03,776 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.755e+01 8.322e+01 8.907e+01 9.751e+01 1.198e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 13:06:11,100 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358750
2023-11-23 13:06:13,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2391666.6666666665, ans=0.125
2023-11-23 13:06:19,863 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:06:36,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2391733.3333333335, ans=0.0
2023-11-23 13:06:45,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2391800.0, ans=0.125
2023-11-23 13:06:47,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2391800.0, ans=0.0
2023-11-23 13:07:05,471 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10100, loss[loss=0.062, simple_loss=0.08449, pruned_loss=0.01061, audio_tagging_loss=0.009148, over 15067.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09229, pruned_loss=0.01388, audio_tagging_loss=0.009014, over 3041715.95 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:07:10,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2391933.3333333335, ans=0.2
2023-11-23 13:07:14,311 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.14 vs. limit=22.5
2023-11-23 13:07:16,181 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358800
2023-11-23 13:07:28,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2392000.0, ans=0.125
2023-11-23 13:07:57,879 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:07:59,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2392200.0, ans=0.125
2023-11-23 13:08:10,178 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10150, loss[loss=0.05628, simple_loss=0.08298, pruned_loss=0.006998, audio_tagging_loss=0.007789, over 14323.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09243, pruned_loss=0.01392, audio_tagging_loss=0.009071, over 3043750.36 frames. ], batch size: 52, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:08:13,707 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.825e+01 8.548e+01 9.170e+01 9.737e+01 1.223e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 13:08:19,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358850
2023-11-23 13:08:21,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2392333.3333333335, ans=0.125
2023-11-23 13:08:28,144 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.54 vs. limit=15.0
2023-11-23 13:08:40,909 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:08:42,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2392400.0, ans=0.0
2023-11-23 13:09:10,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2392533.3333333335, ans=0.125
2023-11-23 13:09:10,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2392533.3333333335, ans=0.0
2023-11-23 13:09:13,910 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10200, loss[loss=0.06336, simple_loss=0.08314, pruned_loss=0.01263, audio_tagging_loss=0.009159, over 15168.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.0919, pruned_loss=0.01383, audio_tagging_loss=0.00913, over 3038875.08 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:09:23,743 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358900
2023-11-23 13:09:39,580 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:09:39,921 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:09:39,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2392733.3333333335, ans=0.0
2023-11-23 13:10:04,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2392866.6666666665, ans=0.125
2023-11-23 13:10:06,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2392866.6666666665, ans=0.125
2023-11-23 13:10:13,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2392866.6666666665, ans=0.125
2023-11-23 13:10:18,485 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10250, loss[loss=0.07138, simple_loss=0.1032, pruned_loss=0.01156, audio_tagging_loss=0.008238, over 14606.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.0926, pruned_loss=0.01387, audio_tagging_loss=0.009092, over 3037872.85 frames. ], batch size: 53, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:10:18,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2392933.3333333335, ans=0.125
2023-11-23 13:10:22,068 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.777e+01 8.314e+01 9.044e+01 9.616e+01 1.457e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 13:10:28,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 358950
2023-11-23 13:10:28,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2392933.3333333335, ans=0.125
2023-11-23 13:10:31,112 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.67 vs. limit=5.0
2023-11-23 13:10:49,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2393066.6666666665, ans=0.125
2023-11-23 13:10:51,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2393066.6666666665, ans=0.125
2023-11-23 13:11:22,494 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.63 vs. limit=15.0
2023-11-23 13:11:23,028 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10300, loss[loss=0.07043, simple_loss=0.1019, pruned_loss=0.01037, audio_tagging_loss=0.009104, over 15284.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09187, pruned_loss=0.01372, audio_tagging_loss=0.009174, over 3043737.66 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:11:26,792 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.78 vs. limit=22.5
2023-11-23 13:11:27,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2393266.6666666665, ans=0.125
2023-11-23 13:11:33,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359000
2023-11-23 13:12:04,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2393466.6666666665, ans=0.1
2023-11-23 13:12:11,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2393466.6666666665, ans=0.125
2023-11-23 13:12:26,620 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10350, loss[loss=0.0668, simple_loss=0.08793, pruned_loss=0.01273, audio_tagging_loss=0.01011, over 14937.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09198, pruned_loss=0.01369, audio_tagging_loss=0.009268, over 3043895.94 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:12:30,161 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.913e+01 8.344e+01 8.692e+01 9.326e+01 1.149e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-23 13:12:36,430 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359050
2023-11-23 13:12:40,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2393666.6666666665, ans=0.0
2023-11-23 13:12:42,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2393666.6666666665, ans=0.125
2023-11-23 13:13:00,440 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0
2023-11-23 13:13:22,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2393866.6666666665, ans=0.125
2023-11-23 13:13:30,179 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10400, loss[loss=0.05628, simple_loss=0.06685, pruned_loss=0.009969, audio_tagging_loss=0.01289, over 14259.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09346, pruned_loss=0.01392, audio_tagging_loss=0.009223, over 3046236.48 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:13:30,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2393933.3333333335, ans=0.125
2023-11-23 13:13:39,923 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359100
2023-11-23 13:13:45,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2394000.0, ans=0.125
2023-11-23 13:14:05,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2394066.6666666665, ans=0.1
2023-11-23 13:14:21,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2394200.0, ans=0.1
2023-11-23 13:14:23,119 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.69 vs. limit=15.0
2023-11-23 13:14:33,792 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10450, loss[loss=0.06376, simple_loss=0.08096, pruned_loss=0.01104, audio_tagging_loss=0.01224, over 14489.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09274, pruned_loss=0.01379, audio_tagging_loss=0.009144, over 3039886.04 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:14:37,445 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.401e+01 9.132e+01 1.001e+02 1.338e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-23 13:14:43,731 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359150
2023-11-23 13:14:57,142 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.24 vs. limit=15.0
2023-11-23 13:14:59,109 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:15:15,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2394466.6666666665, ans=0.1
2023-11-23 13:15:30,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2394533.3333333335, ans=0.125
2023-11-23 13:15:33,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2394533.3333333335, ans=0.125
2023-11-23 13:15:37,335 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10500, loss[loss=0.05964, simple_loss=0.07595, pruned_loss=0.01116, audio_tagging_loss=0.01051, over 15104.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09257, pruned_loss=0.01368, audio_tagging_loss=0.009126, over 3044954.21 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:15:47,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359200
2023-11-23 13:15:58,604 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:16:10,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0
2023-11-23 13:16:23,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2394800.0, ans=0.0
2023-11-23 13:16:36,961 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=11.60 vs. limit=15.0
2023-11-23 13:16:40,188 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:16:41,121 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10550, loss[loss=0.08219, simple_loss=0.1072, pruned_loss=0.02084, audio_tagging_loss=0.007742, over 15207.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09215, pruned_loss=0.01362, audio_tagging_loss=0.009025, over 3046062.82 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:16:45,351 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.241e+01 8.844e+01 9.462e+01 1.240e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 13:16:51,722 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359250
2023-11-23 13:16:53,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2395000.0, ans=0.125
2023-11-23 13:17:02,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.76 vs. limit=22.5
2023-11-23 13:17:30,581 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.45 vs. limit=15.0
2023-11-23 13:17:42,022 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.51 vs. limit=15.0
2023-11-23 13:17:45,208 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10600, loss[loss=0.07914, simple_loss=0.1043, pruned_loss=0.01906, audio_tagging_loss=0.00793, over 13867.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09169, pruned_loss=0.01354, audio_tagging_loss=0.009026, over 3051816.84 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:17:55,841 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359300
2023-11-23 13:18:10,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2395400.0, ans=0.0
2023-11-23 13:18:45,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2395533.3333333335, ans=0.1
2023-11-23 13:18:47,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2395533.3333333335, ans=0.125
2023-11-23 13:18:49,240 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10650, loss[loss=0.06524, simple_loss=0.08539, pruned_loss=0.01411, audio_tagging_loss=0.008434, over 15840.00 frames. ], tot_loss[loss=0.06903, simple_loss=0.09292, pruned_loss=0.01365, audio_tagging_loss=0.008919, over 3053942.26 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:18:54,001 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.581e+01 8.219e+01 8.921e+01 9.796e+01 1.166e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 13:18:56,480 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.64 vs. limit=15.0
2023-11-23 13:18:59,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359350
2023-11-23 13:19:31,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2395800.0, ans=0.0
2023-11-23 13:19:34,588 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.94 vs. limit=22.5
2023-11-23 13:19:38,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2395800.0, ans=0.0
2023-11-23 13:19:44,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2395866.6666666665, ans=0.125
2023-11-23 13:19:52,827 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10700, loss[loss=0.05956, simple_loss=0.08272, pruned_loss=0.007725, audio_tagging_loss=0.01047, over 15330.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09269, pruned_loss=0.01373, audio_tagging_loss=0.008965, over 3053708.31 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:20:03,402 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359400
2023-11-23 13:20:06,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2396000.0, ans=0.07
2023-11-23 13:20:18,641 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.86 vs. limit=15.0
2023-11-23 13:20:19,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2396066.6666666665, ans=0.125
2023-11-23 13:20:34,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2396133.3333333335, ans=0.125
2023-11-23 13:20:35,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2396133.3333333335, ans=0.0
2023-11-23 13:20:38,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.22 vs. limit=12.0
2023-11-23 13:20:55,012 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.80 vs. limit=6.0
2023-11-23 13:20:57,075 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10750, loss[loss=0.06971, simple_loss=0.09679, pruned_loss=0.01456, audio_tagging_loss=0.006757, over 14904.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09324, pruned_loss=0.01382, audio_tagging_loss=0.008882, over 3053592.16 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:21:02,308 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.422e+01 8.871e+01 9.738e+01 1.299e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 13:21:07,224 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359450
2023-11-23 13:21:29,171 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.76 vs. limit=15.0
2023-11-23 13:21:41,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2396466.6666666665, ans=0.125
2023-11-23 13:21:54,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2396533.3333333335, ans=0.0
2023-11-23 13:21:57,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2396533.3333333335, ans=0.1
2023-11-23 13:22:01,023 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10800, loss[loss=0.04683, simple_loss=0.05464, pruned_loss=0.009046, audio_tagging_loss=0.01046, over 16453.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09339, pruned_loss=0.01382, audio_tagging_loss=0.00885, over 3053499.71 frames. ], batch size: 64, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:22:02,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2396600.0, ans=0.2
2023-11-23 13:22:11,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359500
2023-11-23 13:22:25,279 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.47 vs. limit=10.0
2023-11-23 13:22:43,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2396800.0, ans=0.2
2023-11-23 13:23:04,159 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10850, loss[loss=0.06779, simple_loss=0.08922, pruned_loss=0.01483, audio_tagging_loss=0.008352, over 14747.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09296, pruned_loss=0.01389, audio_tagging_loss=0.008932, over 3049681.53 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:23:11,467 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.664e+01 8.074e+01 8.715e+01 9.516e+01 1.146e+02, threshold=1.743e+02, percent-clipped=0.0
2023-11-23 13:23:15,334 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359550
2023-11-23 13:23:15,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2396933.3333333335, ans=0.125
2023-11-23 13:23:21,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2397000.0, ans=0.0
2023-11-23 13:23:24,509 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.35 vs. limit=15.0
2023-11-23 13:23:45,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2397133.3333333335, ans=0.125
2023-11-23 13:24:05,401 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:24:06,065 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.62 vs. limit=15.0
2023-11-23 13:24:07,829 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10900, loss[loss=0.06264, simple_loss=0.08324, pruned_loss=0.01123, audio_tagging_loss=0.009792, over 14617.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09312, pruned_loss=0.01403, audio_tagging_loss=0.008965, over 3045620.81 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:24:10,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2397266.6666666665, ans=0.1
2023-11-23 13:24:11,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2397266.6666666665, ans=0.05
2023-11-23 13:24:13,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2397266.6666666665, ans=0.125
2023-11-23 13:24:13,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2397266.6666666665, ans=0.125
2023-11-23 13:24:18,502 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359600
2023-11-23 13:24:22,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2397333.3333333335, ans=0.2
2023-11-23 13:24:27,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2397333.3333333335, ans=0.125
2023-11-23 13:24:52,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2397466.6666666665, ans=0.1
2023-11-23 13:25:11,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2397600.0, ans=0.0
2023-11-23 13:25:12,106 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 10950, loss[loss=0.0766, simple_loss=0.1122, pruned_loss=0.01162, audio_tagging_loss=0.008894, over 16093.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09216, pruned_loss=0.01388, audio_tagging_loss=0.008999, over 3047362.63 frames. ], batch size: 59, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:25:18,224 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.360e+01 9.047e+01 9.764e+01 1.279e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 13:25:21,982 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359650
2023-11-23 13:25:48,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2397733.3333333335, ans=0.05
2023-11-23 13:26:06,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2397866.6666666665, ans=0.125
2023-11-23 13:26:11,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2397866.6666666665, ans=0.0
2023-11-23 13:26:16,487 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11000, loss[loss=0.03521, simple_loss=0.03858, pruned_loss=0.00449, audio_tagging_loss=0.01143, over 16597.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09176, pruned_loss=0.01363, audio_tagging_loss=0.008997, over 3046802.07 frames. ], batch size: 65, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:26:16,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2397933.3333333335, ans=0.0
2023-11-23 13:26:21,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2397933.3333333335, ans=0.125
2023-11-23 13:26:27,198 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359700
2023-11-23 13:26:28,854 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:26:44,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2398066.6666666665, ans=0.1
2023-11-23 13:27:05,320 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.99 vs. limit=6.0
2023-11-23 13:27:13,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2398200.0, ans=0.0
2023-11-23 13:27:14,062 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.00 vs. limit=22.5
2023-11-23 13:27:22,150 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11050, loss[loss=0.07318, simple_loss=0.09969, pruned_loss=0.01474, audio_tagging_loss=0.008592, over 16132.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09197, pruned_loss=0.01371, audio_tagging_loss=0.009005, over 3046154.47 frames. ], batch size: 61, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:27:22,730 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.05 vs. limit=10.0
2023-11-23 13:27:28,286 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.620e+01 9.222e+01 9.990e+01 1.274e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-23 13:27:28,526 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:27:32,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359750
2023-11-23 13:27:37,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2398333.3333333335, ans=0.0
2023-11-23 13:27:46,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2398400.0, ans=0.125
2023-11-23 13:27:53,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2398400.0, ans=0.0
2023-11-23 13:28:27,248 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11100, loss[loss=0.07031, simple_loss=0.08437, pruned_loss=0.01687, audio_tagging_loss=0.01126, over 16104.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.0925, pruned_loss=0.01387, audio_tagging_loss=0.009085, over 3043823.51 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:28:37,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359800
2023-11-23 13:28:50,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2398666.6666666665, ans=0.0
2023-11-23 13:28:50,358 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.60 vs. limit=15.0
2023-11-23 13:28:51,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2398733.3333333335, ans=0.035
2023-11-23 13:29:13,411 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:29:22,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2398866.6666666665, ans=0.0
2023-11-23 13:29:31,462 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11150, loss[loss=0.093, simple_loss=0.1338, pruned_loss=0.01513, audio_tagging_loss=0.01097, over 16980.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09251, pruned_loss=0.01391, audio_tagging_loss=0.009179, over 3047063.11 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:29:33,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2398933.3333333335, ans=0.125
2023-11-23 13:29:35,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2398933.3333333335, ans=0.0
2023-11-23 13:29:37,471 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.168e+01 8.369e+01 8.812e+01 9.424e+01 1.136e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 13:29:41,366 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359850
2023-11-23 13:29:41,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2398933.3333333335, ans=0.0
2023-11-23 13:29:52,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2399000.0, ans=0.0
2023-11-23 13:30:16,906 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.01 vs. limit=22.5
2023-11-23 13:30:19,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2399133.3333333335, ans=0.125
2023-11-23 13:30:34,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2399266.6666666665, ans=0.125
2023-11-23 13:30:35,623 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11200, loss[loss=0.08039, simple_loss=0.1093, pruned_loss=0.01805, audio_tagging_loss=0.007701, over 15290.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09232, pruned_loss=0.01387, audio_tagging_loss=0.009232, over 3051689.82 frames. ], batch size: 55, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:30:37,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2399266.6666666665, ans=0.125
2023-11-23 13:30:46,061 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359900
2023-11-23 13:31:14,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2399466.6666666665, ans=0.2
2023-11-23 13:31:17,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2399466.6666666665, ans=0.0
2023-11-23 13:31:26,187 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.68 vs. limit=15.0
2023-11-23 13:31:39,965 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11250, loss[loss=0.06543, simple_loss=0.08484, pruned_loss=0.01475, audio_tagging_loss=0.008259, over 16640.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09162, pruned_loss=0.0138, audio_tagging_loss=0.009288, over 3048777.28 frames. ], batch size: 61, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:31:47,332 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.154e+01 8.449e+01 9.308e+01 1.008e+02 1.171e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-23 13:31:49,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 359950
2023-11-23 13:31:56,180 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:32:14,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2399733.3333333335, ans=10.0
2023-11-23 13:32:21,284 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.77 vs. limit=22.5
2023-11-23 13:32:31,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2399866.6666666665, ans=0.125
2023-11-23 13:32:43,479 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11300, loss[loss=0.08055, simple_loss=0.1173, pruned_loss=0.01577, audio_tagging_loss=0.006119, over 15337.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09231, pruned_loss=0.01402, audio_tagging_loss=0.009079, over 3051573.60 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:32:46,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2399933.3333333335, ans=0.125
2023-11-23 13:32:53,431 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360000
2023-11-23 13:32:54,957 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-360000.pt
2023-11-23 13:33:10,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.68 vs. limit=10.0
2023-11-23 13:33:14,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2400066.6666666665, ans=0.07
2023-11-23 13:33:15,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2400066.6666666665, ans=0.1
2023-11-23 13:33:18,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2400066.6666666665, ans=0.125
2023-11-23 13:33:42,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2400200.0, ans=0.125
2023-11-23 13:33:50,670 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11350, loss[loss=0.05988, simple_loss=0.08318, pruned_loss=0.009464, audio_tagging_loss=0.008829, over 15764.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09262, pruned_loss=0.0141, audio_tagging_loss=0.009004, over 3053406.68 frames. ], batch size: 63, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:33:51,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2400266.6666666665, ans=0.2
2023-11-23 13:33:57,999 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.206e+01 9.153e+01 9.834e+01 1.397e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-23 13:34:00,516 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360050
2023-11-23 13:34:13,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2400333.3333333335, ans=0.125
2023-11-23 13:34:36,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2400466.6666666665, ans=0.0
2023-11-23 13:34:37,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2400466.6666666665, ans=0.0
2023-11-23 13:34:45,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2400533.3333333335, ans=0.125
2023-11-23 13:34:54,325 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11400, loss[loss=0.08961, simple_loss=0.1181, pruned_loss=0.02063, audio_tagging_loss=0.009929, over 15638.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09327, pruned_loss=0.01415, audio_tagging_loss=0.008775, over 3048689.96 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:34:59,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2400600.0, ans=0.2
2023-11-23 13:35:04,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360100
2023-11-23 13:35:26,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2400733.3333333335, ans=0.125
2023-11-23 13:35:30,147 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.13 vs. limit=15.0
2023-11-23 13:35:34,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2400800.0, ans=0.1
2023-11-23 13:35:39,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2400800.0, ans=0.125
2023-11-23 13:35:41,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2400800.0, ans=0.125
2023-11-23 13:35:46,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2400866.6666666665, ans=0.2
2023-11-23 13:35:57,412 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11450, loss[loss=0.08481, simple_loss=0.1107, pruned_loss=0.01847, audio_tagging_loss=0.01102, over 15679.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09312, pruned_loss=0.0142, audio_tagging_loss=0.008834, over 3041060.68 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:36:04,739 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 8.159e+01 8.794e+01 9.452e+01 1.261e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 13:36:07,372 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360150
2023-11-23 13:36:08,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2401000.0, ans=0.1
2023-11-23 13:36:21,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2401000.0, ans=0.0
2023-11-23 13:36:22,347 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:36:29,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2401066.6666666665, ans=0.0
2023-11-23 13:36:42,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2401133.3333333335, ans=0.125
2023-11-23 13:36:55,986 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.97 vs. limit=6.0
2023-11-23 13:37:01,748 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11500, loss[loss=0.05734, simple_loss=0.0742, pruned_loss=0.008262, audio_tagging_loss=0.01197, over 16803.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09331, pruned_loss=0.01435, audio_tagging_loss=0.00891, over 3047474.34 frames. ], batch size: 62, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:37:05,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2401266.6666666665, ans=0.2
2023-11-23 13:37:12,696 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360200
2023-11-23 13:37:12,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2401266.6666666665, ans=0.0
2023-11-23 13:37:16,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2401333.3333333335, ans=0.125
2023-11-23 13:37:22,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2401333.3333333335, ans=0.07
2023-11-23 13:37:22,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2401333.3333333335, ans=0.125
2023-11-23 13:37:30,979 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.36 vs. limit=15.0
2023-11-23 13:37:47,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2401466.6666666665, ans=0.125
2023-11-23 13:37:50,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2401466.6666666665, ans=0.125
2023-11-23 13:37:54,842 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:38:02,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2401533.3333333335, ans=0.2
2023-11-23 13:38:06,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2401600.0, ans=0.0
2023-11-23 13:38:07,653 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11550, loss[loss=0.06687, simple_loss=0.08558, pruned_loss=0.01466, audio_tagging_loss=0.009414, over 14567.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09312, pruned_loss=0.0143, audio_tagging_loss=0.008924, over 3047457.98 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:38:15,051 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.476e+01 9.086e+01 9.753e+01 1.372e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 13:38:18,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360250
2023-11-23 13:38:18,737 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.16 vs. limit=6.0
2023-11-23 13:38:39,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2401733.3333333335, ans=0.125
2023-11-23 13:38:47,259 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 13:38:48,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2401800.0, ans=0.125
2023-11-23 13:38:51,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2401800.0, ans=0.2
2023-11-23 13:38:56,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2401800.0, ans=0.125
2023-11-23 13:39:00,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2401866.6666666665, ans=0.125
2023-11-23 13:39:01,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2401866.6666666665, ans=0.2
2023-11-23 13:39:04,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2401866.6666666665, ans=0.125
2023-11-23 13:39:11,788 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11600, loss[loss=0.06608, simple_loss=0.07445, pruned_loss=0.01496, audio_tagging_loss=0.01389, over 13866.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09248, pruned_loss=0.01413, audio_tagging_loss=0.00895, over 3051891.32 frames. ], batch size: 54, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:39:21,678 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360300
2023-11-23 13:39:37,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2402066.6666666665, ans=0.125
2023-11-23 13:39:42,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2402066.6666666665, ans=0.2
2023-11-23 13:40:00,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2402133.3333333335, ans=0.125
2023-11-23 13:40:13,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2402266.6666666665, ans=0.2
2023-11-23 13:40:14,613 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11650, loss[loss=0.07259, simple_loss=0.1064, pruned_loss=0.01269, audio_tagging_loss=0.006701, over 14850.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09254, pruned_loss=0.01415, audio_tagging_loss=0.008905, over 3048902.46 frames. ], batch size: 53, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:40:22,541 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.635e+01 8.364e+01 9.300e+01 1.004e+02 1.361e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-23 13:40:25,051 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360350
2023-11-23 13:40:39,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2402400.0, ans=0.0
2023-11-23 13:40:47,734 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.80 vs. limit=15.0
2023-11-23 13:40:48,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2402400.0, ans=0.0
2023-11-23 13:41:11,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2402533.3333333335, ans=0.0
2023-11-23 13:41:16,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2402533.3333333335, ans=0.1
2023-11-23 13:41:16,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2402533.3333333335, ans=0.125
2023-11-23 13:41:17,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.50 vs. limit=15.0
2023-11-23 13:41:18,472 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11700, loss[loss=0.076, simple_loss=0.1031, pruned_loss=0.01438, audio_tagging_loss=0.01009, over 15539.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09212, pruned_loss=0.01396, audio_tagging_loss=0.009052, over 3049803.43 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:41:24,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2402600.0, ans=0.125
2023-11-23 13:41:29,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360400
2023-11-23 13:41:32,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2402666.6666666665, ans=0.1
2023-11-23 13:41:40,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2402666.6666666665, ans=0.125
2023-11-23 13:41:47,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2402733.3333333335, ans=0.125
2023-11-23 13:41:49,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2402733.3333333335, ans=15.0
2023-11-23 13:42:05,972 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.37 vs. limit=15.0
2023-11-23 13:42:17,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2402866.6666666665, ans=0.1
2023-11-23 13:42:23,004 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11750, loss[loss=0.06453, simple_loss=0.09285, pruned_loss=0.01038, audio_tagging_loss=0.00773, over 15587.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09221, pruned_loss=0.01392, audio_tagging_loss=0.009066, over 3054285.85 frames. ], batch size: 57, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:42:32,128 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.747e+01 8.361e+01 8.957e+01 9.749e+01 1.339e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 13:42:33,411 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360450
2023-11-23 13:42:37,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2403000.0, ans=0.125
2023-11-23 13:42:39,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2403000.0, ans=0.125
2023-11-23 13:42:59,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2403066.6666666665, ans=0.125
2023-11-23 13:43:14,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2403200.0, ans=0.1
2023-11-23 13:43:17,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2403200.0, ans=0.2
2023-11-23 13:43:22,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2403200.0, ans=0.0
2023-11-23 13:43:27,023 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11800, loss[loss=0.07094, simple_loss=0.09717, pruned_loss=0.01262, audio_tagging_loss=0.009741, over 15203.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.0912, pruned_loss=0.01376, audio_tagging_loss=0.009072, over 3044581.27 frames. ], batch size: 56, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:43:29,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff3.min_abs, batch_count=2403266.6666666665, ans=0.2
2023-11-23 13:43:37,704 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360500
2023-11-23 13:43:55,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2403400.0, ans=0.125
2023-11-23 13:44:02,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2403400.0, ans=0.125
2023-11-23 13:44:06,457 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.78 vs. limit=15.0
2023-11-23 13:44:31,209 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11850, loss[loss=0.06801, simple_loss=0.1, pruned_loss=0.009958, audio_tagging_loss=0.008056, over 15938.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09183, pruned_loss=0.01382, audio_tagging_loss=0.009248, over 3051282.67 frames. ], batch size: 58, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:44:40,395 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.225e+01 8.325e+01 8.959e+01 9.729e+01 1.337e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 13:44:41,696 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360550
2023-11-23 13:44:41,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2403600.0, ans=0.0
2023-11-23 13:45:00,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2403733.3333333335, ans=0.0
2023-11-23 13:45:15,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.44 vs. limit=22.5
2023-11-23 13:45:18,033 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.92 vs. limit=15.0
2023-11-23 13:45:19,529 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.92 vs. limit=15.0
2023-11-23 13:45:21,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2403866.6666666665, ans=0.125
2023-11-23 13:45:21,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2403866.6666666665, ans=0.125
2023-11-23 13:45:34,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2403933.3333333335, ans=0.2
2023-11-23 13:45:35,379 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11900, loss[loss=0.06738, simple_loss=0.0843, pruned_loss=0.01385, audio_tagging_loss=0.01138, over 16437.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09181, pruned_loss=0.01372, audio_tagging_loss=0.009354, over 3052040.68 frames. ], batch size: 63, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:45:45,354 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360600
2023-11-23 13:45:45,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2403933.3333333335, ans=0.0
2023-11-23 13:45:48,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2404000.0, ans=0.0
2023-11-23 13:45:58,040 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.12 vs. limit=10.0
2023-11-23 13:46:19,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2404133.3333333335, ans=0.0
2023-11-23 13:46:24,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2404133.3333333335, ans=0.1
2023-11-23 13:46:40,741 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 11950, loss[loss=0.05877, simple_loss=0.06924, pruned_loss=0.01332, audio_tagging_loss=0.01083, over 15435.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09063, pruned_loss=0.01357, audio_tagging_loss=0.009442, over 3054274.80 frames. ], batch size: 60, lr: 2.25e-03, grad_scale: 16.0
2023-11-23 13:46:44,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2404266.6666666665, ans=0.0
2023-11-23 13:46:50,079 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.842e+01 8.271e+01 8.961e+01 9.651e+01 1.560e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 13:46:51,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360650
2023-11-23 13:47:07,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2404400.0, ans=0.0
2023-11-23 13:47:15,953 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:47:23,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2404466.6666666665, ans=0.1
2023-11-23 13:47:30,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2404533.3333333335, ans=0.2
2023-11-23 13:47:31,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2404533.3333333335, ans=0.2
2023-11-23 13:47:42,940 INFO [train_asr.py:1221] (0/4) Epoch 30, batch 12000, loss[loss=0.0596, simple_loss=0.07407, pruned_loss=0.0118, audio_tagging_loss=0.01076, over 16462.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09165, pruned_loss=0.01367, audio_tagging_loss=0.009461, over 3060026.10 frames. ], batch size: 63, lr: 2.25e-03, grad_scale: 32.0
2023-11-23 13:47:42,943 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 13:48:25,611 INFO [train_asr.py:1253] (0/4) Epoch 30, validation: loss=0.05798, simple_loss=0.05115, pruned_loss=0.00515, audio_tagging_loss=0.02725, over 4681554.00 frames. 
2023-11-23 13:48:25,612 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 13:48:31,523 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.59 vs. limit=10.0
2023-11-23 13:48:33,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2404600.0, ans=0.125
2023-11-23 13:48:35,649 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360700
2023-11-23 13:48:56,244 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-30.pt
2023-11-23 13:49:30,374 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 0, loss[loss=0.07302, simple_loss=0.07602, pruned_loss=0.01179, audio_tagging_loss=0.02322, over 15097.00 frames. ], tot_loss[loss=0.07302, simple_loss=0.07602, pruned_loss=0.01179, audio_tagging_loss=0.02322, over 15097.00 frames. ], batch size: 60, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:49:30,377 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 13:50:05,329 INFO [train_asr.py:1253] (0/4) Epoch 31, validation: loss=0.05797, simple_loss=0.05105, pruned_loss=0.005059, audio_tagging_loss=0.02738, over 4681554.00 frames. 
2023-11-23 13:50:05,330 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 13:50:28,909 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.61 vs. limit=6.0
2023-11-23 13:50:46,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2404966.6666666665, ans=0.125
2023-11-23 13:50:47,544 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.427e+01 8.784e+01 9.433e+01 1.048e+02 1.296e+02, threshold=1.887e+02, percent-clipped=0.0
2023-11-23 13:50:48,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360750
2023-11-23 13:51:10,467 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 50, loss[loss=0.06769, simple_loss=0.07753, pruned_loss=0.009838, audio_tagging_loss=0.01909, over 15092.00 frames. ], tot_loss[loss=0.0738, simple_loss=0.08821, pruned_loss=0.01234, audio_tagging_loss=0.01736, over 688717.13 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:51:27,534 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.98 vs. limit=10.0
2023-11-23 13:51:29,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2405166.6666666665, ans=0.2
2023-11-23 13:51:34,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2405166.6666666665, ans=0.07
2023-11-23 13:51:48,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2405300.0, ans=0.0
2023-11-23 13:51:53,106 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.25 vs. limit=15.0
2023-11-23 13:51:53,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360800
2023-11-23 13:52:13,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.59 vs. limit=15.0
2023-11-23 13:52:16,523 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 100, loss[loss=0.05881, simple_loss=0.0674, pruned_loss=0.007586, audio_tagging_loss=0.01752, over 14714.00 frames. ], tot_loss[loss=0.07395, simple_loss=0.08917, pruned_loss=0.01247, audio_tagging_loss=0.01689, over 1204421.35 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:52:18,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2405433.3333333335, ans=0.0
2023-11-23 13:52:19,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2405433.3333333335, ans=0.1
2023-11-23 13:52:26,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2405433.3333333335, ans=0.125
2023-11-23 13:52:49,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2405566.6666666665, ans=0.125
2023-11-23 13:52:56,117 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.93 vs. limit=15.0
2023-11-23 13:52:57,685 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.952e+01 8.885e+01 9.566e+01 1.092e+02 1.525e+02, threshold=1.913e+02, percent-clipped=0.0
2023-11-23 13:52:59,048 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360850
2023-11-23 13:53:18,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2405700.0, ans=0.125
2023-11-23 13:53:20,665 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 150, loss[loss=0.05709, simple_loss=0.06451, pruned_loss=0.01095, audio_tagging_loss=0.01388, over 15630.00 frames. ], tot_loss[loss=0.07347, simple_loss=0.0906, pruned_loss=0.01315, audio_tagging_loss=0.01503, over 1617424.97 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:53:21,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2405766.6666666665, ans=0.07
2023-11-23 13:53:34,198 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.50 vs. limit=15.0
2023-11-23 13:53:43,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2405833.3333333335, ans=0.125
2023-11-23 13:54:03,644 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360900
2023-11-23 13:54:06,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2405966.6666666665, ans=0.0
2023-11-23 13:54:15,329 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.02 vs. limit=15.0
2023-11-23 13:54:25,015 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 200, loss[loss=0.05633, simple_loss=0.07326, pruned_loss=0.01182, audio_tagging_loss=0.007887, over 15354.00 frames. ], tot_loss[loss=0.07279, simple_loss=0.09197, pruned_loss=0.01359, audio_tagging_loss=0.01322, over 1933313.60 frames. ], batch size: 62, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:54:40,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2406166.6666666665, ans=0.0
2023-11-23 13:55:06,203 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.199e+01 8.350e+01 9.246e+01 9.961e+01 1.409e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-23 13:55:07,607 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 360950
2023-11-23 13:55:12,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2406300.0, ans=0.1
2023-11-23 13:55:13,592 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.56 vs. limit=15.0
2023-11-23 13:55:25,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2406366.6666666665, ans=0.125
2023-11-23 13:55:25,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2406366.6666666665, ans=0.2
2023-11-23 13:55:30,784 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 250, loss[loss=0.08738, simple_loss=0.1264, pruned_loss=0.01847, audio_tagging_loss=0.005701, over 14830.00 frames. ], tot_loss[loss=0.07253, simple_loss=0.09372, pruned_loss=0.01385, audio_tagging_loss=0.01183, over 2188580.53 frames. ], batch size: 56, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:55:31,491 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.17 vs. limit=22.5
2023-11-23 13:55:42,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2406500.0, ans=0.0
2023-11-23 13:55:56,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.24 vs. limit=10.0
2023-11-23 13:56:08,359 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.25 vs. limit=15.0
2023-11-23 13:56:11,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2406633.3333333335, ans=0.1
2023-11-23 13:56:13,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361000
2023-11-23 13:56:35,024 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 300, loss[loss=0.07367, simple_loss=0.09912, pruned_loss=0.01412, audio_tagging_loss=0.009992, over 15319.00 frames. ], tot_loss[loss=0.07207, simple_loss=0.09391, pruned_loss=0.01398, audio_tagging_loss=0.01114, over 2380807.51 frames. ], batch size: 58, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:56:50,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2406833.3333333335, ans=0.2
2023-11-23 13:56:53,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2406833.3333333335, ans=0.1
2023-11-23 13:56:58,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2406833.3333333335, ans=0.125
2023-11-23 13:57:01,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2406900.0, ans=0.125
2023-11-23 13:57:16,328 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.477e+01 9.124e+01 9.859e+01 1.340e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 13:57:17,707 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361050
2023-11-23 13:57:30,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2407033.3333333335, ans=0.1
2023-11-23 13:57:32,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2407033.3333333335, ans=0.125
2023-11-23 13:57:38,725 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 350, loss[loss=0.08097, simple_loss=0.1145, pruned_loss=0.01879, audio_tagging_loss=0.004923, over 14579.00 frames. ], tot_loss[loss=0.07161, simple_loss=0.09423, pruned_loss=0.01401, audio_tagging_loss=0.01048, over 2535617.81 frames. ], batch size: 54, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:58:01,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2407166.6666666665, ans=0.125
2023-11-23 13:58:21,938 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361100
2023-11-23 13:58:44,373 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 400, loss[loss=0.06201, simple_loss=0.0854, pruned_loss=0.01231, audio_tagging_loss=0.007002, over 15332.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09309, pruned_loss=0.01381, audio_tagging_loss=0.01016, over 2651356.62 frames. ], batch size: 58, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 13:58:52,708 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 13:58:56,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2407500.0, ans=0.125
2023-11-23 13:59:04,204 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.82 vs. limit=15.0
2023-11-23 13:59:14,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2407566.6666666665, ans=0.0
2023-11-23 13:59:23,521 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.61 vs. limit=15.0
2023-11-23 13:59:27,039 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.911e+01 8.381e+01 8.890e+01 9.709e+01 1.192e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-23 13:59:27,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361150
2023-11-23 13:59:30,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2407633.3333333335, ans=0.2
2023-11-23 13:59:32,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2407633.3333333335, ans=0.1
2023-11-23 13:59:44,768 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.97 vs. limit=6.0
2023-11-23 13:59:49,037 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 450, loss[loss=0.0603, simple_loss=0.0817, pruned_loss=0.01232, audio_tagging_loss=0.007127, over 15168.00 frames. ], tot_loss[loss=0.0701, simple_loss=0.09287, pruned_loss=0.01382, audio_tagging_loss=0.009849, over 2740348.05 frames. ], batch size: 58, lr: 2.21e-03, grad_scale: 32.0
2023-11-23 14:00:05,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2407833.3333333335, ans=0.0
2023-11-23 14:00:17,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2407900.0, ans=0.95
2023-11-23 14:00:25,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2407900.0, ans=0.2
2023-11-23 14:00:25,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2407900.0, ans=0.1
2023-11-23 14:00:31,514 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361200
2023-11-23 14:00:38,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0
2023-11-23 14:00:40,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2408033.3333333335, ans=0.07
2023-11-23 14:00:52,587 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 500, loss[loss=0.05981, simple_loss=0.08304, pruned_loss=0.01121, audio_tagging_loss=0.007076, over 15394.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09103, pruned_loss=0.01365, audio_tagging_loss=0.009629, over 2803400.24 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:01:22,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2408233.3333333335, ans=0.125
2023-11-23 14:01:29,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2408233.3333333335, ans=0.125
2023-11-23 14:01:35,862 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361250
2023-11-23 14:01:38,176 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.971e+01 8.202e+01 8.738e+01 9.216e+01 1.113e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 14:01:41,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2408300.0, ans=0.1
2023-11-23 14:01:55,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2408366.6666666665, ans=0.125
2023-11-23 14:01:57,511 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.92 vs. limit=15.0
2023-11-23 14:01:58,084 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 550, loss[loss=0.0696, simple_loss=0.08861, pruned_loss=0.01551, audio_tagging_loss=0.009778, over 14500.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09072, pruned_loss=0.01373, audio_tagging_loss=0.009611, over 2860238.35 frames. ], batch size: 54, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:02:03,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2408433.3333333335, ans=0.0
2023-11-23 14:02:05,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2408433.3333333335, ans=0.125
2023-11-23 14:02:35,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2408633.3333333335, ans=0.125
2023-11-23 14:02:40,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361300
2023-11-23 14:02:56,986 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:03:02,849 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 600, loss[loss=0.083, simple_loss=0.1133, pruned_loss=0.0167, audio_tagging_loss=0.009619, over 15587.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.0917, pruned_loss=0.01381, audio_tagging_loss=0.009477, over 2899557.58 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:03:12,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2408766.6666666665, ans=0.1
2023-11-23 14:03:22,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2408833.3333333335, ans=0.125
2023-11-23 14:03:23,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2408833.3333333335, ans=0.0
2023-11-23 14:03:24,113 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0
2023-11-23 14:03:33,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2408900.0, ans=0.0
2023-11-23 14:03:45,714 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361350
2023-11-23 14:03:48,037 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.005e+01 8.522e+01 9.044e+01 1.001e+02 1.721e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 14:03:50,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2408966.6666666665, ans=0.2
2023-11-23 14:04:04,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2409033.3333333335, ans=0.125
2023-11-23 14:04:06,540 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 650, loss[loss=0.06107, simple_loss=0.07755, pruned_loss=0.01342, audio_tagging_loss=0.008876, over 16379.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09223, pruned_loss=0.01404, audio_tagging_loss=0.0094, over 2925736.29 frames. ], batch size: 62, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:04:16,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2409100.0, ans=0.125
2023-11-23 14:04:41,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2409233.3333333335, ans=0.0
2023-11-23 14:04:46,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2409300.0, ans=0.125
2023-11-23 14:04:49,501 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361400
2023-11-23 14:05:06,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2409366.6666666665, ans=0.0
2023-11-23 14:05:11,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2409433.3333333335, ans=0.0
2023-11-23 14:05:12,028 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 700, loss[loss=0.05084, simple_loss=0.06621, pruned_loss=0.007596, audio_tagging_loss=0.01013, over 14100.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09018, pruned_loss=0.01345, audio_tagging_loss=0.009486, over 2948110.56 frames. ], batch size: 52, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:05:13,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2409433.3333333335, ans=0.125
2023-11-23 14:05:24,073 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.10 vs. limit=15.0
2023-11-23 14:05:25,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2409500.0, ans=0.0
2023-11-23 14:05:30,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2409500.0, ans=0.0
2023-11-23 14:05:44,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2409566.6666666665, ans=0.125
2023-11-23 14:05:53,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2409633.3333333335, ans=0.0
2023-11-23 14:05:54,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361450
2023-11-23 14:05:54,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2409633.3333333335, ans=0.125
2023-11-23 14:05:56,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2409633.3333333335, ans=0.0
2023-11-23 14:05:57,071 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.383e+01 9.026e+01 9.852e+01 1.230e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 14:05:57,787 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.53 vs. limit=15.0
2023-11-23 14:06:05,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2409700.0, ans=0.125
2023-11-23 14:06:11,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2409700.0, ans=0.0
2023-11-23 14:06:17,515 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 750, loss[loss=0.06808, simple_loss=0.09644, pruned_loss=0.01056, audio_tagging_loss=0.009302, over 14919.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09091, pruned_loss=0.01356, audio_tagging_loss=0.009445, over 2979074.68 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 8.0
2023-11-23 14:06:26,628 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.47 vs. limit=15.0
2023-11-23 14:06:37,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2409833.3333333335, ans=0.125
2023-11-23 14:06:43,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2409900.0, ans=0.2
2023-11-23 14:06:59,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361500
2023-11-23 14:07:01,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2409966.6666666665, ans=0.125
2023-11-23 14:07:21,106 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 800, loss[loss=0.05668, simple_loss=0.07035, pruned_loss=0.0106, audio_tagging_loss=0.0109, over 13654.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09068, pruned_loss=0.01358, audio_tagging_loss=0.009584, over 2995209.17 frames. ], batch size: 54, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:07:23,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2410100.0, ans=0.125
2023-11-23 14:07:23,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2410100.0, ans=0.125
2023-11-23 14:07:26,635 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.00 vs. limit=15.0
2023-11-23 14:07:31,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2410100.0, ans=0.1
2023-11-23 14:07:42,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2410166.6666666665, ans=15.0
2023-11-23 14:07:43,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2410166.6666666665, ans=0.1
2023-11-23 14:07:48,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2410233.3333333335, ans=0.0
2023-11-23 14:08:04,599 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361550
2023-11-23 14:08:06,965 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.503e+01 9.179e+01 9.718e+01 1.363e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-23 14:08:12,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2410366.6666666665, ans=0.1
2023-11-23 14:08:26,915 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 850, loss[loss=0.06535, simple_loss=0.09313, pruned_loss=0.01085, audio_tagging_loss=0.00794, over 15340.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09107, pruned_loss=0.01367, audio_tagging_loss=0.009602, over 3009857.06 frames. ], batch size: 59, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:09:09,272 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361600
2023-11-23 14:09:32,506 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 900, loss[loss=0.07987, simple_loss=0.106, pruned_loss=0.01736, audio_tagging_loss=0.009504, over 15272.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09146, pruned_loss=0.01383, audio_tagging_loss=0.009613, over 3021463.40 frames. ], batch size: 58, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:09:33,402 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.32 vs. limit=12.0
2023-11-23 14:09:47,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2410833.3333333335, ans=0.1
2023-11-23 14:09:48,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2410833.3333333335, ans=0.125
2023-11-23 14:10:15,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361650
2023-11-23 14:10:18,171 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.422e+01 9.031e+01 9.667e+01 1.145e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 14:10:20,178 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.30 vs. limit=22.5
2023-11-23 14:10:23,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2411033.3333333335, ans=0.125
2023-11-23 14:10:29,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2411033.3333333335, ans=0.0
2023-11-23 14:10:37,452 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 950, loss[loss=0.05768, simple_loss=0.08504, pruned_loss=0.007541, audio_tagging_loss=0.007615, over 15258.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.092, pruned_loss=0.0139, audio_tagging_loss=0.009489, over 3028209.19 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:10:40,985 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.12 vs. limit=5.0
2023-11-23 14:10:59,938 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:11:02,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2411233.3333333335, ans=10.0
2023-11-23 14:11:03,679 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:11:19,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361700
2023-11-23 14:11:21,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2411300.0, ans=0.0
2023-11-23 14:11:34,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2411366.6666666665, ans=0.125
2023-11-23 14:11:41,907 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1000, loss[loss=0.08448, simple_loss=0.1181, pruned_loss=0.02001, audio_tagging_loss=0.005426, over 15235.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09175, pruned_loss=0.01397, audio_tagging_loss=0.009351, over 3026951.85 frames. ], batch size: 54, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:12:08,899 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:12:23,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2411633.3333333335, ans=0.0
2023-11-23 14:12:24,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361750
2023-11-23 14:12:25,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2411633.3333333335, ans=0.125
2023-11-23 14:12:26,480 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.336e+01 9.096e+01 9.576e+01 1.309e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-23 14:12:38,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2411700.0, ans=0.07
2023-11-23 14:12:46,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2411766.6666666665, ans=0.125
2023-11-23 14:12:47,011 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1050, loss[loss=0.07286, simple_loss=0.09117, pruned_loss=0.01776, audio_tagging_loss=0.009512, over 14193.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09157, pruned_loss=0.01378, audio_tagging_loss=0.009235, over 3028077.19 frames. ], batch size: 55, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:12:49,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2411766.6666666665, ans=0.1
2023-11-23 14:12:53,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2411766.6666666665, ans=0.125
2023-11-23 14:13:00,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2411833.3333333335, ans=0.0
2023-11-23 14:13:06,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2411833.3333333335, ans=0.125
2023-11-23 14:13:30,047 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361800
2023-11-23 14:13:30,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2411966.6666666665, ans=0.125
2023-11-23 14:13:42,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.out_whiten.whitening_limit, batch_count=2412033.3333333335, ans=8.0
2023-11-23 14:13:44,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2412033.3333333335, ans=0.0
2023-11-23 14:13:51,454 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1100, loss[loss=0.06097, simple_loss=0.08338, pruned_loss=0.01136, audio_tagging_loss=0.007925, over 15775.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09092, pruned_loss=0.01366, audio_tagging_loss=0.009261, over 3028344.76 frames. ], batch size: 62, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:13:53,971 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:14:15,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2412166.6666666665, ans=0.05
2023-11-23 14:14:27,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2412233.3333333335, ans=0.125
2023-11-23 14:14:30,592 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.02 vs. limit=15.0
2023-11-23 14:14:32,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2412300.0, ans=0.04949747468305833
2023-11-23 14:14:34,859 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361850
2023-11-23 14:14:37,254 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.000e+01 8.240e+01 8.744e+01 9.286e+01 1.135e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-23 14:14:56,291 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1150, loss[loss=0.07951, simple_loss=0.1062, pruned_loss=0.01507, audio_tagging_loss=0.01132, over 15137.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09179, pruned_loss=0.01376, audio_tagging_loss=0.009174, over 3028552.97 frames. ], batch size: 54, lr: 2.21e-03, grad_scale: 16.0
2023-11-23 14:15:18,905 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.37 vs. limit=15.0
2023-11-23 14:15:39,822 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361900
2023-11-23 14:16:02,440 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1200, loss[loss=0.06361, simple_loss=0.08439, pruned_loss=0.01388, audio_tagging_loss=0.007531, over 15050.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09185, pruned_loss=0.01377, audio_tagging_loss=0.009084, over 3026847.21 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:16:07,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2412766.6666666665, ans=0.0
2023-11-23 14:16:30,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2412900.0, ans=0.125
2023-11-23 14:16:35,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2412900.0, ans=0.2
2023-11-23 14:16:44,790 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 361950
2023-11-23 14:16:47,860 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.15 vs. limit=12.0
2023-11-23 14:16:48,313 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.465e+01 9.389e+01 1.006e+02 1.334e+02, threshold=1.878e+02, percent-clipped=0.0
2023-11-23 14:17:06,320 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1250, loss[loss=0.07805, simple_loss=0.1096, pruned_loss=0.01612, audio_tagging_loss=0.007155, over 14416.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09135, pruned_loss=0.01384, audio_tagging_loss=0.009016, over 3032468.65 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:17:07,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2413100.0, ans=0.125
2023-11-23 14:17:32,445 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0
2023-11-23 14:17:48,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2413300.0, ans=0.0
2023-11-23 14:17:50,039 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362000
2023-11-23 14:17:56,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2413300.0, ans=0.05
2023-11-23 14:18:11,307 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1300, loss[loss=0.06113, simple_loss=0.07289, pruned_loss=0.01262, audio_tagging_loss=0.01207, over 15098.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09148, pruned_loss=0.01394, audio_tagging_loss=0.009048, over 3033694.71 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:18:13,844 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.35 vs. limit=6.0
2023-11-23 14:18:22,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2413433.3333333335, ans=0.125
2023-11-23 14:18:25,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2413500.0, ans=0.1
2023-11-23 14:18:28,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2413500.0, ans=0.0
2023-11-23 14:18:48,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2413566.6666666665, ans=0.125
2023-11-23 14:18:49,324 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.01 vs. limit=12.0
2023-11-23 14:18:54,965 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362050
2023-11-23 14:18:58,452 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.540e+01 8.227e+01 8.895e+01 9.502e+01 1.940e+02, threshold=1.779e+02, percent-clipped=1.0
2023-11-23 14:19:17,507 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1350, loss[loss=0.06568, simple_loss=0.08624, pruned_loss=0.012, audio_tagging_loss=0.01056, over 15802.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09062, pruned_loss=0.01369, audio_tagging_loss=0.009138, over 3046393.63 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:19:26,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2413766.6666666665, ans=0.035
2023-11-23 14:19:28,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2413766.6666666665, ans=0.2
2023-11-23 14:19:54,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2413900.0, ans=0.0
2023-11-23 14:20:01,086 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362100
2023-11-23 14:20:04,688 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:20:22,791 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1400, loss[loss=0.06548, simple_loss=0.08172, pruned_loss=0.0135, audio_tagging_loss=0.01111, over 15633.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09043, pruned_loss=0.01355, audio_tagging_loss=0.009209, over 3047124.92 frames. ], batch size: 63, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:20:52,289 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.26 vs. limit=12.0
2023-11-23 14:20:58,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2414233.3333333335, ans=0.0
2023-11-23 14:21:01,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2414300.0, ans=0.0
2023-11-23 14:21:05,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2414300.0, ans=0.0
2023-11-23 14:21:06,220 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362150
2023-11-23 14:21:07,568 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:21:08,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2414300.0, ans=0.2
2023-11-23 14:21:09,851 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.930e+01 8.379e+01 8.906e+01 9.568e+01 1.474e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 14:21:25,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2414366.6666666665, ans=0.2
2023-11-23 14:21:25,794 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.42 vs. limit=22.5
2023-11-23 14:21:26,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2414433.3333333335, ans=0.125
2023-11-23 14:21:27,361 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1450, loss[loss=0.07183, simple_loss=0.0916, pruned_loss=0.01649, audio_tagging_loss=0.009545, over 14904.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09121, pruned_loss=0.01375, audio_tagging_loss=0.009162, over 3038120.12 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:21:59,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2414566.6666666665, ans=0.0
2023-11-23 14:22:10,939 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362200
2023-11-23 14:22:14,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2414633.3333333335, ans=0.07
2023-11-23 14:22:22,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2414700.0, ans=0.125
2023-11-23 14:22:23,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.06 vs. limit=15.0
2023-11-23 14:22:25,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2414700.0, ans=0.0
2023-11-23 14:22:34,351 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1500, loss[loss=0.07983, simple_loss=0.1081, pruned_loss=0.01709, audio_tagging_loss=0.008679, over 15373.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09215, pruned_loss=0.01395, audio_tagging_loss=0.009167, over 3043089.09 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:23:01,633 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.11 vs. limit=10.0
2023-11-23 14:23:16,527 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362250
2023-11-23 14:23:20,757 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.572e+01 9.255e+01 1.007e+02 1.352e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-23 14:23:29,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2415033.3333333335, ans=0.1
2023-11-23 14:23:29,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2415033.3333333335, ans=0.09899494936611666
2023-11-23 14:23:31,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2415033.3333333335, ans=0.0
2023-11-23 14:23:35,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2415033.3333333335, ans=0.125
2023-11-23 14:23:38,723 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1550, loss[loss=0.05946, simple_loss=0.0711, pruned_loss=0.01081, audio_tagging_loss=0.0131, over 15123.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09245, pruned_loss=0.01408, audio_tagging_loss=0.009273, over 3050164.21 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:24:12,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.26 vs. limit=15.0
2023-11-23 14:24:13,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2415233.3333333335, ans=0.125
2023-11-23 14:24:22,274 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362300
2023-11-23 14:24:32,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.69 vs. limit=22.5
2023-11-23 14:24:32,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.79 vs. limit=15.0
2023-11-23 14:24:33,864 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.91 vs. limit=15.0
2023-11-23 14:24:42,970 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1600, loss[loss=0.07775, simple_loss=0.1155, pruned_loss=0.01499, audio_tagging_loss=0.004985, over 14641.00 frames. ], tot_loss[loss=0.06942, simple_loss=0.09223, pruned_loss=0.01396, audio_tagging_loss=0.009337, over 3053977.63 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:24:48,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2415433.3333333335, ans=0.125
2023-11-23 14:24:52,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2415433.3333333335, ans=0.125
2023-11-23 14:24:54,401 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.55 vs. limit=15.0
2023-11-23 14:25:02,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2415500.0, ans=0.125
2023-11-23 14:25:06,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2415500.0, ans=0.0
2023-11-23 14:25:26,586 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362350
2023-11-23 14:25:30,057 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.117e+01 8.201e+01 8.953e+01 9.579e+01 1.488e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 14:25:48,507 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1650, loss[loss=0.05169, simple_loss=0.06927, pruned_loss=0.006652, audio_tagging_loss=0.0104, over 14740.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09211, pruned_loss=0.0138, audio_tagging_loss=0.009263, over 3048362.86 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:25:50,378 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.61 vs. limit=15.0
2023-11-23 14:25:57,852 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.53 vs. limit=15.0
2023-11-23 14:26:27,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2415966.6666666665, ans=0.2
2023-11-23 14:26:31,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362400
2023-11-23 14:26:34,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2415966.6666666665, ans=0.2
2023-11-23 14:26:37,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2415966.6666666665, ans=0.04949747468305833
2023-11-23 14:26:53,711 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1700, loss[loss=0.06356, simple_loss=0.08152, pruned_loss=0.01355, audio_tagging_loss=0.009252, over 14783.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09254, pruned_loss=0.01384, audio_tagging_loss=0.009306, over 3051778.63 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:27:11,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2416166.6666666665, ans=0.125
2023-11-23 14:27:11,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2416166.6666666665, ans=0.025
2023-11-23 14:27:20,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2416233.3333333335, ans=0.1
2023-11-23 14:27:24,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2416233.3333333335, ans=0.2
2023-11-23 14:27:29,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2416233.3333333335, ans=0.125
2023-11-23 14:27:31,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2416300.0, ans=0.2
2023-11-23 14:27:36,503 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362450
2023-11-23 14:27:38,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2416300.0, ans=0.1
2023-11-23 14:27:40,067 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.569e+01 8.211e+01 9.052e+01 9.581e+01 1.125e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 14:27:40,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2416300.0, ans=0.0
2023-11-23 14:27:49,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2416366.6666666665, ans=0.125
2023-11-23 14:27:50,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.18 vs. limit=15.0
2023-11-23 14:27:57,276 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1750, loss[loss=0.05806, simple_loss=0.07969, pruned_loss=0.01199, audio_tagging_loss=0.006222, over 14978.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09283, pruned_loss=0.01403, audio_tagging_loss=0.00917, over 3052946.38 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:28:18,642 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.50 vs. limit=15.0
2023-11-23 14:28:22,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2416566.6666666665, ans=0.0
2023-11-23 14:28:35,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2416633.3333333335, ans=0.125
2023-11-23 14:28:35,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2416633.3333333335, ans=0.05
2023-11-23 14:28:40,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362500
2023-11-23 14:28:55,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2416700.0, ans=0.1
2023-11-23 14:29:02,037 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1800, loss[loss=0.06579, simple_loss=0.08801, pruned_loss=0.01169, audio_tagging_loss=0.01009, over 15441.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09163, pruned_loss=0.01374, audio_tagging_loss=0.009076, over 3051375.14 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:29:19,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2416833.3333333335, ans=0.1
2023-11-23 14:29:21,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2416833.3333333335, ans=0.125
2023-11-23 14:29:44,840 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362550
2023-11-23 14:29:48,998 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.623e+01 8.278e+01 8.801e+01 9.362e+01 1.432e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-23 14:29:50,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2416966.6666666665, ans=0.0
2023-11-23 14:29:56,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2417033.3333333335, ans=0.125
2023-11-23 14:29:58,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2417033.3333333335, ans=0.125
2023-11-23 14:30:02,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2417033.3333333335, ans=0.2
2023-11-23 14:30:07,996 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1850, loss[loss=0.0845, simple_loss=0.1172, pruned_loss=0.01562, audio_tagging_loss=0.0103, over 14647.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09201, pruned_loss=0.01398, audio_tagging_loss=0.009034, over 3044680.28 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:30:17,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.18 vs. limit=22.5
2023-11-23 14:30:38,853 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.39 vs. limit=15.0
2023-11-23 14:30:50,699 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362600
2023-11-23 14:30:55,993 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.61 vs. limit=15.0
2023-11-23 14:31:02,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2417366.6666666665, ans=0.125
2023-11-23 14:31:12,191 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1900, loss[loss=0.05324, simple_loss=0.06921, pruned_loss=0.008839, audio_tagging_loss=0.009803, over 15789.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09176, pruned_loss=0.01396, audio_tagging_loss=0.009066, over 3050297.58 frames. ], batch size: 61, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:31:14,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2417433.3333333335, ans=0.07
2023-11-23 14:31:16,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2417433.3333333335, ans=0.09899494936611666
2023-11-23 14:31:19,935 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.25 vs. limit=15.0
2023-11-23 14:31:20,190 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.42 vs. limit=15.0
2023-11-23 14:31:27,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2417500.0, ans=0.1
2023-11-23 14:31:28,826 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:31:54,858 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362650
2023-11-23 14:31:58,355 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.275e+01 8.858e+01 9.597e+01 1.368e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 14:32:16,146 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 1950, loss[loss=0.0631, simple_loss=0.08338, pruned_loss=0.009986, audio_tagging_loss=0.01143, over 15323.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09117, pruned_loss=0.01389, audio_tagging_loss=0.009095, over 3053115.35 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:32:16,825 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.73 vs. limit=15.0
2023-11-23 14:32:32,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2417833.3333333335, ans=0.0
2023-11-23 14:32:50,600 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.13 vs. limit=15.0
2023-11-23 14:32:58,336 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362700
2023-11-23 14:33:13,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2418033.3333333335, ans=0.0
2023-11-23 14:33:20,871 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2000, loss[loss=0.05282, simple_loss=0.06918, pruned_loss=0.00728, audio_tagging_loss=0.01095, over 14957.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09086, pruned_loss=0.01388, audio_tagging_loss=0.009165, over 3047975.39 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:33:23,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2418100.0, ans=0.1
2023-11-23 14:33:41,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2418166.6666666665, ans=0.125
2023-11-23 14:33:53,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2418233.3333333335, ans=0.125
2023-11-23 14:34:01,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2418300.0, ans=0.2
2023-11-23 14:34:03,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362750
2023-11-23 14:34:06,084 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.72 vs. limit=15.0
2023-11-23 14:34:10,256 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.308e+01 8.872e+01 9.432e+01 1.250e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 14:34:10,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2418300.0, ans=0.125
2023-11-23 14:34:13,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2418366.6666666665, ans=0.125
2023-11-23 14:34:17,997 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.83 vs. limit=15.0
2023-11-23 14:34:26,003 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2050, loss[loss=0.04685, simple_loss=0.06522, pruned_loss=0.007882, audio_tagging_loss=0.006358, over 13927.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.0913, pruned_loss=0.01404, audio_tagging_loss=0.00904, over 3047887.75 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:34:35,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2418433.3333333335, ans=0.125
2023-11-23 14:34:47,568 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=14.12 vs. limit=15.0
2023-11-23 14:34:55,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2418566.6666666665, ans=0.0
2023-11-23 14:34:55,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2418566.6666666665, ans=0.125
2023-11-23 14:34:56,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2418566.6666666665, ans=0.125
2023-11-23 14:35:05,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2418633.3333333335, ans=0.125
2023-11-23 14:35:09,534 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362800
2023-11-23 14:35:27,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2418700.0, ans=0.125
2023-11-23 14:35:28,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2418700.0, ans=0.125
2023-11-23 14:35:31,083 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2100, loss[loss=0.05675, simple_loss=0.07019, pruned_loss=0.009647, audio_tagging_loss=0.01201, over 14537.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09112, pruned_loss=0.01396, audio_tagging_loss=0.008989, over 3042279.47 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:36:08,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2418900.0, ans=0.125
2023-11-23 14:36:11,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2418966.6666666665, ans=0.125
2023-11-23 14:36:14,761 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362850
2023-11-23 14:36:20,844 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.929e+01 8.402e+01 8.985e+01 9.676e+01 1.115e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 14:36:26,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2419033.3333333335, ans=0.1
2023-11-23 14:36:37,685 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2150, loss[loss=0.06137, simple_loss=0.08336, pruned_loss=0.0112, audio_tagging_loss=0.008496, over 15373.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09091, pruned_loss=0.01374, audio_tagging_loss=0.009003, over 3049419.04 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:36:48,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.whiten.whitening_limit, batch_count=2419100.0, ans=12.0
2023-11-23 14:36:50,132 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:36:54,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2419166.6666666665, ans=0.125
2023-11-23 14:36:55,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2419166.6666666665, ans=0.125
2023-11-23 14:37:08,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2419233.3333333335, ans=0.0
2023-11-23 14:37:14,955 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:37:20,585 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362900
2023-11-23 14:37:42,023 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2200, loss[loss=0.0619, simple_loss=0.07912, pruned_loss=0.009458, audio_tagging_loss=0.01288, over 15024.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09146, pruned_loss=0.01366, audio_tagging_loss=0.009083, over 3045676.03 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:37:43,188 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.46 vs. limit=15.0
2023-11-23 14:37:56,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2419500.0, ans=0.125
2023-11-23 14:37:56,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2419500.0, ans=0.2
2023-11-23 14:38:15,148 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.24 vs. limit=22.5
2023-11-23 14:38:18,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2419566.6666666665, ans=0.1
2023-11-23 14:38:22,281 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.36 vs. limit=12.0
2023-11-23 14:38:25,452 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 362950
2023-11-23 14:38:31,613 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.445e+01 9.122e+01 9.777e+01 1.344e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-23 14:38:41,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2419700.0, ans=0.2
2023-11-23 14:38:47,340 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2250, loss[loss=0.06777, simple_loss=0.0933, pruned_loss=0.01375, audio_tagging_loss=0.007366, over 14499.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09219, pruned_loss=0.01393, audio_tagging_loss=0.009084, over 3038870.45 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:39:02,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2419833.3333333335, ans=0.0
2023-11-23 14:39:02,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2419833.3333333335, ans=0.2
2023-11-23 14:39:10,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2419833.3333333335, ans=0.015
2023-11-23 14:39:19,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2419900.0, ans=0.2
2023-11-23 14:39:23,549 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.71 vs. limit=12.0
2023-11-23 14:39:30,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2419966.6666666665, ans=0.0
2023-11-23 14:39:31,106 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363000
2023-11-23 14:39:50,895 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.55 vs. limit=10.0
2023-11-23 14:39:54,461 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2300, loss[loss=0.0763, simple_loss=0.1022, pruned_loss=0.0182, audio_tagging_loss=0.007017, over 15576.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09211, pruned_loss=0.0139, audio_tagging_loss=0.009125, over 3047315.68 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:39:58,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2420100.0, ans=0.2
2023-11-23 14:40:14,669 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:40:36,402 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363050
2023-11-23 14:40:43,594 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.427e+01 9.145e+01 9.809e+01 1.490e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-23 14:40:49,890 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:40:51,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2420366.6666666665, ans=0.125
2023-11-23 14:40:58,423 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2350, loss[loss=0.1058, simple_loss=0.1442, pruned_loss=0.025, audio_tagging_loss=0.008672, over 14951.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09182, pruned_loss=0.01381, audio_tagging_loss=0.009172, over 3041733.92 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:41:04,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2420433.3333333335, ans=0.1
2023-11-23 14:41:09,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2420433.3333333335, ans=0.0
2023-11-23 14:41:25,060 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.27 vs. limit=15.0
2023-11-23 14:41:31,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2420566.6666666665, ans=10.0
2023-11-23 14:41:40,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2420633.3333333335, ans=0.125
2023-11-23 14:41:41,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363100
2023-11-23 14:41:46,033 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.01 vs. limit=6.0
2023-11-23 14:42:02,802 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2400, loss[loss=0.06407, simple_loss=0.08318, pruned_loss=0.01294, audio_tagging_loss=0.009534, over 14678.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09208, pruned_loss=0.01386, audio_tagging_loss=0.009327, over 3046144.69 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:42:06,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2420766.6666666665, ans=0.2
2023-11-23 14:42:09,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2420766.6666666665, ans=0.1
2023-11-23 14:42:37,579 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.24 vs. limit=15.0
2023-11-23 14:42:45,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363150
2023-11-23 14:42:46,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2420966.6666666665, ans=10.0
2023-11-23 14:42:51,923 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.436e+01 9.019e+01 9.718e+01 1.173e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 14:43:07,834 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2450, loss[loss=0.1161, simple_loss=0.1614, pruned_loss=0.02929, audio_tagging_loss=0.006071, over 16428.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09243, pruned_loss=0.01391, audio_tagging_loss=0.009261, over 3048304.91 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:43:25,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2421166.6666666665, ans=0.0
2023-11-23 14:43:45,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2421300.0, ans=0.1
2023-11-23 14:43:50,383 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363200
2023-11-23 14:44:12,626 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2500, loss[loss=0.05219, simple_loss=0.07109, pruned_loss=0.009769, audio_tagging_loss=0.00687, over 15027.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.0924, pruned_loss=0.0139, audio_tagging_loss=0.009282, over 3042889.04 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:44:18,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2421433.3333333335, ans=0.0
2023-11-23 14:44:22,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2421433.3333333335, ans=0.125
2023-11-23 14:44:28,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2421500.0, ans=0.2
2023-11-23 14:44:41,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2421566.6666666665, ans=0.0
2023-11-23 14:44:52,929 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.62 vs. limit=22.5
2023-11-23 14:44:56,111 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363250
2023-11-23 14:45:01,993 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.211e+01 9.083e+01 9.843e+01 1.373e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 14:45:07,754 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.40 vs. limit=15.0
2023-11-23 14:45:08,825 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.54 vs. limit=12.0
2023-11-23 14:45:17,038 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2550, loss[loss=0.07869, simple_loss=0.1176, pruned_loss=0.01239, audio_tagging_loss=0.00748, over 15654.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.09168, pruned_loss=0.01379, audio_tagging_loss=0.009284, over 3047669.96 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:45:23,975 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:45:55,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2421966.6666666665, ans=0.2
2023-11-23 14:46:00,367 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363300
2023-11-23 14:46:06,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2421966.6666666665, ans=0.125
2023-11-23 14:46:22,982 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2600, loss[loss=0.06201, simple_loss=0.08443, pruned_loss=0.01176, audio_tagging_loss=0.008035, over 15503.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09095, pruned_loss=0.0135, audio_tagging_loss=0.009089, over 3042948.83 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:46:27,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2422100.0, ans=0.05
2023-11-23 14:46:32,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2422100.0, ans=0.125
2023-11-23 14:46:38,526 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.08 vs. limit=15.0
2023-11-23 14:47:00,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2422300.0, ans=0.2
2023-11-23 14:47:05,766 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363350
2023-11-23 14:47:12,903 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.854e+01 8.370e+01 9.064e+01 9.837e+01 1.240e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 14:47:28,334 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2650, loss[loss=0.06406, simple_loss=0.07907, pruned_loss=0.01527, audio_tagging_loss=0.009255, over 14374.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09086, pruned_loss=0.01334, audio_tagging_loss=0.009076, over 3037907.04 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:47:33,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.97 vs. limit=8.0
2023-11-23 14:47:37,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2422433.3333333335, ans=0.125
2023-11-23 14:47:38,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2422433.3333333335, ans=0.125
2023-11-23 14:47:45,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2422500.0, ans=0.125
2023-11-23 14:48:07,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2422633.3333333335, ans=0.125
2023-11-23 14:48:11,170 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363400
2023-11-23 14:48:32,429 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2700, loss[loss=0.05185, simple_loss=0.06275, pruned_loss=0.008247, audio_tagging_loss=0.01222, over 15273.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09052, pruned_loss=0.01319, audio_tagging_loss=0.008988, over 3041716.40 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:48:47,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2422833.3333333335, ans=0.0
2023-11-23 14:48:48,996 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.32 vs. limit=22.5
2023-11-23 14:49:15,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363450
2023-11-23 14:49:23,352 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.936e+01 8.399e+01 8.847e+01 9.605e+01 1.237e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 14:49:27,531 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:49:30,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.56 vs. limit=15.0
2023-11-23 14:49:37,795 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2750, loss[loss=0.06169, simple_loss=0.0793, pruned_loss=0.01393, audio_tagging_loss=0.008109, over 15152.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09111, pruned_loss=0.01337, audio_tagging_loss=0.008892, over 3039142.53 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:50:02,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2423166.6666666665, ans=0.05
2023-11-23 14:50:02,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2423166.6666666665, ans=0.125
2023-11-23 14:50:06,385 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.15 vs. limit=15.0
2023-11-23 14:50:18,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2423300.0, ans=0.125
2023-11-23 14:50:20,837 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363500
2023-11-23 14:50:33,610 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:50:38,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2423366.6666666665, ans=0.025
2023-11-23 14:50:43,977 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2800, loss[loss=0.06776, simple_loss=0.09267, pruned_loss=0.01256, audio_tagging_loss=0.008866, over 16031.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09059, pruned_loss=0.01331, audio_tagging_loss=0.008969, over 3041192.85 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 14:50:59,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2423500.0, ans=10.0
2023-11-23 14:51:05,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2423500.0, ans=0.0
2023-11-23 14:51:27,449 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363550
2023-11-23 14:51:36,069 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.298e+01 8.140e+01 8.811e+01 9.422e+01 1.418e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-23 14:51:36,859 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.31 vs. limit=15.0
2023-11-23 14:51:46,646 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0
2023-11-23 14:51:48,474 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2850, loss[loss=0.07625, simple_loss=0.1044, pruned_loss=0.01456, audio_tagging_loss=0.00948, over 14812.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09104, pruned_loss=0.0134, audio_tagging_loss=0.008909, over 3040506.32 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:52:12,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2423833.3333333335, ans=0.125
2023-11-23 14:52:13,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2423900.0, ans=0.1
2023-11-23 14:52:22,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2423900.0, ans=0.1
2023-11-23 14:52:23,362 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.65 vs. limit=15.0
2023-11-23 14:52:31,521 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363600
2023-11-23 14:52:34,855 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.86 vs. limit=12.0
2023-11-23 14:52:44,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2424033.3333333335, ans=0.0
2023-11-23 14:52:49,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2424033.3333333335, ans=0.125
2023-11-23 14:52:52,482 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2900, loss[loss=0.07088, simple_loss=0.09026, pruned_loss=0.01419, audio_tagging_loss=0.01155, over 15271.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.0918, pruned_loss=0.01349, audio_tagging_loss=0.008871, over 3047186.84 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:53:19,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2424233.3333333335, ans=0.125
2023-11-23 14:53:21,046 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:53:28,844 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.16 vs. limit=15.0
2023-11-23 14:53:35,945 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363650
2023-11-23 14:53:45,000 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.203e+01 8.514e+01 9.320e+01 9.888e+01 1.457e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-23 14:53:59,192 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 2950, loss[loss=0.08148, simple_loss=0.09285, pruned_loss=0.02673, audio_tagging_loss=0.00832, over 13944.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09283, pruned_loss=0.01381, audio_tagging_loss=0.008894, over 3047206.27 frames. ], batch size: 53, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:54:13,632 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.65 vs. limit=12.0
2023-11-23 14:54:26,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2424566.6666666665, ans=0.0
2023-11-23 14:54:39,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2424633.3333333335, ans=15.0
2023-11-23 14:54:41,430 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363700
2023-11-23 14:54:50,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2424700.0, ans=0.0
2023-11-23 14:54:54,180 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.18 vs. limit=15.0
2023-11-23 14:55:03,643 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3000, loss[loss=0.06862, simple_loss=0.09223, pruned_loss=0.01416, audio_tagging_loss=0.008344, over 16084.00 frames. ], tot_loss[loss=0.06987, simple_loss=0.09392, pruned_loss=0.01402, audio_tagging_loss=0.008887, over 3052718.44 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:55:03,646 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 14:55:27,734 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([1.8085, 2.5676, 2.5996, 2.3039, 2.6401, 2.4848, 2.5201, 2.5717],
       device='cuda:0')
2023-11-23 14:55:45,746 INFO [train_asr.py:1253] (0/4) Epoch 31, validation: loss=0.0577, simple_loss=0.05103, pruned_loss=0.005016, audio_tagging_loss=0.02717, over 4681554.00 frames. 
2023-11-23 14:55:45,746 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 14:55:48,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2424766.6666666665, ans=0.0
2023-11-23 14:56:11,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2424900.0, ans=0.04949747468305833
2023-11-23 14:56:21,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2424900.0, ans=0.2
2023-11-23 14:56:28,247 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363750
2023-11-23 14:56:35,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2424966.6666666665, ans=0.0
2023-11-23 14:56:35,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2424966.6666666665, ans=0.0
2023-11-23 14:56:37,294 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.570e+01 9.368e+01 9.997e+01 1.561e+02, threshold=1.874e+02, percent-clipped=0.0
2023-11-23 14:56:50,884 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3050, loss[loss=0.06156, simple_loss=0.08224, pruned_loss=0.01261, audio_tagging_loss=0.007836, over 16362.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09357, pruned_loss=0.014, audio_tagging_loss=0.008891, over 3053572.21 frames. ], batch size: 61, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:56:55,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2425100.0, ans=0.0
2023-11-23 14:57:16,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2425233.3333333335, ans=0.125
2023-11-23 14:57:20,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2425233.3333333335, ans=0.125
2023-11-23 14:57:20,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2425233.3333333335, ans=0.125
2023-11-23 14:57:27,980 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 14:57:30,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2425300.0, ans=0.125
2023-11-23 14:57:30,697 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.46 vs. limit=22.5
2023-11-23 14:57:34,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363800
2023-11-23 14:57:34,361 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 14:57:41,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2425300.0, ans=0.0
2023-11-23 14:57:56,141 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3100, loss[loss=0.09962, simple_loss=0.1329, pruned_loss=0.02485, audio_tagging_loss=0.00833, over 16065.00 frames. ], tot_loss[loss=0.07, simple_loss=0.09419, pruned_loss=0.01403, audio_tagging_loss=0.00888, over 3052362.17 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:58:31,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2425566.6666666665, ans=0.125
2023-11-23 14:58:39,603 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363850
2023-11-23 14:58:47,918 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.127e+01 8.546e+01 8.908e+01 9.738e+01 1.357e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 14:58:54,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2425700.0, ans=15.0
2023-11-23 14:58:58,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2425700.0, ans=0.125
2023-11-23 14:59:00,973 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3150, loss[loss=0.06756, simple_loss=0.09126, pruned_loss=0.01253, audio_tagging_loss=0.0094, over 14791.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09261, pruned_loss=0.01369, audio_tagging_loss=0.009077, over 3046847.28 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 14:59:22,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2425833.3333333335, ans=0.1
2023-11-23 14:59:31,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2425900.0, ans=0.0
2023-11-23 14:59:43,595 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363900
2023-11-23 14:59:56,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2426033.3333333335, ans=0.2
2023-11-23 15:00:02,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2426033.3333333335, ans=0.2
2023-11-23 15:00:05,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2426100.0, ans=0.125
2023-11-23 15:00:06,091 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3200, loss[loss=0.06584, simple_loss=0.08102, pruned_loss=0.01368, audio_tagging_loss=0.01165, over 14266.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.0927, pruned_loss=0.01371, audio_tagging_loss=0.009251, over 3050653.00 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:00:13,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2426100.0, ans=0.125
2023-11-23 15:00:14,327 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.13 vs. limit=15.0
2023-11-23 15:00:22,231 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=13.71 vs. limit=15.0
2023-11-23 15:00:34,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2426233.3333333335, ans=0.125
2023-11-23 15:00:49,086 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 363950
2023-11-23 15:00:58,110 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.380e+01 9.015e+01 9.537e+01 1.251e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 15:01:02,447 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.06 vs. limit=15.0
2023-11-23 15:01:11,013 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3250, loss[loss=0.07497, simple_loss=0.1063, pruned_loss=0.01255, audio_tagging_loss=0.009254, over 15721.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09279, pruned_loss=0.01365, audio_tagging_loss=0.009156, over 3053180.21 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:01:32,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2426500.0, ans=0.1
2023-11-23 15:01:53,574 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364000
2023-11-23 15:01:55,015 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-364000.pt
2023-11-23 15:02:11,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2426700.0, ans=0.125
2023-11-23 15:02:18,579 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3300, loss[loss=0.07504, simple_loss=0.1076, pruned_loss=0.0146, audio_tagging_loss=0.006653, over 15811.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.0926, pruned_loss=0.01367, audio_tagging_loss=0.009267, over 3048948.56 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:02:18,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2426766.6666666665, ans=0.0
2023-11-23 15:02:43,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2426900.0, ans=0.125
2023-11-23 15:03:01,212 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364050
2023-11-23 15:03:06,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2426966.6666666665, ans=0.125
2023-11-23 15:03:10,311 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.350e+01 9.048e+01 9.771e+01 1.182e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 15:03:23,953 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3350, loss[loss=0.07022, simple_loss=0.09875, pruned_loss=0.01259, audio_tagging_loss=0.008263, over 14916.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09246, pruned_loss=0.01375, audio_tagging_loss=0.009268, over 3044790.64 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:03:29,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2427100.0, ans=0.1
2023-11-23 15:04:01,456 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:04:02,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2427300.0, ans=0.04949747468305833
2023-11-23 15:04:03,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2427300.0, ans=0.125
2023-11-23 15:04:04,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2427300.0, ans=0.5
2023-11-23 15:04:05,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2427300.0, ans=0.125
2023-11-23 15:04:06,596 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364100
2023-11-23 15:04:24,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2427366.6666666665, ans=0.125
2023-11-23 15:04:28,240 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3400, loss[loss=0.06012, simple_loss=0.07847, pruned_loss=0.01007, audio_tagging_loss=0.01081, over 14656.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09267, pruned_loss=0.01374, audio_tagging_loss=0.009097, over 3042453.66 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:04:35,720 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.69 vs. limit=15.0
2023-11-23 15:04:39,239 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.83 vs. limit=15.0
2023-11-23 15:04:46,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2427500.0, ans=0.125
2023-11-23 15:04:50,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2427500.0, ans=0.0
2023-11-23 15:05:03,528 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.76 vs. limit=22.5
2023-11-23 15:05:06,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2427633.3333333335, ans=0.125
2023-11-23 15:05:10,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2427633.3333333335, ans=0.1
2023-11-23 15:05:11,365 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364150
2023-11-23 15:05:13,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2427633.3333333335, ans=0.0
2023-11-23 15:05:21,248 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.657e+01 9.005e+01 9.760e+01 1.355e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 15:05:32,439 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.61 vs. limit=22.5
2023-11-23 15:05:32,854 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3450, loss[loss=0.0795, simple_loss=0.1076, pruned_loss=0.0163, audio_tagging_loss=0.009385, over 14420.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09303, pruned_loss=0.01383, audio_tagging_loss=0.009006, over 3043801.30 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:05:39,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2427766.6666666665, ans=0.125
2023-11-23 15:06:16,695 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364200
2023-11-23 15:06:21,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.27 vs. limit=15.0
2023-11-23 15:06:35,702 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:06:38,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2428100.0, ans=0.0
2023-11-23 15:06:39,842 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3500, loss[loss=0.04886, simple_loss=0.06798, pruned_loss=0.005795, audio_tagging_loss=0.009079, over 16200.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09254, pruned_loss=0.01382, audio_tagging_loss=0.009043, over 3045650.06 frames. ], batch size: 61, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:06:43,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2428100.0, ans=0.125
2023-11-23 15:06:44,049 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.13 vs. limit=15.0
2023-11-23 15:07:02,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2428166.6666666665, ans=0.1
2023-11-23 15:07:11,444 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:07:21,973 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364250
2023-11-23 15:07:30,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2428366.6666666665, ans=0.125
2023-11-23 15:07:32,825 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.333e+01 8.735e+01 9.330e+01 1.206e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-23 15:07:43,917 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3550, loss[loss=0.05437, simple_loss=0.077, pruned_loss=0.008657, audio_tagging_loss=0.007214, over 14435.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09159, pruned_loss=0.01379, audio_tagging_loss=0.00912, over 3045301.99 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:08:19,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2428566.6666666665, ans=0.1
2023-11-23 15:08:21,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2428566.6666666665, ans=0.2
2023-11-23 15:08:23,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2428633.3333333335, ans=0.125
2023-11-23 15:08:27,293 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364300
2023-11-23 15:08:35,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2428700.0, ans=0.1
2023-11-23 15:08:49,096 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3600, loss[loss=0.08452, simple_loss=0.1078, pruned_loss=0.02156, audio_tagging_loss=0.009052, over 15193.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09122, pruned_loss=0.01364, audio_tagging_loss=0.009134, over 3052679.01 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:09:04,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2428833.3333333335, ans=0.125
2023-11-23 15:09:04,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2428833.3333333335, ans=0.025
2023-11-23 15:09:26,004 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.02 vs. limit=15.0
2023-11-23 15:09:31,788 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364350
2023-11-23 15:09:33,376 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.57 vs. limit=15.0
2023-11-23 15:09:42,098 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.038e+01 8.312e+01 8.847e+01 9.562e+01 1.157e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-23 15:09:49,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2429033.3333333335, ans=0.125
2023-11-23 15:09:54,433 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3650, loss[loss=0.0783, simple_loss=0.09856, pruned_loss=0.01921, audio_tagging_loss=0.009805, over 16567.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.0912, pruned_loss=0.01373, audio_tagging_loss=0.009165, over 3048002.55 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:09:57,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2429100.0, ans=0.125
2023-11-23 15:10:09,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2429166.6666666665, ans=0.2
2023-11-23 15:10:36,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364400
2023-11-23 15:10:52,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2429366.6666666665, ans=0.125
2023-11-23 15:10:54,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2429366.6666666665, ans=0.125
2023-11-23 15:10:59,378 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3700, loss[loss=0.07953, simple_loss=0.09845, pruned_loss=0.02105, audio_tagging_loss=0.009249, over 16432.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09208, pruned_loss=0.0139, audio_tagging_loss=0.009024, over 3051906.19 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:11:00,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2429433.3333333335, ans=0.1
2023-11-23 15:11:02,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2429433.3333333335, ans=0.0
2023-11-23 15:11:02,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2429433.3333333335, ans=0.125
2023-11-23 15:11:03,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2429433.3333333335, ans=0.0
2023-11-23 15:11:09,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2429433.3333333335, ans=0.0
2023-11-23 15:11:22,078 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.89 vs. limit=15.0
2023-11-23 15:11:26,230 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.49 vs. limit=22.5
2023-11-23 15:11:33,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2429566.6666666665, ans=0.0
2023-11-23 15:11:35,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2429566.6666666665, ans=0.2
2023-11-23 15:11:42,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364450
2023-11-23 15:11:45,917 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.00 vs. limit=22.5
2023-11-23 15:11:46,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2429633.3333333335, ans=0.0
2023-11-23 15:11:49,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2429633.3333333335, ans=0.125
2023-11-23 15:11:51,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2429700.0, ans=0.1
2023-11-23 15:11:52,581 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.697e+01 8.537e+01 9.041e+01 9.821e+01 1.313e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-23 15:12:03,716 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3750, loss[loss=0.07112, simple_loss=0.1049, pruned_loss=0.01111, audio_tagging_loss=0.00755, over 16046.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09237, pruned_loss=0.01383, audio_tagging_loss=0.009033, over 3057218.63 frames. ], batch size: 60, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:12:24,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2429833.3333333335, ans=0.125
2023-11-23 15:12:43,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2429966.6666666665, ans=0.125
2023-11-23 15:12:46,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364500
2023-11-23 15:12:46,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2429966.6666666665, ans=0.125
2023-11-23 15:12:47,573 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:12:47,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2429966.6666666665, ans=0.025
2023-11-23 15:12:54,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2430033.3333333335, ans=0.2
2023-11-23 15:12:55,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2430033.3333333335, ans=0.1
2023-11-23 15:13:01,137 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.16 vs. limit=15.0
2023-11-23 15:13:08,470 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3800, loss[loss=0.07732, simple_loss=0.1049, pruned_loss=0.01775, audio_tagging_loss=0.00711, over 16486.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09294, pruned_loss=0.01391, audio_tagging_loss=0.009022, over 3058337.15 frames. ], batch size: 60, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:13:13,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2430100.0, ans=0.1
2023-11-23 15:13:17,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2430100.0, ans=0.125
2023-11-23 15:13:33,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2430233.3333333335, ans=0.1
2023-11-23 15:13:44,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2430233.3333333335, ans=0.125
2023-11-23 15:13:50,886 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364550
2023-11-23 15:13:57,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2430300.0, ans=0.125
2023-11-23 15:14:04,561 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.792e+01 8.445e+01 8.986e+01 9.684e+01 1.334e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 15:14:10,078 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.25 vs. limit=15.0
2023-11-23 15:14:14,494 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3850, loss[loss=0.06407, simple_loss=0.09165, pruned_loss=0.01019, audio_tagging_loss=0.008057, over 16073.00 frames. ], tot_loss[loss=0.07014, simple_loss=0.09394, pruned_loss=0.01404, audio_tagging_loss=0.009129, over 3056814.03 frames. ], batch size: 61, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:14:22,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2430433.3333333335, ans=0.0
2023-11-23 15:14:27,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2430500.0, ans=0.125
2023-11-23 15:14:33,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2430500.0, ans=0.1
2023-11-23 15:14:48,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2430566.6666666665, ans=0.125
2023-11-23 15:14:57,798 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364600
2023-11-23 15:14:58,466 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.76 vs. limit=15.0
2023-11-23 15:15:02,522 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.91 vs. limit=10.0
2023-11-23 15:15:05,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2430700.0, ans=0.5
2023-11-23 15:15:10,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2430700.0, ans=0.0
2023-11-23 15:15:13,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2430700.0, ans=0.0
2023-11-23 15:15:18,956 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3900, loss[loss=0.08212, simple_loss=0.1111, pruned_loss=0.01825, audio_tagging_loss=0.008294, over 14823.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.09358, pruned_loss=0.01417, audio_tagging_loss=0.009166, over 3057490.09 frames. ], batch size: 54, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:15:19,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2430766.6666666665, ans=0.125
2023-11-23 15:15:57,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2430966.6666666665, ans=0.125
2023-11-23 15:15:58,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2430966.6666666665, ans=0.125
2023-11-23 15:16:01,929 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.05 vs. limit=22.5
2023-11-23 15:16:02,397 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364650
2023-11-23 15:16:14,749 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.954e+01 8.382e+01 9.163e+01 9.741e+01 1.346e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 15:16:20,324 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.31 vs. limit=22.5
2023-11-23 15:16:23,931 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 3950, loss[loss=0.071, simple_loss=0.09787, pruned_loss=0.01265, audio_tagging_loss=0.009416, over 15202.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09246, pruned_loss=0.01403, audio_tagging_loss=0.009312, over 3055406.93 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 8.0
2023-11-23 15:16:43,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2431166.6666666665, ans=0.2
2023-11-23 15:16:44,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2431166.6666666665, ans=0.0
2023-11-23 15:17:06,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364700
2023-11-23 15:17:11,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2431300.0, ans=0.0
2023-11-23 15:17:11,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2431300.0, ans=0.125
2023-11-23 15:17:29,994 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4000, loss[loss=0.06347, simple_loss=0.08353, pruned_loss=0.01077, audio_tagging_loss=0.01094, over 14055.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09256, pruned_loss=0.01432, audio_tagging_loss=0.009418, over 3054804.52 frames. ], batch size: 55, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:17:43,155 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.02 vs. limit=15.0
2023-11-23 15:18:01,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2431566.6666666665, ans=0.125
2023-11-23 15:18:09,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2431633.3333333335, ans=0.125
2023-11-23 15:18:12,971 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364750
2023-11-23 15:18:13,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2431633.3333333335, ans=0.5
2023-11-23 15:18:22,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2431700.0, ans=0.0
2023-11-23 15:18:24,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2431700.0, ans=0.2
2023-11-23 15:18:25,122 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.571e+01 8.345e+01 8.888e+01 9.598e+01 2.808e+02, threshold=1.778e+02, percent-clipped=1.0
2023-11-23 15:18:28,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2431700.0, ans=0.125
2023-11-23 15:18:33,726 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4050, loss[loss=0.05345, simple_loss=0.07166, pruned_loss=0.007626, audio_tagging_loss=0.009997, over 15000.00 frames. ], tot_loss[loss=0.07022, simple_loss=0.09275, pruned_loss=0.01441, audio_tagging_loss=0.009437, over 3058237.34 frames. ], batch size: 58, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:18:36,196 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:18:47,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2431833.3333333335, ans=0.125
2023-11-23 15:18:51,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2431833.3333333335, ans=0.1
2023-11-23 15:18:56,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2431833.3333333335, ans=0.125
2023-11-23 15:19:16,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364800
2023-11-23 15:19:23,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.60 vs. limit=15.0
2023-11-23 15:19:37,445 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4100, loss[loss=0.05079, simple_loss=0.06488, pruned_loss=0.007578, audio_tagging_loss=0.01077, over 14491.00 frames. ], tot_loss[loss=0.07066, simple_loss=0.09352, pruned_loss=0.01448, audio_tagging_loss=0.009422, over 3055522.38 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:19:48,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2432100.0, ans=0.0
2023-11-23 15:20:03,152 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.95 vs. limit=15.0
2023-11-23 15:20:09,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2432233.3333333335, ans=0.0
2023-11-23 15:20:15,559 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.62 vs. limit=15.0
2023-11-23 15:20:19,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364850
2023-11-23 15:20:28,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2432366.6666666665, ans=0.0
2023-11-23 15:20:29,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2432366.6666666665, ans=0.0
2023-11-23 15:20:32,691 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.349e+01 8.595e+01 9.054e+01 9.904e+01 2.115e+02, threshold=1.811e+02, percent-clipped=1.0
2023-11-23 15:20:43,287 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4150, loss[loss=0.09109, simple_loss=0.1301, pruned_loss=0.01974, audio_tagging_loss=0.006287, over 15396.00 frames. ], tot_loss[loss=0.06992, simple_loss=0.09275, pruned_loss=0.01431, audio_tagging_loss=0.009238, over 3049742.34 frames. ], batch size: 56, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:20:43,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2432433.3333333335, ans=0.125
2023-11-23 15:20:44,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2432433.3333333335, ans=0.125
2023-11-23 15:20:58,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2432500.0, ans=0.1
2023-11-23 15:21:00,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2432500.0, ans=0.0
2023-11-23 15:21:04,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2432500.0, ans=0.2
2023-11-23 15:21:11,021 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=18.06 vs. limit=22.5
2023-11-23 15:21:16,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2432566.6666666665, ans=0.125
2023-11-23 15:21:23,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2432633.3333333335, ans=0.0
2023-11-23 15:21:25,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364900
2023-11-23 15:21:29,315 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:21:33,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2432700.0, ans=0.0
2023-11-23 15:21:36,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2432700.0, ans=0.125
2023-11-23 15:21:47,317 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4200, loss[loss=0.05228, simple_loss=0.06725, pruned_loss=0.009137, audio_tagging_loss=0.009515, over 15261.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09368, pruned_loss=0.01431, audio_tagging_loss=0.009029, over 3053927.20 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:22:22,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2432900.0, ans=0.1
2023-11-23 15:22:30,191 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 364950
2023-11-23 15:22:42,431 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.314e+01 9.166e+01 9.730e+01 1.323e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 15:22:51,147 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4250, loss[loss=0.07387, simple_loss=0.1064, pruned_loss=0.01351, audio_tagging_loss=0.007154, over 15861.00 frames. ], tot_loss[loss=0.07043, simple_loss=0.09461, pruned_loss=0.01422, audio_tagging_loss=0.008905, over 3059671.76 frames. ], batch size: 59, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:23:12,879 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.19 vs. limit=22.5
2023-11-23 15:23:12,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.88 vs. limit=15.0
2023-11-23 15:23:29,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2433300.0, ans=0.5
2023-11-23 15:23:33,583 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365000
2023-11-23 15:23:35,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2433300.0, ans=0.1
2023-11-23 15:23:49,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2433366.6666666665, ans=0.05
2023-11-23 15:23:56,043 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4300, loss[loss=0.07642, simple_loss=0.1118, pruned_loss=0.01256, audio_tagging_loss=0.007932, over 15661.00 frames. ], tot_loss[loss=0.07026, simple_loss=0.09447, pruned_loss=0.01424, audio_tagging_loss=0.008791, over 3057078.06 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:24:33,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2433633.3333333335, ans=0.07
2023-11-23 15:24:38,093 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365050
2023-11-23 15:24:49,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2433700.0, ans=0.0
2023-11-23 15:24:50,775 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.539e+01 9.202e+01 9.861e+01 1.335e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 15:25:00,110 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4350, loss[loss=0.07434, simple_loss=0.1081, pruned_loss=0.01169, audio_tagging_loss=0.008612, over 15891.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09482, pruned_loss=0.01424, audio_tagging_loss=0.008786, over 3056462.19 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 16.0
2023-11-23 15:25:06,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-23 15:25:14,138 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.26 vs. limit=15.0
2023-11-23 15:25:27,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2433900.0, ans=0.125
2023-11-23 15:25:42,407 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365100
2023-11-23 15:25:42,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2433966.6666666665, ans=0.1
2023-11-23 15:25:50,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2434033.3333333335, ans=0.2
2023-11-23 15:25:54,634 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.07 vs. limit=12.0
2023-11-23 15:26:03,791 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4400, loss[loss=0.08072, simple_loss=0.1124, pruned_loss=0.01722, audio_tagging_loss=0.00728, over 15930.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09466, pruned_loss=0.01429, audio_tagging_loss=0.008848, over 3048167.61 frames. ], batch size: 57, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:26:07,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2434100.0, ans=0.125
2023-11-23 15:26:09,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2434100.0, ans=0.05
2023-11-23 15:26:10,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2434100.0, ans=0.125
2023-11-23 15:26:22,414 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.72 vs. limit=22.5
2023-11-23 15:26:25,945 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.85 vs. limit=6.0
2023-11-23 15:26:40,252 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.58 vs. limit=15.0
2023-11-23 15:26:46,962 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365150
2023-11-23 15:26:59,934 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.792e+01 8.264e+01 9.016e+01 9.697e+01 1.171e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 15:27:09,420 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4450, loss[loss=0.06873, simple_loss=0.09858, pruned_loss=0.01132, audio_tagging_loss=0.008118, over 16628.00 frames. ], tot_loss[loss=0.07017, simple_loss=0.09442, pruned_loss=0.01414, audio_tagging_loss=0.008812, over 3050956.53 frames. ], batch size: 62, lr: 2.20e-03, grad_scale: 32.0
2023-11-23 15:27:14,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2434433.3333333335, ans=0.125
2023-11-23 15:27:19,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2434433.3333333335, ans=0.125
2023-11-23 15:27:20,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2434433.3333333335, ans=0.125
2023-11-23 15:27:21,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2434500.0, ans=0.125
2023-11-23 15:27:40,775 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.06 vs. limit=15.0
2023-11-23 15:27:43,129 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.56 vs. limit=22.5
2023-11-23 15:27:51,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365200
2023-11-23 15:27:52,274 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.04 vs. limit=10.0
2023-11-23 15:28:03,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2434700.0, ans=0.125
2023-11-23 15:28:07,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2434700.0, ans=0.0
2023-11-23 15:28:14,259 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4500, loss[loss=0.08789, simple_loss=0.1124, pruned_loss=0.02023, audio_tagging_loss=0.01143, over 15693.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09429, pruned_loss=0.01404, audio_tagging_loss=0.008855, over 3044892.05 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:28:33,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2434833.3333333335, ans=0.125
2023-11-23 15:28:39,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2434900.0, ans=0.125
2023-11-23 15:28:45,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2434900.0, ans=0.125
2023-11-23 15:28:50,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2434900.0, ans=0.125
2023-11-23 15:28:57,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365250
2023-11-23 15:29:11,952 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.432e+01 8.316e+01 8.938e+01 9.967e+01 1.364e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-23 15:29:19,397 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4550, loss[loss=0.05296, simple_loss=0.06553, pruned_loss=0.009876, audio_tagging_loss=0.01032, over 15407.00 frames. ], tot_loss[loss=0.06959, simple_loss=0.09332, pruned_loss=0.014, audio_tagging_loss=0.008936, over 3042847.16 frames. ], batch size: 61, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:29:20,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2435100.0, ans=0.0
2023-11-23 15:29:25,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2435100.0, ans=0.0
2023-11-23 15:29:56,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2435233.3333333335, ans=0.2
2023-11-23 15:30:03,017 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365300
2023-11-23 15:30:09,165 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:30:09,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2435300.0, ans=0.1
2023-11-23 15:30:14,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2435366.6666666665, ans=0.125
2023-11-23 15:30:21,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2435366.6666666665, ans=0.1
2023-11-23 15:30:24,799 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4600, loss[loss=0.07683, simple_loss=0.1049, pruned_loss=0.01637, audio_tagging_loss=0.008025, over 14705.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.0928, pruned_loss=0.01403, audio_tagging_loss=0.009047, over 3048880.20 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:30:37,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2435500.0, ans=0.125
2023-11-23 15:30:48,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2435500.0, ans=0.0
2023-11-23 15:30:48,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2435500.0, ans=0.2
2023-11-23 15:31:01,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2435566.6666666665, ans=0.125
2023-11-23 15:31:08,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365350
2023-11-23 15:31:21,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2435700.0, ans=0.125
2023-11-23 15:31:22,443 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.155e+01 8.583e+01 9.120e+01 9.726e+01 1.437e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-23 15:31:29,873 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4650, loss[loss=0.08508, simple_loss=0.1071, pruned_loss=0.02269, audio_tagging_loss=0.008825, over 15499.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09349, pruned_loss=0.01418, audio_tagging_loss=0.009059, over 3047233.57 frames. ], batch size: 61, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:31:39,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2435766.6666666665, ans=0.0
2023-11-23 15:31:47,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2435833.3333333335, ans=0.125
2023-11-23 15:31:52,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2435833.3333333335, ans=0.0
2023-11-23 15:32:03,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2435900.0, ans=0.2
2023-11-23 15:32:05,863 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.82 vs. limit=12.0
2023-11-23 15:32:09,738 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.11 vs. limit=6.0
2023-11-23 15:32:10,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2435966.6666666665, ans=0.125
2023-11-23 15:32:12,739 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365400
2023-11-23 15:32:20,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2436033.3333333335, ans=0.1
2023-11-23 15:32:33,933 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4700, loss[loss=0.07133, simple_loss=0.09757, pruned_loss=0.01481, audio_tagging_loss=0.00773, over 15224.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09338, pruned_loss=0.01413, audio_tagging_loss=0.009132, over 3049458.68 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:32:42,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2436100.0, ans=0.125
2023-11-23 15:32:58,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2436166.6666666665, ans=0.125
2023-11-23 15:33:05,704 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.64 vs. limit=5.0
2023-11-23 15:33:14,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2436300.0, ans=0.2
2023-11-23 15:33:17,202 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365450
2023-11-23 15:33:22,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2436300.0, ans=0.0
2023-11-23 15:33:24,022 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.09 vs. limit=22.5
2023-11-23 15:33:31,150 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.926e+01 8.245e+01 8.834e+01 9.507e+01 1.184e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 15:33:31,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2436366.6666666665, ans=0.125
2023-11-23 15:33:36,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2436366.6666666665, ans=0.125
2023-11-23 15:33:39,244 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4750, loss[loss=0.08196, simple_loss=0.1085, pruned_loss=0.02051, audio_tagging_loss=0.007191, over 14427.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09261, pruned_loss=0.01394, audio_tagging_loss=0.009205, over 3048367.87 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:34:22,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365500
2023-11-23 15:34:35,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2436700.0, ans=0.0
2023-11-23 15:34:40,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2436700.0, ans=0.0
2023-11-23 15:34:44,610 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4800, loss[loss=0.06454, simple_loss=0.08895, pruned_loss=0.01396, audio_tagging_loss=0.0061, over 15494.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09237, pruned_loss=0.01384, audio_tagging_loss=0.009195, over 3052755.85 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:34:58,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=2436833.3333333335, ans=10.0
2023-11-23 15:35:15,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2436900.0, ans=0.0
2023-11-23 15:35:15,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2436900.0, ans=0.125
2023-11-23 15:35:16,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2436900.0, ans=0.0
2023-11-23 15:35:27,560 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365550
2023-11-23 15:35:42,124 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.266e+01 8.333e+01 8.912e+01 9.620e+01 1.211e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 15:35:48,311 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4850, loss[loss=0.05968, simple_loss=0.07881, pruned_loss=0.009679, audio_tagging_loss=0.01059, over 15176.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09198, pruned_loss=0.0138, audio_tagging_loss=0.009377, over 3046970.77 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:35:50,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2437100.0, ans=0.125
2023-11-23 15:36:03,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2437166.6666666665, ans=0.07
2023-11-23 15:36:17,405 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.70 vs. limit=6.0
2023-11-23 15:36:31,583 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365600
2023-11-23 15:36:47,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2437366.6666666665, ans=0.09899494936611666
2023-11-23 15:36:53,502 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4900, loss[loss=0.04227, simple_loss=0.05205, pruned_loss=0.003406, audio_tagging_loss=0.01284, over 15011.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09213, pruned_loss=0.0139, audio_tagging_loss=0.009361, over 3046524.93 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:37:16,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2437500.0, ans=0.0
2023-11-23 15:37:24,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2437566.6666666665, ans=0.1
2023-11-23 15:37:35,582 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365650
2023-11-23 15:37:53,798 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.898e+01 8.240e+01 8.743e+01 9.594e+01 1.319e+02, threshold=1.749e+02, percent-clipped=0.0
2023-11-23 15:37:58,811 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 4950, loss[loss=0.05487, simple_loss=0.07084, pruned_loss=0.01096, audio_tagging_loss=0.008494, over 13493.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09292, pruned_loss=0.01401, audio_tagging_loss=0.009222, over 3044369.19 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:38:41,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365700
2023-11-23 15:38:43,785 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.38 vs. limit=15.0
2023-11-23 15:38:52,249 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.19 vs. limit=15.0
2023-11-23 15:38:52,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.56 vs. limit=12.0
2023-11-23 15:39:02,602 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5000, loss[loss=0.06377, simple_loss=0.08486, pruned_loss=0.01206, audio_tagging_loss=0.009282, over 14961.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09201, pruned_loss=0.0138, audio_tagging_loss=0.009085, over 3049564.05 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:39:05,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2438100.0, ans=0.125
2023-11-23 15:39:22,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2438166.6666666665, ans=0.125
2023-11-23 15:39:36,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.50 vs. limit=12.0
2023-11-23 15:39:45,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365750
2023-11-23 15:39:48,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2438300.0, ans=0.0
2023-11-23 15:40:01,797 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.229e+01 8.801e+01 9.337e+01 1.229e+02, threshold=1.760e+02, percent-clipped=0.0
2023-11-23 15:40:07,588 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5050, loss[loss=0.05936, simple_loss=0.07649, pruned_loss=0.01278, audio_tagging_loss=0.008332, over 14353.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09199, pruned_loss=0.01387, audio_tagging_loss=0.009035, over 3049287.17 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:40:09,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2438433.3333333335, ans=0.125
2023-11-23 15:40:21,943 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:40:25,765 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.34 vs. limit=15.0
2023-11-23 15:40:33,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2438566.6666666665, ans=0.0
2023-11-23 15:40:49,526 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365800
2023-11-23 15:40:54,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.80 vs. limit=15.0
2023-11-23 15:41:10,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2438700.0, ans=0.07
2023-11-23 15:41:12,968 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5100, loss[loss=0.07263, simple_loss=0.09676, pruned_loss=0.01619, audio_tagging_loss=0.008058, over 15043.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09154, pruned_loss=0.01379, audio_tagging_loss=0.008996, over 3048735.72 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:41:18,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2438766.6666666665, ans=10.0
2023-11-23 15:41:36,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2438900.0, ans=0.5
2023-11-23 15:41:54,486 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365850
2023-11-23 15:42:03,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2439033.3333333335, ans=0.125
2023-11-23 15:42:06,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2439033.3333333335, ans=0.2
2023-11-23 15:42:07,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2439033.3333333335, ans=0.1
2023-11-23 15:42:11,231 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.471e+01 8.600e+01 9.200e+01 9.982e+01 4.087e+02, threshold=1.840e+02, percent-clipped=1.0
2023-11-23 15:42:15,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2439100.0, ans=0.125
2023-11-23 15:42:16,222 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5150, loss[loss=0.06518, simple_loss=0.08605, pruned_loss=0.01375, audio_tagging_loss=0.008403, over 13771.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09134, pruned_loss=0.01379, audio_tagging_loss=0.008992, over 3044980.95 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 15:42:23,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2439100.0, ans=0.0
2023-11-23 15:42:27,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2439166.6666666665, ans=0.1
2023-11-23 15:42:36,340 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.68 vs. limit=15.0
2023-11-23 15:42:59,320 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365900
2023-11-23 15:43:00,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2439300.0, ans=0.1
2023-11-23 15:43:03,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2439300.0, ans=0.125
2023-11-23 15:43:10,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2439366.6666666665, ans=0.05
2023-11-23 15:43:13,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2439366.6666666665, ans=0.2
2023-11-23 15:43:20,523 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5200, loss[loss=0.05576, simple_loss=0.07762, pruned_loss=0.009389, audio_tagging_loss=0.007563, over 14457.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09159, pruned_loss=0.0137, audio_tagging_loss=0.00894, over 3046858.86 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:43:24,590 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.85 vs. limit=15.0
2023-11-23 15:44:04,186 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 365950
2023-11-23 15:44:20,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2439700.0, ans=0.0
2023-11-23 15:44:21,114 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.921e+01 8.221e+01 8.985e+01 9.483e+01 1.522e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 15:44:27,289 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5250, loss[loss=0.05373, simple_loss=0.06548, pruned_loss=0.009715, audio_tagging_loss=0.01127, over 14258.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09306, pruned_loss=0.01405, audio_tagging_loss=0.008868, over 3043503.71 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:45:01,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2439900.0, ans=0.04949747468305833
2023-11-23 15:45:09,943 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366000
2023-11-23 15:45:29,168 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.54 vs. limit=22.5
2023-11-23 15:45:33,377 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5300, loss[loss=0.06142, simple_loss=0.0768, pruned_loss=0.009477, audio_tagging_loss=0.01354, over 14973.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09264, pruned_loss=0.01409, audio_tagging_loss=0.00891, over 3043241.52 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:45:36,829 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0
2023-11-23 15:45:38,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2440100.0, ans=0.125
2023-11-23 15:46:17,194 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366050
2023-11-23 15:46:32,939 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.000e+01 8.490e+01 8.834e+01 9.644e+01 1.833e+02, threshold=1.767e+02, percent-clipped=1.0
2023-11-23 15:46:37,977 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5350, loss[loss=0.0762, simple_loss=0.1058, pruned_loss=0.0144, audio_tagging_loss=0.008899, over 14733.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09254, pruned_loss=0.01394, audio_tagging_loss=0.008871, over 3042510.54 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:46:58,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2440500.0, ans=0.0
2023-11-23 15:47:05,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.whiten.whitening_limit, batch_count=2440566.6666666665, ans=12.0
2023-11-23 15:47:16,568 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.61 vs. limit=15.0
2023-11-23 15:47:21,104 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366100
2023-11-23 15:47:28,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2440700.0, ans=0.025
2023-11-23 15:47:42,584 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5400, loss[loss=0.07751, simple_loss=0.09758, pruned_loss=0.01803, audio_tagging_loss=0.01069, over 15257.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09331, pruned_loss=0.01398, audio_tagging_loss=0.008905, over 3049811.06 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:47:49,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2440766.6666666665, ans=0.2
2023-11-23 15:47:50,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2440766.6666666665, ans=0.125
2023-11-23 15:47:51,021 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.36 vs. limit=15.0
2023-11-23 15:47:51,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2440766.6666666665, ans=0.0
2023-11-23 15:47:59,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2440833.3333333335, ans=0.0
2023-11-23 15:48:04,600 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:48:14,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2440900.0, ans=0.125
2023-11-23 15:48:26,190 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366150
2023-11-23 15:48:33,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2441033.3333333335, ans=0.1
2023-11-23 15:48:33,961 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:48:42,976 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.765e+01 8.475e+01 9.130e+01 9.785e+01 1.452e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-23 15:48:44,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2441033.3333333335, ans=0.0
2023-11-23 15:48:48,677 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5450, loss[loss=0.06548, simple_loss=0.08538, pruned_loss=0.01416, audio_tagging_loss=0.008635, over 15394.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09239, pruned_loss=0.01395, audio_tagging_loss=0.009017, over 3046757.93 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:49:15,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2441233.3333333335, ans=0.125
2023-11-23 15:49:25,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2441233.3333333335, ans=0.1
2023-11-23 15:49:31,089 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366200
2023-11-23 15:49:32,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2441300.0, ans=0.0
2023-11-23 15:49:41,572 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.39 vs. limit=15.0
2023-11-23 15:49:52,724 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5500, loss[loss=0.06762, simple_loss=0.09155, pruned_loss=0.01138, audio_tagging_loss=0.01046, over 15390.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.0923, pruned_loss=0.01401, audio_tagging_loss=0.00911, over 3048523.88 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:50:11,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2441500.0, ans=0.05
2023-11-23 15:50:35,846 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366250
2023-11-23 15:50:40,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2441633.3333333335, ans=0.125
2023-11-23 15:50:52,275 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.602e+01 8.303e+01 8.937e+01 9.609e+01 1.241e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 15:50:53,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2441700.0, ans=0.0
2023-11-23 15:50:57,399 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5550, loss[loss=0.07918, simple_loss=0.11, pruned_loss=0.0139, audio_tagging_loss=0.01028, over 15315.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09229, pruned_loss=0.01414, audio_tagging_loss=0.009238, over 3045651.02 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:51:06,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.68 vs. limit=15.0
2023-11-23 15:51:20,020 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.71 vs. limit=12.0
2023-11-23 15:51:22,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2441900.0, ans=0.125
2023-11-23 15:51:27,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2441900.0, ans=0.2
2023-11-23 15:51:29,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2441900.0, ans=0.1
2023-11-23 15:51:40,001 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366300
2023-11-23 15:51:44,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2441966.6666666665, ans=0.035
2023-11-23 15:51:48,162 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.18 vs. limit=22.5
2023-11-23 15:51:59,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2442033.3333333335, ans=0.125
2023-11-23 15:52:01,532 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5600, loss[loss=0.07842, simple_loss=0.1112, pruned_loss=0.01585, audio_tagging_loss=0.006952, over 14796.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09176, pruned_loss=0.01383, audio_tagging_loss=0.009295, over 3042363.59 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:52:04,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2442100.0, ans=0.125
2023-11-23 15:52:05,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2442100.0, ans=0.125
2023-11-23 15:52:14,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2442166.6666666665, ans=0.2
2023-11-23 15:52:17,701 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.82 vs. limit=12.0
2023-11-23 15:52:44,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366350
2023-11-23 15:52:48,047 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 15:52:48,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2442300.0, ans=0.125
2023-11-23 15:52:50,734 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:52:56,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2442366.6666666665, ans=0.0
2023-11-23 15:53:00,690 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.372e+01 8.291e+01 8.935e+01 9.655e+01 1.271e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 15:53:05,689 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5650, loss[loss=0.05906, simple_loss=0.07492, pruned_loss=0.01237, audio_tagging_loss=0.009227, over 14829.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09142, pruned_loss=0.01375, audio_tagging_loss=0.009375, over 3045815.32 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:53:44,100 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:53:47,669 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366400
2023-11-23 15:53:56,161 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 15:54:01,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2442700.0, ans=0.1
2023-11-23 15:54:09,825 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5700, loss[loss=0.06156, simple_loss=0.07945, pruned_loss=0.01138, audio_tagging_loss=0.01046, over 14461.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09054, pruned_loss=0.01354, audio_tagging_loss=0.009339, over 3041245.12 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:54:46,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2442966.6666666665, ans=0.0
2023-11-23 15:54:51,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366450
2023-11-23 15:55:08,823 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.337e+01 8.162e+01 8.678e+01 9.471e+01 1.109e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-23 15:55:12,211 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.87 vs. limit=15.0
2023-11-23 15:55:13,708 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5750, loss[loss=0.06548, simple_loss=0.09481, pruned_loss=0.01122, audio_tagging_loss=0.006864, over 14557.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09038, pruned_loss=0.01353, audio_tagging_loss=0.009182, over 3047794.48 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 15:55:14,646 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.96 vs. limit=15.0
2023-11-23 15:55:35,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2443166.6666666665, ans=0.125
2023-11-23 15:55:41,158 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.08 vs. limit=22.5
2023-11-23 15:55:48,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.50 vs. limit=22.5
2023-11-23 15:55:54,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2443300.0, ans=0.05
2023-11-23 15:55:56,259 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366500
2023-11-23 15:56:00,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2443300.0, ans=0.2
2023-11-23 15:56:09,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2443366.6666666665, ans=0.125
2023-11-23 15:56:17,478 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5800, loss[loss=0.07209, simple_loss=0.1041, pruned_loss=0.01426, audio_tagging_loss=0.005808, over 15608.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09051, pruned_loss=0.0136, audio_tagging_loss=0.009042, over 3053050.28 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:56:17,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2443433.3333333335, ans=0.125
2023-11-23 15:56:44,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2443566.6666666665, ans=0.125
2023-11-23 15:57:00,784 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366550
2023-11-23 15:57:04,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2443633.3333333335, ans=0.07
2023-11-23 15:57:18,850 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.421e+01 8.535e+01 9.028e+01 9.591e+01 1.185e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 15:57:23,526 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5850, loss[loss=0.07207, simple_loss=0.09095, pruned_loss=0.017, audio_tagging_loss=0.0096, over 14186.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09065, pruned_loss=0.01375, audio_tagging_loss=0.009002, over 3040485.59 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:57:56,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=2443900.0, ans=15.0
2023-11-23 15:58:00,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0
2023-11-23 15:58:03,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2443966.6666666665, ans=0.125
2023-11-23 15:58:06,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0
2023-11-23 15:58:07,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366600
2023-11-23 15:58:14,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2443966.6666666665, ans=0.2
2023-11-23 15:58:14,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2443966.6666666665, ans=0.125
2023-11-23 15:58:27,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2444033.3333333335, ans=0.125
2023-11-23 15:58:30,440 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5900, loss[loss=0.06695, simple_loss=0.09012, pruned_loss=0.01179, audio_tagging_loss=0.0101, over 15490.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09086, pruned_loss=0.0137, audio_tagging_loss=0.008971, over 3041803.19 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:58:39,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2444100.0, ans=0.0
2023-11-23 15:58:46,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2444166.6666666665, ans=0.0
2023-11-23 15:59:13,091 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.67 vs. limit=22.5
2023-11-23 15:59:14,290 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366650
2023-11-23 15:59:14,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2444300.0, ans=0.125
2023-11-23 15:59:18,526 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.42 vs. limit=15.0
2023-11-23 15:59:23,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2444366.6666666665, ans=0.125
2023-11-23 15:59:31,531 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.592e+01 8.501e+01 9.185e+01 9.833e+01 1.392e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 15:59:31,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2444366.6666666665, ans=0.125
2023-11-23 15:59:35,334 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 5950, loss[loss=0.06593, simple_loss=0.08916, pruned_loss=0.01204, audio_tagging_loss=0.009312, over 15207.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09012, pruned_loss=0.0136, audio_tagging_loss=0.009089, over 3045184.00 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 15:59:53,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2444500.0, ans=0.125
2023-11-23 16:00:12,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2444566.6666666665, ans=0.125
2023-11-23 16:00:19,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366700
2023-11-23 16:00:41,892 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6000, loss[loss=0.08562, simple_loss=0.1174, pruned_loss=0.02019, audio_tagging_loss=0.006748, over 16101.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09039, pruned_loss=0.01365, audio_tagging_loss=0.009078, over 3047586.10 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:00:41,895 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 16:01:26,555 INFO [train_asr.py:1253] (0/4) Epoch 31, validation: loss=0.05852, simple_loss=0.05109, pruned_loss=0.00511, audio_tagging_loss=0.02786, over 4681554.00 frames. 
2023-11-23 16:01:26,556 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 16:01:59,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2444900.0, ans=0.0
2023-11-23 16:02:08,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366750
2023-11-23 16:02:13,148 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 16:02:26,660 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.104e+01 8.225e+01 8.734e+01 9.674e+01 1.175e+02, threshold=1.747e+02, percent-clipped=0.0
2023-11-23 16:02:30,472 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6050, loss[loss=0.1016, simple_loss=0.1425, pruned_loss=0.02275, audio_tagging_loss=0.007598, over 16229.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09168, pruned_loss=0.01387, audio_tagging_loss=0.00899, over 3041126.04 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:02:45,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2445166.6666666665, ans=0.09899494936611666
2023-11-23 16:02:59,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2445233.3333333335, ans=0.0
2023-11-23 16:03:03,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2445233.3333333335, ans=0.125
2023-11-23 16:03:14,281 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366800
2023-11-23 16:03:32,892 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.42 vs. limit=15.0
2023-11-23 16:03:34,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2445366.6666666665, ans=15.0
2023-11-23 16:03:36,441 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6100, loss[loss=0.08463, simple_loss=0.1135, pruned_loss=0.02039, audio_tagging_loss=0.007521, over 14067.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09164, pruned_loss=0.01381, audio_tagging_loss=0.009107, over 3035621.64 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:03:48,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2445433.3333333335, ans=0.125
2023-11-23 16:03:49,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2445500.0, ans=0.125
2023-11-23 16:04:17,651 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.55 vs. limit=6.0
2023-11-23 16:04:19,353 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366850
2023-11-23 16:04:25,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2445633.3333333335, ans=0.1
2023-11-23 16:04:34,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2445700.0, ans=0.2
2023-11-23 16:04:38,279 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.053e+01 8.426e+01 8.983e+01 9.647e+01 1.287e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 16:04:42,609 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6150, loss[loss=0.06011, simple_loss=0.08058, pruned_loss=0.0105, audio_tagging_loss=0.009324, over 15685.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.0913, pruned_loss=0.01379, audio_tagging_loss=0.009153, over 3035348.65 frames. ], batch size: 61, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:04:45,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2445766.6666666665, ans=0.0
2023-11-23 16:05:02,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2445833.3333333335, ans=0.125
2023-11-23 16:05:19,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2445966.6666666665, ans=0.2
2023-11-23 16:05:25,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366900
2023-11-23 16:05:37,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2446033.3333333335, ans=0.125
2023-11-23 16:05:47,295 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6200, loss[loss=0.08206, simple_loss=0.1088, pruned_loss=0.01896, audio_tagging_loss=0.008707, over 15475.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09123, pruned_loss=0.01368, audio_tagging_loss=0.009157, over 3036428.73 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:05:47,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2446100.0, ans=0.0
2023-11-23 16:05:52,894 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.86 vs. limit=12.0
2023-11-23 16:05:59,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.34 vs. limit=22.5
2023-11-23 16:06:21,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2446233.3333333335, ans=0.125
2023-11-23 16:06:23,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2446233.3333333335, ans=0.1
2023-11-23 16:06:31,198 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 366950
2023-11-23 16:06:38,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2446366.6666666665, ans=0.0
2023-11-23 16:06:47,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2446366.6666666665, ans=0.125
2023-11-23 16:06:48,425 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.382e+01 9.012e+01 9.950e+01 2.061e+02, threshold=1.802e+02, percent-clipped=1.0
2023-11-23 16:06:52,800 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6250, loss[loss=0.06869, simple_loss=0.09322, pruned_loss=0.01473, audio_tagging_loss=0.00735, over 14361.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09098, pruned_loss=0.01357, audio_tagging_loss=0.009263, over 3046143.65 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:06:54,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2446433.3333333335, ans=0.125
2023-11-23 16:07:20,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2446566.6666666665, ans=0.2
2023-11-23 16:07:24,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2446566.6666666665, ans=0.0
2023-11-23 16:07:32,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2446633.3333333335, ans=0.2
2023-11-23 16:07:35,898 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367000
2023-11-23 16:07:45,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2446700.0, ans=0.0
2023-11-23 16:07:58,554 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6300, loss[loss=0.05813, simple_loss=0.07458, pruned_loss=0.008337, audio_tagging_loss=0.0125, over 15014.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09128, pruned_loss=0.01366, audio_tagging_loss=0.009326, over 3046101.25 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:08:04,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2446766.6666666665, ans=0.125
2023-11-23 16:08:12,810 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:08:20,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2446833.3333333335, ans=10.0
2023-11-23 16:08:40,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367050
2023-11-23 16:08:50,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2447033.3333333335, ans=0.125
2023-11-23 16:08:59,069 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.957e+01 8.355e+01 9.008e+01 9.768e+01 1.209e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 16:09:02,797 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6350, loss[loss=0.04523, simple_loss=0.05572, pruned_loss=0.004902, audio_tagging_loss=0.01247, over 16906.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09111, pruned_loss=0.01361, audio_tagging_loss=0.009429, over 3052730.52 frames. ], batch size: 67, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:09:09,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2447100.0, ans=0.2
2023-11-23 16:09:35,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2447233.3333333335, ans=0.125
2023-11-23 16:09:39,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2447233.3333333335, ans=0.2
2023-11-23 16:09:43,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2447300.0, ans=0.1
2023-11-23 16:09:45,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367100
2023-11-23 16:10:06,306 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6400, loss[loss=0.05841, simple_loss=0.08007, pruned_loss=0.01075, audio_tagging_loss=0.007631, over 15250.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09113, pruned_loss=0.0137, audio_tagging_loss=0.009499, over 3055532.33 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:10:17,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2447433.3333333335, ans=0.2
2023-11-23 16:10:49,474 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367150
2023-11-23 16:10:49,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2447633.3333333335, ans=0.125
2023-11-23 16:10:49,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2447633.3333333335, ans=0.0
2023-11-23 16:11:07,639 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:11:08,617 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.384e+01 9.024e+01 9.742e+01 1.267e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 16:11:11,824 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6450, loss[loss=0.0793, simple_loss=0.09936, pruned_loss=0.02054, audio_tagging_loss=0.009081, over 15471.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09077, pruned_loss=0.01361, audio_tagging_loss=0.009504, over 3049868.26 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:11:21,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2447766.6666666665, ans=0.125
2023-11-23 16:11:21,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2447766.6666666665, ans=0.0
2023-11-23 16:11:27,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2447833.3333333335, ans=0.125
2023-11-23 16:11:29,100 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.07 vs. limit=10.0
2023-11-23 16:11:52,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2447966.6666666665, ans=0.125
2023-11-23 16:11:53,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2447966.6666666665, ans=15.0
2023-11-23 16:11:53,924 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367200
2023-11-23 16:12:02,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.39 vs. limit=10.0
2023-11-23 16:12:17,106 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6500, loss[loss=0.05565, simple_loss=0.07101, pruned_loss=0.01022, audio_tagging_loss=0.009927, over 15170.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09149, pruned_loss=0.01374, audio_tagging_loss=0.009424, over 3048323.65 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:12:32,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2448166.6666666665, ans=0.125
2023-11-23 16:12:42,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2448233.3333333335, ans=0.125
2023-11-23 16:12:45,638 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.52 vs. limit=12.0
2023-11-23 16:13:00,539 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367250
2023-11-23 16:13:00,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2448300.0, ans=0.0
2023-11-23 16:13:12,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2448366.6666666665, ans=10.0
2023-11-23 16:13:18,764 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.647e+01 8.406e+01 9.076e+01 9.815e+01 1.354e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 16:13:21,308 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6550, loss[loss=0.06624, simple_loss=0.08495, pruned_loss=0.0121, audio_tagging_loss=0.01166, over 16136.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09167, pruned_loss=0.01377, audio_tagging_loss=0.009306, over 3049123.54 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:13:30,297 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.56 vs. limit=15.0
2023-11-23 16:13:35,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.56 vs. limit=22.5
2023-11-23 16:13:35,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2448500.0, ans=0.07
2023-11-23 16:14:04,637 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367300
2023-11-23 16:14:13,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2448700.0, ans=0.125
2023-11-23 16:14:15,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2448700.0, ans=0.0
2023-11-23 16:14:25,925 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6600, loss[loss=0.06676, simple_loss=0.09745, pruned_loss=0.01025, audio_tagging_loss=0.00778, over 15409.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.0921, pruned_loss=0.01394, audio_tagging_loss=0.009268, over 3046019.98 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:14:33,861 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=15.0
2023-11-23 16:14:34,007 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.04 vs. limit=15.0
2023-11-23 16:14:36,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2448766.6666666665, ans=0.125
2023-11-23 16:14:42,997 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.00 vs. limit=15.0
2023-11-23 16:14:49,136 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.02 vs. limit=6.0
2023-11-23 16:14:52,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2448900.0, ans=0.0
2023-11-23 16:14:58,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2448900.0, ans=0.125
2023-11-23 16:15:01,061 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:15:08,973 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367350
2023-11-23 16:15:15,701 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.06 vs. limit=15.0
2023-11-23 16:15:25,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2449033.3333333335, ans=0.2
2023-11-23 16:15:27,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2449033.3333333335, ans=0.0
2023-11-23 16:15:31,458 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.404e+01 9.056e+01 9.809e+01 1.686e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 16:15:31,503 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6650, loss[loss=0.06412, simple_loss=0.08954, pruned_loss=0.01185, audio_tagging_loss=0.0075, over 14843.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09095, pruned_loss=0.01373, audio_tagging_loss=0.009193, over 3034064.03 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:15:31,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2449100.0, ans=0.0
2023-11-23 16:15:37,241 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.36 vs. limit=15.0
2023-11-23 16:15:54,956 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.55 vs. limit=10.0
2023-11-23 16:16:08,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2449300.0, ans=0.125
2023-11-23 16:16:14,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367400
2023-11-23 16:16:28,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2449366.6666666665, ans=0.125
2023-11-23 16:16:31,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2449366.6666666665, ans=0.125
2023-11-23 16:16:34,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2449433.3333333335, ans=0.125
2023-11-23 16:16:35,757 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6700, loss[loss=0.07916, simple_loss=0.111, pruned_loss=0.01616, audio_tagging_loss=0.007466, over 15089.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09139, pruned_loss=0.01389, audio_tagging_loss=0.009155, over 3030121.86 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:16:55,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2449500.0, ans=0.1
2023-11-23 16:17:19,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367450
2023-11-23 16:17:21,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2449633.3333333335, ans=0.1
2023-11-23 16:17:40,237 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.536e+01 8.542e+01 9.005e+01 9.931e+01 1.410e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 16:17:40,281 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6750, loss[loss=0.04921, simple_loss=0.04884, pruned_loss=0.009679, audio_tagging_loss=0.01511, over 13008.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09043, pruned_loss=0.01384, audio_tagging_loss=0.009151, over 3026152.31 frames. ], batch size: 52, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:17:40,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2449766.6666666665, ans=0.0
2023-11-23 16:17:43,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2449766.6666666665, ans=0.2
2023-11-23 16:17:44,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2449766.6666666665, ans=0.1
2023-11-23 16:17:52,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.01 vs. limit=15.0
2023-11-23 16:17:53,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2449833.3333333335, ans=0.125
2023-11-23 16:18:06,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2449900.0, ans=0.0
2023-11-23 16:18:15,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2449900.0, ans=0.0
2023-11-23 16:18:23,591 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367500
2023-11-23 16:18:40,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2450033.3333333335, ans=0.125
2023-11-23 16:18:45,568 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6800, loss[loss=0.06661, simple_loss=0.09332, pruned_loss=0.01018, audio_tagging_loss=0.009778, over 14937.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.08972, pruned_loss=0.01355, audio_tagging_loss=0.009115, over 3032653.57 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:18:50,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2450100.0, ans=0.125
2023-11-23 16:19:02,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2450166.6666666665, ans=0.0
2023-11-23 16:19:14,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2450233.3333333335, ans=0.125
2023-11-23 16:19:28,374 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367550
2023-11-23 16:19:28,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2450300.0, ans=0.1
2023-11-23 16:19:44,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2450366.6666666665, ans=0.0
2023-11-23 16:19:50,474 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6850, loss[loss=0.06509, simple_loss=0.07994, pruned_loss=0.01376, audio_tagging_loss=0.01135, over 14769.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09013, pruned_loss=0.01356, audio_tagging_loss=0.00908, over 3029720.16 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:19:50,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2450433.3333333335, ans=0.125
2023-11-23 16:19:51,630 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 8.211e+01 8.952e+01 9.815e+01 1.222e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 16:19:54,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2450433.3333333335, ans=0.125
2023-11-23 16:20:01,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2450433.3333333335, ans=0.09899494936611666
2023-11-23 16:20:14,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.19 vs. limit=15.0
2023-11-23 16:20:17,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2450566.6666666665, ans=0.125
2023-11-23 16:20:19,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2450566.6666666665, ans=0.0
2023-11-23 16:20:32,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2450633.3333333335, ans=0.1
2023-11-23 16:20:33,209 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367600
2023-11-23 16:20:39,454 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:20:55,992 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6900, loss[loss=0.05466, simple_loss=0.0756, pruned_loss=0.007676, audio_tagging_loss=0.009181, over 15818.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09028, pruned_loss=0.01359, audio_tagging_loss=0.00905, over 3034443.59 frames. ], batch size: 60, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:20:56,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2450766.6666666665, ans=0.125
2023-11-23 16:21:05,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2450766.6666666665, ans=0.0
2023-11-23 16:21:25,133 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.15 vs. limit=15.0
2023-11-23 16:21:38,520 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367650
2023-11-23 16:21:41,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2450966.6666666665, ans=0.125
2023-11-23 16:21:46,580 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 16:21:51,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2451033.3333333335, ans=0.2
2023-11-23 16:21:56,381 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.23 vs. limit=15.0
2023-11-23 16:22:01,293 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 6950, loss[loss=0.09542, simple_loss=0.1359, pruned_loss=0.02111, audio_tagging_loss=0.006371, over 15748.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09137, pruned_loss=0.01375, audio_tagging_loss=0.008934, over 3036971.58 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:22:02,445 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.751e+01 8.116e+01 8.969e+01 9.790e+01 1.259e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-23 16:22:05,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2451100.0, ans=0.125
2023-11-23 16:22:37,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2451233.3333333335, ans=0.125
2023-11-23 16:22:44,059 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367700
2023-11-23 16:22:49,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2451300.0, ans=0.125
2023-11-23 16:22:53,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2451366.6666666665, ans=0.035
2023-11-23 16:22:57,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2451366.6666666665, ans=0.125
2023-11-23 16:23:00,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2451366.6666666665, ans=0.09899494936611666
2023-11-23 16:23:05,274 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7000, loss[loss=0.05646, simple_loss=0.07471, pruned_loss=0.01015, audio_tagging_loss=0.008952, over 15676.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09159, pruned_loss=0.01372, audio_tagging_loss=0.008866, over 3040970.10 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:23:11,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2451433.3333333335, ans=0.125
2023-11-23 16:23:46,450 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.91 vs. limit=12.0
2023-11-23 16:23:48,452 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367750
2023-11-23 16:23:48,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2451633.3333333335, ans=0.125
2023-11-23 16:23:53,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2451633.3333333335, ans=0.125
2023-11-23 16:23:55,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2451700.0, ans=0.0
2023-11-23 16:24:07,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2451700.0, ans=0.0
2023-11-23 16:24:10,643 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7050, loss[loss=0.05803, simple_loss=0.07582, pruned_loss=0.008874, audio_tagging_loss=0.01124, over 15120.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09169, pruned_loss=0.01378, audio_tagging_loss=0.008946, over 3048143.97 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:24:11,794 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.925e+01 8.295e+01 8.920e+01 9.938e+01 1.396e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 16:24:34,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2451833.3333333335, ans=10.0
2023-11-23 16:24:40,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2451900.0, ans=0.1
2023-11-23 16:24:50,544 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.89 vs. limit=12.0
2023-11-23 16:24:53,012 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367800
2023-11-23 16:24:58,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2451966.6666666665, ans=0.125
2023-11-23 16:25:13,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2452033.3333333335, ans=0.125
2023-11-23 16:25:15,569 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7100, loss[loss=0.06873, simple_loss=0.09306, pruned_loss=0.01449, audio_tagging_loss=0.007712, over 14970.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09147, pruned_loss=0.01376, audio_tagging_loss=0.009162, over 3045841.12 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:25:42,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2452233.3333333335, ans=0.125
2023-11-23 16:25:45,625 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.63 vs. limit=12.0
2023-11-23 16:25:58,064 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367850
2023-11-23 16:26:18,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2452366.6666666665, ans=0.0
2023-11-23 16:26:20,286 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7150, loss[loss=0.07811, simple_loss=0.1113, pruned_loss=0.01519, audio_tagging_loss=0.007262, over 15216.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09206, pruned_loss=0.01372, audio_tagging_loss=0.009185, over 3047230.28 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:26:21,459 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.184e+01 8.537e+01 9.022e+01 9.919e+01 1.379e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 16:26:30,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2452433.3333333335, ans=0.125
2023-11-23 16:26:37,887 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0
2023-11-23 16:26:58,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2452633.3333333335, ans=0.125
2023-11-23 16:27:03,260 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367900
2023-11-23 16:27:12,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2452700.0, ans=0.0
2023-11-23 16:27:14,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=2452700.0, ans=0.02
2023-11-23 16:27:18,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2452700.0, ans=0.1
2023-11-23 16:27:24,467 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7200, loss[loss=0.05963, simple_loss=0.07521, pruned_loss=0.01362, audio_tagging_loss=0.008408, over 16089.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09173, pruned_loss=0.01366, audio_tagging_loss=0.009204, over 3045986.83 frames. ], batch size: 61, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:27:29,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2452766.6666666665, ans=0.125
2023-11-23 16:27:32,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2452766.6666666665, ans=0.0
2023-11-23 16:27:43,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2452833.3333333335, ans=0.0
2023-11-23 16:27:44,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2452833.3333333335, ans=0.125
2023-11-23 16:27:51,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2452900.0, ans=0.125
2023-11-23 16:27:51,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2452900.0, ans=0.125
2023-11-23 16:28:00,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2452900.0, ans=0.0
2023-11-23 16:28:06,768 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 367950
2023-11-23 16:28:25,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2453033.3333333335, ans=0.125
2023-11-23 16:28:28,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.78 vs. limit=22.5
2023-11-23 16:28:29,954 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7250, loss[loss=0.07171, simple_loss=0.107, pruned_loss=0.01254, audio_tagging_loss=0.005658, over 14616.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09146, pruned_loss=0.01368, audio_tagging_loss=0.009226, over 3045644.60 frames. ], batch size: 55, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:28:30,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2453100.0, ans=0.1
2023-11-23 16:28:31,189 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.952e+01 8.513e+01 9.044e+01 9.572e+01 1.281e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 16:28:47,536 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.35 vs. limit=22.5
2023-11-23 16:29:11,043 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.70 vs. limit=15.0
2023-11-23 16:29:13,004 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368000
2023-11-23 16:29:14,476 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-368000.pt
2023-11-23 16:29:24,940 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.04 vs. limit=22.5
2023-11-23 16:29:38,266 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7300, loss[loss=0.06457, simple_loss=0.08533, pruned_loss=0.01317, audio_tagging_loss=0.008726, over 15609.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09137, pruned_loss=0.01363, audio_tagging_loss=0.009207, over 3038684.31 frames. ], batch size: 56, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:29:43,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2453433.3333333335, ans=0.125
2023-11-23 16:29:48,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2453433.3333333335, ans=0.125
2023-11-23 16:30:01,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2453500.0, ans=0.0
2023-11-23 16:30:22,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368050
2023-11-23 16:30:29,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.87 vs. limit=15.0
2023-11-23 16:30:32,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2453700.0, ans=0.125
2023-11-23 16:30:42,848 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7350, loss[loss=0.08025, simple_loss=0.1069, pruned_loss=0.02099, audio_tagging_loss=0.005835, over 15442.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09105, pruned_loss=0.0135, audio_tagging_loss=0.009056, over 3046784.76 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:30:44,153 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.064e+01 8.269e+01 8.908e+01 9.511e+01 1.304e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 16:30:46,406 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2453766.6666666665, ans=0.125
2023-11-23 16:30:52,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2453766.6666666665, ans=0.5
2023-11-23 16:30:52,672 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.94 vs. limit=15.0
2023-11-23 16:31:04,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2453833.3333333335, ans=0.1
2023-11-23 16:31:21,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2453966.6666666665, ans=0.125
2023-11-23 16:31:25,890 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.60 vs. limit=15.0
2023-11-23 16:31:26,589 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368100
2023-11-23 16:31:37,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2454033.3333333335, ans=0.125
2023-11-23 16:31:39,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2454033.3333333335, ans=0.125
2023-11-23 16:31:49,071 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7400, loss[loss=0.05275, simple_loss=0.07816, pruned_loss=0.004995, audio_tagging_loss=0.008671, over 14674.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09113, pruned_loss=0.01347, audio_tagging_loss=0.008976, over 3045395.38 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:31:52,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2454100.0, ans=0.07
2023-11-23 16:32:13,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2454166.6666666665, ans=0.0
2023-11-23 16:32:30,243 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:32:32,565 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368150
2023-11-23 16:32:38,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2454300.0, ans=0.1
2023-11-23 16:32:55,372 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7450, loss[loss=0.08039, simple_loss=0.1111, pruned_loss=0.01565, audio_tagging_loss=0.009205, over 15197.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09233, pruned_loss=0.01358, audio_tagging_loss=0.008885, over 3043623.46 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:32:56,567 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.753e+01 8.234e+01 8.919e+01 9.567e+01 1.256e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 16:33:03,312 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.53 vs. limit=15.0
2023-11-23 16:33:20,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2454566.6666666665, ans=0.07
2023-11-23 16:33:38,311 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368200
2023-11-23 16:33:50,202 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.55 vs. limit=15.0
2023-11-23 16:33:52,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2454700.0, ans=0.125
2023-11-23 16:33:59,407 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7500, loss[loss=0.05792, simple_loss=0.08222, pruned_loss=0.008694, audio_tagging_loss=0.00812, over 13893.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.0922, pruned_loss=0.0135, audio_tagging_loss=0.008885, over 3039591.41 frames. ], batch size: 52, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:34:12,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2454833.3333333335, ans=0.125
2023-11-23 16:34:27,842 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:34:38,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2454966.6666666665, ans=0.05
2023-11-23 16:34:42,474 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368250
2023-11-23 16:34:42,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2454966.6666666665, ans=0.125
2023-11-23 16:35:01,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2455033.3333333335, ans=0.125
2023-11-23 16:35:03,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2455100.0, ans=0.125
2023-11-23 16:35:04,299 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7550, loss[loss=0.06378, simple_loss=0.08143, pruned_loss=0.01307, audio_tagging_loss=0.009999, over 15150.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09266, pruned_loss=0.01365, audio_tagging_loss=0.008867, over 3040363.98 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 16.0
2023-11-23 16:35:05,440 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.880e+01 8.421e+01 8.983e+01 9.712e+01 1.318e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 16:35:08,064 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.44 vs. limit=15.0
2023-11-23 16:35:28,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2455166.6666666665, ans=0.125
2023-11-23 16:35:39,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.19 vs. limit=12.0
2023-11-23 16:35:41,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2455233.3333333335, ans=0.0
2023-11-23 16:35:48,369 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368300
2023-11-23 16:35:59,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2455366.6666666665, ans=0.95
2023-11-23 16:36:07,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2455366.6666666665, ans=0.125
2023-11-23 16:36:10,443 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7600, loss[loss=0.06759, simple_loss=0.09253, pruned_loss=0.01106, audio_tagging_loss=0.01027, over 14848.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09242, pruned_loss=0.01367, audio_tagging_loss=0.008864, over 3044488.60 frames. ], batch size: 57, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:36:11,103 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.19 vs. limit=6.0
2023-11-23 16:36:49,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2455633.3333333335, ans=0.0
2023-11-23 16:36:53,354 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368350
2023-11-23 16:36:53,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2455633.3333333335, ans=0.2
2023-11-23 16:36:55,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2455633.3333333335, ans=0.125
2023-11-23 16:37:00,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2455633.3333333335, ans=0.125
2023-11-23 16:37:15,067 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7650, loss[loss=0.05974, simple_loss=0.08166, pruned_loss=0.008879, audio_tagging_loss=0.01003, over 14640.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.0923, pruned_loss=0.01369, audio_tagging_loss=0.008877, over 3039272.55 frames. ], batch size: 53, lr: 2.19e-03, grad_scale: 32.0
2023-11-23 16:37:15,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2455766.6666666665, ans=0.125
2023-11-23 16:37:15,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2455766.6666666665, ans=0.0
2023-11-23 16:37:16,303 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.823e+01 8.296e+01 8.999e+01 9.815e+01 1.332e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 16:37:17,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2455766.6666666665, ans=0.0
2023-11-23 16:37:18,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2455766.6666666665, ans=0.125
2023-11-23 16:37:19,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2455766.6666666665, ans=0.0
2023-11-23 16:37:24,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2455766.6666666665, ans=0.125
2023-11-23 16:37:28,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2455833.3333333335, ans=0.0
2023-11-23 16:37:37,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2455833.3333333335, ans=0.0
2023-11-23 16:37:38,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2455833.3333333335, ans=0.125
2023-11-23 16:37:58,782 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368400
2023-11-23 16:38:04,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2455966.6666666665, ans=0.125
2023-11-23 16:38:10,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2456033.3333333335, ans=0.2
2023-11-23 16:38:19,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2456100.0, ans=0.0
2023-11-23 16:38:20,828 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7700, loss[loss=0.07509, simple_loss=0.1058, pruned_loss=0.01437, audio_tagging_loss=0.007827, over 15367.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09241, pruned_loss=0.01367, audio_tagging_loss=0.008898, over 3039217.97 frames. ], batch size: 58, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:38:21,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2456100.0, ans=0.0
2023-11-23 16:38:21,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2456100.0, ans=0.0
2023-11-23 16:38:30,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2456100.0, ans=0.0
2023-11-23 16:38:43,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.69 vs. limit=22.5
2023-11-23 16:39:03,998 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368450
2023-11-23 16:39:26,205 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7750, loss[loss=0.0759, simple_loss=0.1008, pruned_loss=0.0148, audio_tagging_loss=0.01069, over 14569.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09239, pruned_loss=0.01375, audio_tagging_loss=0.008919, over 3037451.41 frames. ], batch size: 54, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:39:29,816 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.166e+01 8.983e+01 1.017e+02 1.300e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 16:39:30,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2456433.3333333335, ans=0.125
2023-11-23 16:39:30,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2456433.3333333335, ans=0.2
2023-11-23 16:39:40,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2456500.0, ans=0.1
2023-11-23 16:39:49,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2456566.6666666665, ans=0.125
2023-11-23 16:39:55,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2456566.6666666665, ans=0.125
2023-11-23 16:40:02,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2456566.6666666665, ans=0.125
2023-11-23 16:40:08,782 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368500
2023-11-23 16:40:17,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2456700.0, ans=0.07
2023-11-23 16:40:20,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2456700.0, ans=0.125
2023-11-23 16:40:24,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2456700.0, ans=0.0
2023-11-23 16:40:30,627 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7800, loss[loss=0.05131, simple_loss=0.06355, pruned_loss=0.007083, audio_tagging_loss=0.01245, over 15034.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09221, pruned_loss=0.01385, audio_tagging_loss=0.008993, over 3039423.67 frames. ], batch size: 59, lr: 2.19e-03, grad_scale: 8.0
2023-11-23 16:40:32,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2456766.6666666665, ans=0.0
2023-11-23 16:40:42,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2456833.3333333335, ans=0.125
2023-11-23 16:40:49,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2456833.3333333335, ans=0.0
2023-11-23 16:41:02,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.95 vs. limit=12.0
2023-11-23 16:41:13,773 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368550
2023-11-23 16:41:35,462 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7850, loss[loss=0.05842, simple_loss=0.08064, pruned_loss=0.009879, audio_tagging_loss=0.008218, over 16198.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09213, pruned_loss=0.01373, audio_tagging_loss=0.009055, over 3042064.35 frames. ], batch size: 62, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:41:39,078 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.884e+01 8.382e+01 9.207e+01 1.008e+02 1.649e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-23 16:41:51,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2457166.6666666665, ans=0.2
2023-11-23 16:41:54,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2457166.6666666665, ans=0.2
2023-11-23 16:42:08,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2457233.3333333335, ans=0.015
2023-11-23 16:42:17,997 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368600
2023-11-23 16:42:23,209 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.21 vs. limit=12.0
2023-11-23 16:42:40,594 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7900, loss[loss=0.07236, simple_loss=0.09584, pruned_loss=0.01413, audio_tagging_loss=0.01031, over 15538.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09195, pruned_loss=0.01363, audio_tagging_loss=0.009077, over 3040352.76 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:42:41,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2457433.3333333335, ans=0.125
2023-11-23 16:43:09,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2457566.6666666665, ans=0.125
2023-11-23 16:43:14,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2457566.6666666665, ans=0.0
2023-11-23 16:43:15,460 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.44 vs. limit=15.0
2023-11-23 16:43:23,363 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368650
2023-11-23 16:43:41,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2457700.0, ans=0.0
2023-11-23 16:43:44,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0
2023-11-23 16:43:45,884 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 7950, loss[loss=0.05282, simple_loss=0.06356, pruned_loss=0.009087, audio_tagging_loss=0.01195, over 15321.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09212, pruned_loss=0.0138, audio_tagging_loss=0.00914, over 3041082.20 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:43:49,525 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.092e+01 8.234e+01 8.856e+01 9.622e+01 1.359e+02, threshold=1.771e+02, percent-clipped=0.0
2023-11-23 16:44:00,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2457833.3333333335, ans=0.95
2023-11-23 16:44:01,287 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 16:44:14,947 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.89 vs. limit=15.0
2023-11-23 16:44:15,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2457900.0, ans=0.0
2023-11-23 16:44:28,272 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368700
2023-11-23 16:44:28,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2457966.6666666665, ans=0.1
2023-11-23 16:44:44,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2458033.3333333335, ans=0.1
2023-11-23 16:44:50,976 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8000, loss[loss=0.06054, simple_loss=0.07415, pruned_loss=0.01198, audio_tagging_loss=0.01149, over 14899.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09135, pruned_loss=0.01381, audio_tagging_loss=0.009214, over 3038710.97 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:45:01,151 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:45:23,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2458233.3333333335, ans=0.125
2023-11-23 16:45:26,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2458233.3333333335, ans=0.125
2023-11-23 16:45:26,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2458233.3333333335, ans=0.2
2023-11-23 16:45:27,874 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.44 vs. limit=12.0
2023-11-23 16:45:33,479 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368750
2023-11-23 16:45:47,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2458366.6666666665, ans=0.125
2023-11-23 16:45:55,500 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8050, loss[loss=0.07528, simple_loss=0.1006, pruned_loss=0.01543, audio_tagging_loss=0.009541, over 17166.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09163, pruned_loss=0.01374, audio_tagging_loss=0.009228, over 3042540.73 frames. ], batch size: 64, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:46:00,368 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.362e+01 8.796e+01 9.330e+01 1.321e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 16:46:19,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2458500.0, ans=0.2
2023-11-23 16:46:38,191 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368800
2023-11-23 16:46:47,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2458700.0, ans=0.0
2023-11-23 16:47:00,261 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8100, loss[loss=0.07147, simple_loss=0.09563, pruned_loss=0.01428, audio_tagging_loss=0.009372, over 15791.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09219, pruned_loss=0.01388, audio_tagging_loss=0.009189, over 3044473.00 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:47:07,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2458766.6666666665, ans=0.1
2023-11-23 16:47:22,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.96 vs. limit=15.0
2023-11-23 16:47:43,033 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368850
2023-11-23 16:47:52,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2459033.3333333335, ans=0.2
2023-11-23 16:47:53,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2459033.3333333335, ans=0.1
2023-11-23 16:47:58,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2459033.3333333335, ans=0.1
2023-11-23 16:48:04,227 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8150, loss[loss=0.05195, simple_loss=0.06965, pruned_loss=0.008953, audio_tagging_loss=0.008172, over 14805.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09245, pruned_loss=0.01391, audio_tagging_loss=0.009021, over 3045189.32 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:48:04,408 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:48:09,577 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.557e+01 9.149e+01 9.819e+01 1.226e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 16:48:20,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2459166.6666666665, ans=0.0
2023-11-23 16:48:25,451 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.70 vs. limit=15.0
2023-11-23 16:48:27,824 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.63 vs. limit=15.0
2023-11-23 16:48:45,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2459300.0, ans=0.1
2023-11-23 16:48:46,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368900
2023-11-23 16:48:51,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2459300.0, ans=0.0
2023-11-23 16:48:55,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2459366.6666666665, ans=0.125
2023-11-23 16:49:08,928 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8200, loss[loss=0.07373, simple_loss=0.09866, pruned_loss=0.01652, audio_tagging_loss=0.007877, over 15080.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09186, pruned_loss=0.01373, audio_tagging_loss=0.008997, over 3048049.62 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:49:09,632 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.51 vs. limit=22.5
2023-11-23 16:49:10,217 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 16:49:16,536 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:49:24,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2459500.0, ans=0.2
2023-11-23 16:49:28,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2459500.0, ans=0.125
2023-11-23 16:49:29,267 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:49:51,298 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 368950
2023-11-23 16:50:03,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2459700.0, ans=0.0
2023-11-23 16:50:13,326 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8250, loss[loss=0.06257, simple_loss=0.08547, pruned_loss=0.01183, audio_tagging_loss=0.008009, over 14720.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09145, pruned_loss=0.01353, audio_tagging_loss=0.008993, over 3041736.65 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:50:15,177 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.84 vs. limit=15.0
2023-11-23 16:50:18,096 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.120e+01 8.966e+01 9.710e+01 1.268e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 16:50:29,344 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.75 vs. limit=12.0
2023-11-23 16:50:56,548 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369000
2023-11-23 16:51:07,443 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.19 vs. limit=15.0
2023-11-23 16:51:18,617 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8300, loss[loss=0.08895, simple_loss=0.1216, pruned_loss=0.0198, audio_tagging_loss=0.008372, over 14904.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09263, pruned_loss=0.01384, audio_tagging_loss=0.008927, over 3051437.93 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:52:01,658 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369050
2023-11-23 16:52:16,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2460366.6666666665, ans=0.1
2023-11-23 16:52:23,785 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8350, loss[loss=0.05637, simple_loss=0.07647, pruned_loss=0.009856, audio_tagging_loss=0.008278, over 16114.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09384, pruned_loss=0.01406, audio_tagging_loss=0.00882, over 3064481.16 frames. ], batch size: 62, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 16:52:29,181 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.409e+01 8.427e+01 9.150e+01 9.760e+01 1.195e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 16:52:38,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2460500.0, ans=0.2
2023-11-23 16:52:58,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2460566.6666666665, ans=0.0
2023-11-23 16:53:06,708 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369100
2023-11-23 16:53:28,829 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8400, loss[loss=0.07778, simple_loss=0.1017, pruned_loss=0.01796, audio_tagging_loss=0.008966, over 15803.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.0932, pruned_loss=0.01399, audio_tagging_loss=0.008909, over 3061282.69 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:53:30,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2460766.6666666665, ans=0.0
2023-11-23 16:53:43,155 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.90 vs. limit=6.0
2023-11-23 16:53:58,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2460900.0, ans=0.0
2023-11-23 16:54:09,837 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:54:10,936 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369150
2023-11-23 16:54:30,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2461033.3333333335, ans=0.125
2023-11-23 16:54:32,573 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8450, loss[loss=0.06198, simple_loss=0.08483, pruned_loss=0.01095, audio_tagging_loss=0.008615, over 15094.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09304, pruned_loss=0.01391, audio_tagging_loss=0.00892, over 3057936.76 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:54:36,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2461100.0, ans=0.0
2023-11-23 16:54:37,478 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.445e+01 8.837e+01 9.573e+01 1.326e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 16:54:37,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2461100.0, ans=0.125
2023-11-23 16:54:45,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2461166.6666666665, ans=0.0
2023-11-23 16:54:46,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2461166.6666666665, ans=0.125
2023-11-23 16:55:02,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2461233.3333333335, ans=0.125
2023-11-23 16:55:16,207 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369200
2023-11-23 16:55:20,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2461300.0, ans=0.125
2023-11-23 16:55:25,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2461366.6666666665, ans=0.0
2023-11-23 16:55:38,837 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8500, loss[loss=0.04938, simple_loss=0.06487, pruned_loss=0.007415, audio_tagging_loss=0.009526, over 14993.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09308, pruned_loss=0.014, audio_tagging_loss=0.008918, over 3056822.74 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:55:52,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2461500.0, ans=0.1
2023-11-23 16:55:53,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2461500.0, ans=0.1
2023-11-23 16:56:13,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2461566.6666666665, ans=0.0
2023-11-23 16:56:21,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369250
2023-11-23 16:56:23,622 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=22.5
2023-11-23 16:56:24,710 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:56:27,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2461633.3333333335, ans=0.0
2023-11-23 16:56:44,051 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8550, loss[loss=0.0681, simple_loss=0.0931, pruned_loss=0.01145, audio_tagging_loss=0.01011, over 14229.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09348, pruned_loss=0.01411, audio_tagging_loss=0.008971, over 3058226.46 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:56:49,051 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.318e+01 8.758e+01 9.491e+01 1.282e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-23 16:57:26,753 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369300
2023-11-23 16:57:29,325 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 16:57:33,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2461966.6666666665, ans=0.125
2023-11-23 16:57:37,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2462033.3333333335, ans=0.1
2023-11-23 16:57:48,085 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8600, loss[loss=0.06979, simple_loss=0.09298, pruned_loss=0.01218, audio_tagging_loss=0.01113, over 15721.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09269, pruned_loss=0.0139, audio_tagging_loss=0.009028, over 3054131.29 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:57:52,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2462100.0, ans=0.125
2023-11-23 16:57:56,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.37 vs. limit=15.0
2023-11-23 16:58:16,358 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.52 vs. limit=15.0
2023-11-23 16:58:27,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2462300.0, ans=0.125
2023-11-23 16:58:31,403 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369350
2023-11-23 16:58:34,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2462300.0, ans=0.1
2023-11-23 16:58:48,275 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.12 vs. limit=22.5
2023-11-23 16:58:49,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=4.39 vs. limit=15.0
2023-11-23 16:58:52,587 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8650, loss[loss=0.0664, simple_loss=0.08249, pruned_loss=0.01518, audio_tagging_loss=0.009973, over 15652.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09337, pruned_loss=0.01397, audio_tagging_loss=0.00908, over 3054882.43 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 16:58:57,969 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.075e+01 8.582e+01 9.201e+01 9.858e+01 1.215e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 16:59:06,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2462500.0, ans=0.0
2023-11-23 16:59:35,348 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369400
2023-11-23 16:59:40,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2462633.3333333335, ans=0.025
2023-11-23 16:59:56,113 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.29 vs. limit=22.5
2023-11-23 16:59:57,838 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8700, loss[loss=0.06097, simple_loss=0.08546, pruned_loss=0.009105, audio_tagging_loss=0.009137, over 15460.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09342, pruned_loss=0.01391, audio_tagging_loss=0.009104, over 3051736.22 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:00:00,395 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.98 vs. limit=12.0
2023-11-23 17:00:16,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2462833.3333333335, ans=0.2
2023-11-23 17:00:41,095 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369450
2023-11-23 17:00:42,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2462966.6666666665, ans=0.025
2023-11-23 17:01:02,658 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8750, loss[loss=0.05646, simple_loss=0.0678, pruned_loss=0.0121, audio_tagging_loss=0.01045, over 15209.00 frames. ], tot_loss[loss=0.06949, simple_loss=0.09287, pruned_loss=0.01385, audio_tagging_loss=0.009203, over 3052694.86 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:01:07,434 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.270e+01 8.973e+01 9.908e+01 1.550e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-23 17:01:23,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2463166.6666666665, ans=0.0
2023-11-23 17:01:24,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2463166.6666666665, ans=0.0
2023-11-23 17:01:36,841 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=12.0
2023-11-23 17:01:40,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2463300.0, ans=0.05
2023-11-23 17:01:42,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2463300.0, ans=0.0
2023-11-23 17:01:45,522 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369500
2023-11-23 17:01:48,542 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.66 vs. limit=22.5
2023-11-23 17:02:00,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2463366.6666666665, ans=0.125
2023-11-23 17:02:01,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2463366.6666666665, ans=0.125
2023-11-23 17:02:05,054 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.94 vs. limit=12.0
2023-11-23 17:02:06,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2463433.3333333335, ans=10.0
2023-11-23 17:02:06,821 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8800, loss[loss=0.06977, simple_loss=0.0862, pruned_loss=0.01566, audio_tagging_loss=0.01101, over 14521.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09298, pruned_loss=0.0138, audio_tagging_loss=0.00927, over 3046925.17 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:02:13,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2463433.3333333335, ans=0.05
2023-11-23 17:02:13,748 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.48 vs. limit=15.0
2023-11-23 17:02:39,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2463566.6666666665, ans=0.2
2023-11-23 17:02:49,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369550
2023-11-23 17:02:57,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2463700.0, ans=0.125
2023-11-23 17:03:12,315 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8850, loss[loss=0.06756, simple_loss=0.09121, pruned_loss=0.01391, audio_tagging_loss=0.00804, over 14393.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09352, pruned_loss=0.01381, audio_tagging_loss=0.009285, over 3050697.42 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:03:16,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2463766.6666666665, ans=0.04949747468305833
2023-11-23 17:03:16,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2463766.6666666665, ans=0.125
2023-11-23 17:03:17,847 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.515e+01 9.142e+01 9.954e+01 1.396e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 17:03:24,152 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:03:31,989 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.22 vs. limit=15.0
2023-11-23 17:03:46,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2463900.0, ans=0.125
2023-11-23 17:03:47,571 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:03:54,679 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369600
2023-11-23 17:04:03,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2464033.3333333335, ans=0.1
2023-11-23 17:04:13,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=12.0
2023-11-23 17:04:16,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2464100.0, ans=0.1
2023-11-23 17:04:17,308 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8900, loss[loss=0.07172, simple_loss=0.08758, pruned_loss=0.01654, audio_tagging_loss=0.01139, over 14279.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09289, pruned_loss=0.01376, audio_tagging_loss=0.009308, over 3062163.63 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:04:26,192 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:04:35,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2464166.6666666665, ans=0.0
2023-11-23 17:04:53,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2464233.3333333335, ans=0.2
2023-11-23 17:05:00,792 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369650
2023-11-23 17:05:22,417 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 8950, loss[loss=0.07529, simple_loss=0.1046, pruned_loss=0.01704, audio_tagging_loss=0.00596, over 15363.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09261, pruned_loss=0.01376, audio_tagging_loss=0.00914, over 3054607.83 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:05:27,243 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.154e+01 8.323e+01 9.092e+01 9.814e+01 1.259e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 17:06:04,580 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.18 vs. limit=15.0
2023-11-23 17:06:04,939 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369700
2023-11-23 17:06:16,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2464700.0, ans=0.125
2023-11-23 17:06:18,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2464700.0, ans=0.125
2023-11-23 17:06:21,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2464700.0, ans=0.125
2023-11-23 17:06:27,180 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9000, loss[loss=0.06958, simple_loss=0.09434, pruned_loss=0.01256, audio_tagging_loss=0.009846, over 15123.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09294, pruned_loss=0.0136, audio_tagging_loss=0.00907, over 3058103.97 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:06:27,183 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 17:06:56,074 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8068, 4.9204, 5.0643, 4.9086], device='cuda:0')
2023-11-23 17:07:11,622 INFO [train_asr.py:1253] (0/4) Epoch 31, validation: loss=0.05916, simple_loss=0.05108, pruned_loss=0.005166, audio_tagging_loss=0.02845, over 4681554.00 frames. 
2023-11-23 17:07:11,622 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 17:07:21,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2464766.6666666665, ans=0.1
2023-11-23 17:07:24,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2464833.3333333335, ans=0.0
2023-11-23 17:07:43,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2464900.0, ans=0.0
2023-11-23 17:07:45,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2464900.0, ans=0.0
2023-11-23 17:07:47,642 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.14 vs. limit=15.0
2023-11-23 17:07:48,475 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0
2023-11-23 17:07:51,827 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.81 vs. limit=15.0
2023-11-23 17:07:54,855 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369750
2023-11-23 17:08:07,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2465033.3333333335, ans=0.05
2023-11-23 17:08:11,426 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.66 vs. limit=22.5
2023-11-23 17:08:14,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2465033.3333333335, ans=0.125
2023-11-23 17:08:16,242 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9050, loss[loss=0.05768, simple_loss=0.07607, pruned_loss=0.009987, audio_tagging_loss=0.009653, over 15195.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09305, pruned_loss=0.01376, audio_tagging_loss=0.008975, over 3052499.87 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:08:21,064 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.342e+01 8.456e+01 8.970e+01 9.727e+01 1.359e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-23 17:08:21,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2465100.0, ans=0.125
2023-11-23 17:08:28,928 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.31 vs. limit=15.0
2023-11-23 17:08:34,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2465166.6666666665, ans=0.2
2023-11-23 17:08:45,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2465233.3333333335, ans=0.125
2023-11-23 17:08:58,829 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369800
2023-11-23 17:09:10,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2465366.6666666665, ans=0.125
2023-11-23 17:09:20,911 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9100, loss[loss=0.05347, simple_loss=0.07419, pruned_loss=0.007441, audio_tagging_loss=0.008926, over 14547.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09285, pruned_loss=0.0137, audio_tagging_loss=0.00893, over 3052327.70 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:09:24,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2465433.3333333335, ans=0.05
2023-11-23 17:09:27,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2465433.3333333335, ans=0.0
2023-11-23 17:09:48,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2465566.6666666665, ans=0.2
2023-11-23 17:10:03,299 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369850
2023-11-23 17:10:09,292 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.91 vs. limit=15.0
2023-11-23 17:10:25,953 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9150, loss[loss=0.07603, simple_loss=0.1098, pruned_loss=0.01449, audio_tagging_loss=0.006622, over 15926.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.0931, pruned_loss=0.01377, audio_tagging_loss=0.008893, over 3050365.84 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:10:30,450 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.17 vs. limit=22.5
2023-11-23 17:10:30,941 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.474e+01 8.198e+01 8.578e+01 9.347e+01 1.177e+02, threshold=1.716e+02, percent-clipped=0.0
2023-11-23 17:10:43,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2465833.3333333335, ans=0.125
2023-11-23 17:10:44,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2465833.3333333335, ans=0.0
2023-11-23 17:10:45,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2465833.3333333335, ans=0.0
2023-11-23 17:10:54,862 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.28 vs. limit=6.0
2023-11-23 17:11:05,207 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.85 vs. limit=15.0
2023-11-23 17:11:08,793 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369900
2023-11-23 17:11:30,959 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9200, loss[loss=0.05979, simple_loss=0.08122, pruned_loss=0.01247, audio_tagging_loss=0.006708, over 14994.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09239, pruned_loss=0.01368, audio_tagging_loss=0.008834, over 3040464.49 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:11:36,073 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:11:43,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2466166.6666666665, ans=0.0
2023-11-23 17:11:45,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2466166.6666666665, ans=0.0
2023-11-23 17:11:54,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2466166.6666666665, ans=0.2
2023-11-23 17:11:57,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2466233.3333333335, ans=0.125
2023-11-23 17:12:11,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2466300.0, ans=0.125
2023-11-23 17:12:13,758 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 369950
2023-11-23 17:12:20,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2466300.0, ans=0.125
2023-11-23 17:12:35,944 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9250, loss[loss=0.06433, simple_loss=0.0844, pruned_loss=0.01169, audio_tagging_loss=0.01044, over 15669.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.0921, pruned_loss=0.01368, audio_tagging_loss=0.008832, over 3050475.49 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:12:40,850 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.301e+01 8.925e+01 9.795e+01 1.226e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 17:12:51,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2466500.0, ans=0.5
2023-11-23 17:12:56,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2466500.0, ans=0.125
2023-11-23 17:13:10,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.67 vs. limit=22.5
2023-11-23 17:13:15,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2466633.3333333335, ans=0.125
2023-11-23 17:13:17,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2466633.3333333335, ans=0.125
2023-11-23 17:13:18,957 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370000
2023-11-23 17:13:22,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.72 vs. limit=15.0
2023-11-23 17:13:27,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2466700.0, ans=0.2
2023-11-23 17:13:29,651 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.75 vs. limit=12.0
2023-11-23 17:13:42,067 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9300, loss[loss=0.05662, simple_loss=0.07485, pruned_loss=0.009796, audio_tagging_loss=0.009403, over 16666.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09146, pruned_loss=0.01358, audio_tagging_loss=0.00889, over 3054545.01 frames. ], batch size: 64, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:13:55,583 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=7.00 vs. limit=12.0
2023-11-23 17:14:13,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2466900.0, ans=0.0
2023-11-23 17:14:20,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2466966.6666666665, ans=0.0
2023-11-23 17:14:21,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2466966.6666666665, ans=0.125
2023-11-23 17:14:25,611 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370050
2023-11-23 17:14:29,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2466966.6666666665, ans=0.125
2023-11-23 17:14:35,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2467033.3333333335, ans=0.0
2023-11-23 17:14:47,291 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9350, loss[loss=0.07947, simple_loss=0.1092, pruned_loss=0.01525, audio_tagging_loss=0.009636, over 14636.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09154, pruned_loss=0.01356, audio_tagging_loss=0.008878, over 3050396.24 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:14:52,707 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.132e+01 8.275e+01 9.065e+01 9.778e+01 1.188e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 17:14:53,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2467100.0, ans=0.1
2023-11-23 17:14:55,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2467100.0, ans=0.125
2023-11-23 17:15:26,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2467300.0, ans=0.0
2023-11-23 17:15:29,568 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370100
2023-11-23 17:15:50,785 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9400, loss[loss=0.04893, simple_loss=0.06338, pruned_loss=0.006417, audio_tagging_loss=0.01082, over 14888.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09142, pruned_loss=0.01348, audio_tagging_loss=0.008996, over 3051221.74 frames. ], batch size: 58, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:15:53,357 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.71 vs. limit=15.0
2023-11-23 17:16:07,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2467500.0, ans=0.0
2023-11-23 17:16:13,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2467500.0, ans=0.125
2023-11-23 17:16:19,888 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.95 vs. limit=15.0
2023-11-23 17:16:23,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2467566.6666666665, ans=0.2
2023-11-23 17:16:30,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2467633.3333333335, ans=0.125
2023-11-23 17:16:33,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370150
2023-11-23 17:16:33,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2467633.3333333335, ans=0.0
2023-11-23 17:16:38,575 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.36 vs. limit=12.0
2023-11-23 17:16:44,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2467700.0, ans=0.2
2023-11-23 17:16:53,404 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:16:54,588 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9450, loss[loss=0.05563, simple_loss=0.07535, pruned_loss=0.009218, audio_tagging_loss=0.008744, over 15161.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09231, pruned_loss=0.01365, audio_tagging_loss=0.009039, over 3055138.65 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:16:59,341 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.446e+01 8.380e+01 9.034e+01 1.002e+02 1.394e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 17:17:07,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2467833.3333333335, ans=0.125
2023-11-23 17:17:10,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2467833.3333333335, ans=0.05
2023-11-23 17:17:36,339 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370200
2023-11-23 17:17:58,780 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9500, loss[loss=0.05446, simple_loss=0.07314, pruned_loss=0.009708, audio_tagging_loss=0.008183, over 15369.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09197, pruned_loss=0.01376, audio_tagging_loss=0.009137, over 3050601.94 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:18:01,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2468100.0, ans=0.125
2023-11-23 17:18:04,986 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.65 vs. limit=15.0
2023-11-23 17:18:21,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2468166.6666666665, ans=0.1
2023-11-23 17:18:32,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2468233.3333333335, ans=0.0
2023-11-23 17:18:40,453 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370250
2023-11-23 17:18:55,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2468366.6666666665, ans=0.2
2023-11-23 17:19:02,045 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9550, loss[loss=0.05188, simple_loss=0.0645, pruned_loss=0.01035, audio_tagging_loss=0.009285, over 14511.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09179, pruned_loss=0.01382, audio_tagging_loss=0.009256, over 3057020.20 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:19:07,295 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.559e+01 9.097e+01 9.877e+01 1.537e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-23 17:19:32,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2468566.6666666665, ans=0.1
2023-11-23 17:19:37,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2468566.6666666665, ans=0.1
2023-11-23 17:19:40,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2468633.3333333335, ans=0.125
2023-11-23 17:19:41,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2468633.3333333335, ans=0.125
2023-11-23 17:19:43,688 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370300
2023-11-23 17:19:58,966 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:20:02,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2468700.0, ans=0.125
2023-11-23 17:20:05,168 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9600, loss[loss=0.06943, simple_loss=0.09257, pruned_loss=0.01606, audio_tagging_loss=0.007085, over 14732.00 frames. ], tot_loss[loss=0.06943, simple_loss=0.09233, pruned_loss=0.01395, audio_tagging_loss=0.009316, over 3056983.45 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:20:09,442 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.19 vs. limit=22.5
2023-11-23 17:20:27,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2468833.3333333335, ans=0.125
2023-11-23 17:20:43,278 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.85 vs. limit=15.0
2023-11-23 17:20:44,586 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.78 vs. limit=22.5
2023-11-23 17:20:46,372 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370350
2023-11-23 17:21:02,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2469033.3333333335, ans=0.125
2023-11-23 17:21:06,831 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9650, loss[loss=0.06112, simple_loss=0.08453, pruned_loss=0.01096, audio_tagging_loss=0.007893, over 14871.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09262, pruned_loss=0.01401, audio_tagging_loss=0.009138, over 3052949.64 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:21:11,537 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.236e+01 8.814e+01 9.557e+01 1.553e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-23 17:21:22,469 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:21:48,757 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370400
2023-11-23 17:22:07,795 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.56 vs. limit=15.0
2023-11-23 17:22:10,464 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9700, loss[loss=0.08087, simple_loss=0.1043, pruned_loss=0.01929, audio_tagging_loss=0.009417, over 15120.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09225, pruned_loss=0.01398, audio_tagging_loss=0.009003, over 3052897.89 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:22:33,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2469500.0, ans=0.2
2023-11-23 17:22:52,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370450
2023-11-23 17:23:13,583 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9750, loss[loss=0.06516, simple_loss=0.09093, pruned_loss=0.01251, audio_tagging_loss=0.007185, over 14705.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09314, pruned_loss=0.0141, audio_tagging_loss=0.008844, over 3053888.26 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:23:20,164 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.276e+01 8.363e+01 8.980e+01 9.912e+01 1.548e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 17:23:51,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.03 vs. limit=12.0
2023-11-23 17:23:55,039 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370500
2023-11-23 17:24:16,010 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9800, loss[loss=0.07714, simple_loss=0.09459, pruned_loss=0.01926, audio_tagging_loss=0.01059, over 15371.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09322, pruned_loss=0.01408, audio_tagging_loss=0.008787, over 3052800.90 frames. ], batch size: 59, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:24:17,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.66 vs. limit=15.0
2023-11-23 17:24:21,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2470100.0, ans=0.125
2023-11-23 17:24:50,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2470233.3333333335, ans=0.0
2023-11-23 17:24:51,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2470233.3333333335, ans=0.0
2023-11-23 17:24:54,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2470300.0, ans=0.0
2023-11-23 17:24:55,994 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.63 vs. limit=15.0
2023-11-23 17:24:57,543 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370550
2023-11-23 17:25:01,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2470300.0, ans=0.2
2023-11-23 17:25:03,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2470300.0, ans=0.125
2023-11-23 17:25:10,345 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:25:18,015 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9850, loss[loss=0.09773, simple_loss=0.1329, pruned_loss=0.02381, audio_tagging_loss=0.007464, over 15577.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09411, pruned_loss=0.01417, audio_tagging_loss=0.00876, over 3053484.30 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:25:24,369 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.399e+01 8.482e+01 9.119e+01 9.855e+01 1.403e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-23 17:25:27,312 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.95 vs. limit=15.0
2023-11-23 17:25:30,012 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=7.68 vs. limit=15.0
2023-11-23 17:25:34,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2470500.0, ans=0.125
2023-11-23 17:25:37,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2470500.0, ans=0.0
2023-11-23 17:25:40,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2470500.0, ans=0.05
2023-11-23 17:25:44,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2470566.6666666665, ans=0.125
2023-11-23 17:25:50,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2470566.6666666665, ans=0.04949747468305833
2023-11-23 17:25:58,181 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370600
2023-11-23 17:26:20,138 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9900, loss[loss=0.06758, simple_loss=0.08755, pruned_loss=0.01283, audio_tagging_loss=0.01098, over 14134.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09412, pruned_loss=0.01417, audio_tagging_loss=0.008801, over 3052294.36 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:26:26,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2470766.6666666665, ans=0.125
2023-11-23 17:26:30,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2470766.6666666665, ans=0.2
2023-11-23 17:26:55,134 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.42 vs. limit=10.0
2023-11-23 17:26:56,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2470966.6666666665, ans=0.125
2023-11-23 17:27:00,878 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370650
2023-11-23 17:27:05,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2470966.6666666665, ans=15.0
2023-11-23 17:27:08,015 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.18 vs. limit=15.0
2023-11-23 17:27:22,252 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 9950, loss[loss=0.08259, simple_loss=0.1192, pruned_loss=0.01493, audio_tagging_loss=0.008029, over 15317.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.0935, pruned_loss=0.01408, audio_tagging_loss=0.008853, over 3043478.24 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:27:27,529 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0
2023-11-23 17:27:27,746 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.44 vs. limit=15.0
2023-11-23 17:27:28,090 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.973e+01 8.205e+01 8.691e+01 9.411e+01 1.200e+02, threshold=1.738e+02, percent-clipped=0.0
2023-11-23 17:27:30,989 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.13 vs. limit=6.0
2023-11-23 17:27:52,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2471233.3333333335, ans=0.04949747468305833
2023-11-23 17:28:03,493 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370700
2023-11-23 17:28:07,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2471300.0, ans=0.125
2023-11-23 17:28:24,115 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10000, loss[loss=0.06253, simple_loss=0.08363, pruned_loss=0.01213, audio_tagging_loss=0.008578, over 15204.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09378, pruned_loss=0.01403, audio_tagging_loss=0.008745, over 3043431.78 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:28:24,857 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.43 vs. limit=15.0
2023-11-23 17:28:33,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2471433.3333333335, ans=0.2
2023-11-23 17:28:37,214 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.59 vs. limit=15.0
2023-11-23 17:28:38,633 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:28:48,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2471566.6666666665, ans=0.0
2023-11-23 17:28:50,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2471566.6666666665, ans=0.125
2023-11-23 17:29:00,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2471633.3333333335, ans=0.2
2023-11-23 17:29:01,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2471633.3333333335, ans=0.0
2023-11-23 17:29:05,005 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370750
2023-11-23 17:29:26,672 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10050, loss[loss=0.08708, simple_loss=0.1096, pruned_loss=0.02127, audio_tagging_loss=0.011, over 14886.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09215, pruned_loss=0.01376, audio_tagging_loss=0.008901, over 3043553.89 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:29:28,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2471766.6666666665, ans=0.2
2023-11-23 17:29:31,952 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.54 vs. limit=22.5
2023-11-23 17:29:33,089 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.306e+01 9.076e+01 9.554e+01 1.175e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 17:29:47,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2471833.3333333335, ans=0.1
2023-11-23 17:29:48,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2471833.3333333335, ans=0.0
2023-11-23 17:30:06,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370800
2023-11-23 17:30:15,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2472033.3333333335, ans=0.125
2023-11-23 17:30:21,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2472033.3333333335, ans=0.2
2023-11-23 17:30:28,777 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10100, loss[loss=0.06236, simple_loss=0.07683, pruned_loss=0.0135, audio_tagging_loss=0.01045, over 14919.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09155, pruned_loss=0.01362, audio_tagging_loss=0.009037, over 3043642.75 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:30:38,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2472100.0, ans=0.035
2023-11-23 17:30:46,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2472166.6666666665, ans=0.0
2023-11-23 17:30:57,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2472233.3333333335, ans=0.0
2023-11-23 17:31:03,728 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.68 vs. limit=15.0
2023-11-23 17:31:10,029 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370850
2023-11-23 17:31:13,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2472300.0, ans=0.0
2023-11-23 17:31:18,160 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:31:20,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2472366.6666666665, ans=0.1
2023-11-23 17:31:28,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2472433.3333333335, ans=0.0
2023-11-23 17:31:29,910 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10150, loss[loss=0.05687, simple_loss=0.08263, pruned_loss=0.006758, audio_tagging_loss=0.008796, over 14730.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09165, pruned_loss=0.01381, audio_tagging_loss=0.009172, over 3043744.46 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:31:36,356 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.103e+01 8.204e+01 9.033e+01 9.611e+01 1.160e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 17:31:56,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2472566.6666666665, ans=0.125
2023-11-23 17:31:59,353 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:32:05,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2472566.6666666665, ans=0.2
2023-11-23 17:32:05,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2472566.6666666665, ans=0.125
2023-11-23 17:32:11,338 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370900
2023-11-23 17:32:31,951 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10200, loss[loss=0.07574, simple_loss=0.09985, pruned_loss=0.01475, audio_tagging_loss=0.01107, over 14296.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09206, pruned_loss=0.01386, audio_tagging_loss=0.00919, over 3045889.19 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:32:55,247 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:33:13,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 370950
2023-11-23 17:33:17,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2472966.6666666665, ans=0.125
2023-11-23 17:33:35,208 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10250, loss[loss=0.07209, simple_loss=0.08499, pruned_loss=0.01913, audio_tagging_loss=0.01046, over 14496.00 frames. ], tot_loss[loss=0.06973, simple_loss=0.09287, pruned_loss=0.01408, audio_tagging_loss=0.009213, over 3048608.98 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:33:36,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2473100.0, ans=0.0
2023-11-23 17:33:42,416 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.022e+01 8.633e+01 9.319e+01 1.005e+02 1.656e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-23 17:33:51,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2473166.6666666665, ans=0.125
2023-11-23 17:34:06,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2473233.3333333335, ans=0.125
2023-11-23 17:34:12,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2473300.0, ans=0.125
2023-11-23 17:34:16,115 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371000
2023-11-23 17:34:28,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2473366.6666666665, ans=0.1
2023-11-23 17:34:30,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2473366.6666666665, ans=0.125
2023-11-23 17:34:33,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2473366.6666666665, ans=0.125
2023-11-23 17:34:36,430 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10300, loss[loss=0.0663, simple_loss=0.089, pruned_loss=0.01267, audio_tagging_loss=0.009128, over 15663.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09214, pruned_loss=0.01383, audio_tagging_loss=0.009244, over 3052138.47 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:34:49,320 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0
2023-11-23 17:34:55,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2473500.0, ans=0.125
2023-11-23 17:34:57,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2473500.0, ans=0.0
2023-11-23 17:35:08,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0
2023-11-23 17:35:11,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2473566.6666666665, ans=15.0
2023-11-23 17:35:12,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2473566.6666666665, ans=0.125
2023-11-23 17:35:16,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2473633.3333333335, ans=0.5
2023-11-23 17:35:17,953 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371050
2023-11-23 17:35:24,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2473633.3333333335, ans=0.125
2023-11-23 17:35:28,559 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.83 vs. limit=15.0
2023-11-23 17:35:38,909 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10350, loss[loss=0.04964, simple_loss=0.07043, pruned_loss=0.005889, audio_tagging_loss=0.008535, over 14455.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09248, pruned_loss=0.0138, audio_tagging_loss=0.009195, over 3045859.84 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:35:46,710 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.382e+01 9.122e+01 9.955e+01 1.360e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-23 17:36:17,701 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.00 vs. limit=10.0
2023-11-23 17:36:19,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2473966.6666666665, ans=0.0
2023-11-23 17:36:20,709 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371100
2023-11-23 17:36:21,240 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.80 vs. limit=22.5
2023-11-23 17:36:21,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2473966.6666666665, ans=0.1
2023-11-23 17:36:27,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2474033.3333333335, ans=0.125
2023-11-23 17:36:35,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2474033.3333333335, ans=0.0
2023-11-23 17:36:41,598 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10400, loss[loss=0.06575, simple_loss=0.08608, pruned_loss=0.01335, audio_tagging_loss=0.009366, over 14680.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09189, pruned_loss=0.01361, audio_tagging_loss=0.009267, over 3040740.24 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:36:44,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2474100.0, ans=0.125
2023-11-23 17:37:10,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2474233.3333333335, ans=0.1
2023-11-23 17:37:10,417 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.79 vs. limit=6.0
2023-11-23 17:37:12,922 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.20 vs. limit=15.0
2023-11-23 17:37:22,497 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371150
2023-11-23 17:37:28,638 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.20 vs. limit=12.0
2023-11-23 17:37:32,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2474366.6666666665, ans=15.0
2023-11-23 17:37:33,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2474366.6666666665, ans=0.1
2023-11-23 17:37:43,940 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10450, loss[loss=0.09043, simple_loss=0.1238, pruned_loss=0.02146, audio_tagging_loss=0.007059, over 14464.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.0919, pruned_loss=0.01362, audio_tagging_loss=0.009279, over 3037213.09 frames. ], batch size: 52, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:37:50,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2474433.3333333335, ans=0.125
2023-11-23 17:37:51,536 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.218e+01 8.367e+01 9.085e+01 9.738e+01 1.469e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 17:37:57,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2474500.0, ans=0.2
2023-11-23 17:38:03,074 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.02 vs. limit=10.0
2023-11-23 17:38:25,402 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371200
2023-11-23 17:38:46,221 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10500, loss[loss=0.06469, simple_loss=0.08866, pruned_loss=0.01333, audio_tagging_loss=0.00703, over 15937.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09144, pruned_loss=0.01359, audio_tagging_loss=0.009165, over 3041625.34 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:38:52,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2474766.6666666665, ans=0.0
2023-11-23 17:39:27,675 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371250
2023-11-23 17:39:30,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2474966.6666666665, ans=0.1
2023-11-23 17:39:44,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2475033.3333333335, ans=0.2
2023-11-23 17:39:48,779 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10550, loss[loss=0.06282, simple_loss=0.07916, pruned_loss=0.01261, audio_tagging_loss=0.01063, over 14609.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09152, pruned_loss=0.01353, audio_tagging_loss=0.009112, over 3033647.26 frames. ], batch size: 55, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:39:50,616 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.54 vs. limit=15.0
2023-11-23 17:39:53,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2475100.0, ans=0.0
2023-11-23 17:39:55,937 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.253e+01 8.407e+01 8.928e+01 9.553e+01 1.582e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-23 17:40:30,001 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371300
2023-11-23 17:40:33,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2475300.0, ans=0.125
2023-11-23 17:40:36,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2475300.0, ans=0.125
2023-11-23 17:40:48,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2475366.6666666665, ans=0.125
2023-11-23 17:40:50,998 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10600, loss[loss=0.09565, simple_loss=0.1322, pruned_loss=0.02026, audio_tagging_loss=0.009293, over 15673.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09062, pruned_loss=0.01343, audio_tagging_loss=0.009001, over 3034279.49 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:40:57,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2475433.3333333335, ans=0.0
2023-11-23 17:40:57,554 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.02 vs. limit=6.0
2023-11-23 17:41:01,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2475433.3333333335, ans=0.125
2023-11-23 17:41:09,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2475500.0, ans=0.125
2023-11-23 17:41:16,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.75 vs. limit=15.0
2023-11-23 17:41:20,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2475566.6666666665, ans=0.0
2023-11-23 17:41:31,694 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371350
2023-11-23 17:41:40,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2475700.0, ans=0.1
2023-11-23 17:41:53,207 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10650, loss[loss=0.06099, simple_loss=0.07789, pruned_loss=0.01264, audio_tagging_loss=0.009401, over 14401.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09132, pruned_loss=0.01364, audio_tagging_loss=0.009003, over 3039453.86 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:41:53,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2475766.6666666665, ans=10.0
2023-11-23 17:42:00,377 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.195e+01 8.706e+01 9.399e+01 1.158e+02, threshold=1.741e+02, percent-clipped=0.0
2023-11-23 17:42:03,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2475766.6666666665, ans=0.0
2023-11-23 17:42:04,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2475833.3333333335, ans=0.0
2023-11-23 17:42:04,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2475833.3333333335, ans=0.1
2023-11-23 17:42:22,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2475900.0, ans=0.2
2023-11-23 17:42:25,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2475900.0, ans=0.125
2023-11-23 17:42:34,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371400
2023-11-23 17:42:52,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2476033.3333333335, ans=0.125
2023-11-23 17:42:55,552 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10700, loss[loss=0.05852, simple_loss=0.07697, pruned_loss=0.0114, audio_tagging_loss=0.008639, over 16944.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09199, pruned_loss=0.01368, audio_tagging_loss=0.008947, over 3050921.86 frames. ], batch size: 66, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:43:18,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2476166.6666666665, ans=0.125
2023-11-23 17:43:19,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2476233.3333333335, ans=0.1
2023-11-23 17:43:20,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2476233.3333333335, ans=0.125
2023-11-23 17:43:36,887 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371450
2023-11-23 17:43:38,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2476300.0, ans=0.0
2023-11-23 17:43:40,404 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.12 vs. limit=15.0
2023-11-23 17:43:58,062 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10750, loss[loss=0.05428, simple_loss=0.06926, pruned_loss=0.009301, audio_tagging_loss=0.01035, over 15311.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09185, pruned_loss=0.01371, audio_tagging_loss=0.008968, over 3054209.65 frames. ], batch size: 60, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:44:05,725 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.639e+01 8.362e+01 8.891e+01 9.533e+01 1.093e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-23 17:44:20,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2476500.0, ans=0.04949747468305833
2023-11-23 17:44:22,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2476566.6666666665, ans=0.0
2023-11-23 17:44:34,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2476566.6666666665, ans=0.2
2023-11-23 17:44:39,961 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371500
2023-11-23 17:44:44,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2476633.3333333335, ans=0.0
2023-11-23 17:44:48,870 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.44 vs. limit=15.0
2023-11-23 17:45:01,416 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10800, loss[loss=0.09362, simple_loss=0.1276, pruned_loss=0.02265, audio_tagging_loss=0.007154, over 15016.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09142, pruned_loss=0.01352, audio_tagging_loss=0.009041, over 3055819.78 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 32.0
2023-11-23 17:45:06,689 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.99 vs. limit=15.0
2023-11-23 17:45:19,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.49 vs. limit=22.5
2023-11-23 17:45:22,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2476833.3333333335, ans=0.1
2023-11-23 17:45:41,924 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371550
2023-11-23 17:45:53,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2477033.3333333335, ans=0.125
2023-11-23 17:45:56,324 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.19 vs. limit=15.0
2023-11-23 17:45:58,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2477033.3333333335, ans=0.0
2023-11-23 17:46:03,196 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10850, loss[loss=0.07128, simple_loss=0.09085, pruned_loss=0.01824, audio_tagging_loss=0.007614, over 15456.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09119, pruned_loss=0.01344, audio_tagging_loss=0.008971, over 3056949.88 frames. ], batch size: 56, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:46:11,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2477100.0, ans=0.0
2023-11-23 17:46:14,351 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.439e+01 8.913e+01 9.439e+01 1.225e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 17:46:16,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2477166.6666666665, ans=0.0
2023-11-23 17:46:30,586 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.09 vs. limit=15.0
2023-11-23 17:46:43,584 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371600
2023-11-23 17:46:50,561 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:47:01,415 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:47:01,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2477366.6666666665, ans=0.125
2023-11-23 17:47:04,942 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10900, loss[loss=0.08294, simple_loss=0.1127, pruned_loss=0.0199, audio_tagging_loss=0.006692, over 14403.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09087, pruned_loss=0.01333, audio_tagging_loss=0.009073, over 3056810.82 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:47:10,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2477433.3333333335, ans=0.125
2023-11-23 17:47:27,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2477500.0, ans=0.2
2023-11-23 17:47:46,037 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371650
2023-11-23 17:47:58,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2477700.0, ans=0.125
2023-11-23 17:48:06,482 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 10950, loss[loss=0.05442, simple_loss=0.07919, pruned_loss=0.006795, audio_tagging_loss=0.008028, over 15379.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09075, pruned_loss=0.0132, audio_tagging_loss=0.009084, over 3054921.40 frames. ], batch size: 57, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:48:15,967 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.93 vs. limit=15.0
2023-11-23 17:48:17,613 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.148e+01 8.169e+01 8.798e+01 9.612e+01 4.443e+02, threshold=1.760e+02, percent-clipped=1.0
2023-11-23 17:48:25,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2477833.3333333335, ans=0.0
2023-11-23 17:48:35,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2477900.0, ans=0.1
2023-11-23 17:48:47,259 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371700
2023-11-23 17:49:01,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2478033.3333333335, ans=0.0
2023-11-23 17:49:08,795 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11000, loss[loss=0.06641, simple_loss=0.08795, pruned_loss=0.01182, audio_tagging_loss=0.01061, over 14758.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09119, pruned_loss=0.0134, audio_tagging_loss=0.009083, over 3055191.33 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:49:10,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2478100.0, ans=0.0
2023-11-23 17:49:18,886 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 17:49:39,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2478233.3333333335, ans=0.0
2023-11-23 17:49:49,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371750
2023-11-23 17:49:58,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2478366.6666666665, ans=0.2
2023-11-23 17:50:11,111 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11050, loss[loss=0.08295, simple_loss=0.1084, pruned_loss=0.01903, audio_tagging_loss=0.009716, over 14780.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09089, pruned_loss=0.01334, audio_tagging_loss=0.009138, over 3054185.01 frames. ], batch size: 53, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:50:21,701 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.456e+01 8.354e+01 9.034e+01 9.901e+01 1.113e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 17:50:26,372 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.04 vs. limit=6.0
2023-11-23 17:50:27,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2478500.0, ans=0.015
2023-11-23 17:50:39,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2478566.6666666665, ans=0.125
2023-11-23 17:50:52,527 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371800
2023-11-23 17:50:53,210 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.83 vs. limit=15.0
2023-11-23 17:50:59,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2478633.3333333335, ans=0.125
2023-11-23 17:51:04,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2478700.0, ans=0.125
2023-11-23 17:51:13,283 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11100, loss[loss=0.07123, simple_loss=0.08961, pruned_loss=0.01205, audio_tagging_loss=0.01438, over 14490.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09207, pruned_loss=0.0136, audio_tagging_loss=0.009194, over 3064829.90 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:51:40,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2478900.0, ans=0.125
2023-11-23 17:51:49,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2478966.6666666665, ans=0.07
2023-11-23 17:51:54,208 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371850
2023-11-23 17:51:54,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2478966.6666666665, ans=0.1
2023-11-23 17:52:07,226 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.63 vs. limit=15.0
2023-11-23 17:52:12,561 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.65 vs. limit=15.0
2023-11-23 17:52:15,293 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11150, loss[loss=0.04917, simple_loss=0.06318, pruned_loss=0.007252, audio_tagging_loss=0.01033, over 14377.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09183, pruned_loss=0.01358, audio_tagging_loss=0.009364, over 3066901.22 frames. ], batch size: 54, lr: 2.18e-03, grad_scale: 8.0
2023-11-23 17:52:16,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2479100.0, ans=0.125
2023-11-23 17:52:26,749 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.139e+01 8.417e+01 9.306e+01 9.835e+01 1.557e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-23 17:52:28,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2479166.6666666665, ans=0.125
2023-11-23 17:52:31,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2479166.6666666665, ans=0.0
2023-11-23 17:52:35,798 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.96 vs. limit=15.0
2023-11-23 17:52:51,242 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:52:55,912 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371900
2023-11-23 17:52:57,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2479300.0, ans=0.0
2023-11-23 17:53:09,411 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.88 vs. limit=12.0
2023-11-23 17:53:17,765 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11200, loss[loss=0.07361, simple_loss=0.1021, pruned_loss=0.01291, audio_tagging_loss=0.009658, over 16135.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09203, pruned_loss=0.01345, audio_tagging_loss=0.009374, over 3067962.11 frames. ], batch size: 61, lr: 2.18e-03, grad_scale: 16.0
2023-11-23 17:53:23,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2479433.3333333335, ans=0.125
2023-11-23 17:53:26,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2479433.3333333335, ans=0.125
2023-11-23 17:53:58,318 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 371950
2023-11-23 17:54:00,576 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.41 vs. limit=6.0
2023-11-23 17:54:08,929 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.48 vs. limit=12.0
2023-11-23 17:54:11,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2479700.0, ans=0.0
2023-11-23 17:54:19,193 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11250, loss[loss=0.07314, simple_loss=0.1034, pruned_loss=0.01239, audio_tagging_loss=0.009063, over 15210.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09119, pruned_loss=0.01333, audio_tagging_loss=0.009362, over 3061540.67 frames. ], batch size: 54, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 17:54:29,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2479766.6666666665, ans=0.125
2023-11-23 17:54:30,511 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.457e+01 8.293e+01 8.818e+01 9.873e+01 1.266e+02, threshold=1.764e+02, percent-clipped=0.0
2023-11-23 17:54:35,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.22 vs. limit=15.0
2023-11-23 17:54:40,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2479833.3333333335, ans=0.125
2023-11-23 17:55:00,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.42 vs. limit=15.0
2023-11-23 17:55:00,507 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372000
2023-11-23 17:55:01,935 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-372000.pt
2023-11-23 17:55:24,134 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11300, loss[loss=0.09859, simple_loss=0.1345, pruned_loss=0.02255, audio_tagging_loss=0.008793, over 15298.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.0917, pruned_loss=0.01336, audio_tagging_loss=0.009188, over 3053874.93 frames. ], batch size: 54, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:55:28,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2480100.0, ans=0.04949747468305833
2023-11-23 17:55:35,867 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.01 vs. limit=15.0
2023-11-23 17:55:51,790 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.29 vs. limit=6.0
2023-11-23 17:56:05,837 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372050
2023-11-23 17:56:10,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2480300.0, ans=0.125
2023-11-23 17:56:27,518 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11350, loss[loss=0.06721, simple_loss=0.08454, pruned_loss=0.01313, audio_tagging_loss=0.01181, over 14148.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09184, pruned_loss=0.01344, audio_tagging_loss=0.009113, over 3051589.45 frames. ], batch size: 57, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:56:39,395 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.419e+01 8.973e+01 9.627e+01 1.152e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-23 17:56:49,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2480500.0, ans=0.125
2023-11-23 17:56:50,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2480566.6666666665, ans=0.125
2023-11-23 17:57:08,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372100
2023-11-23 17:57:24,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2480700.0, ans=0.1
2023-11-23 17:57:28,754 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11400, loss[loss=0.07197, simple_loss=0.09361, pruned_loss=0.01653, audio_tagging_loss=0.008632, over 14433.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09224, pruned_loss=0.01352, audio_tagging_loss=0.008973, over 3047700.69 frames. ], batch size: 54, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:57:42,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2480833.3333333335, ans=0.0
2023-11-23 17:57:54,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2480900.0, ans=0.1
2023-11-23 17:58:02,836 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.99 vs. limit=22.5
2023-11-23 17:58:07,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2480966.6666666665, ans=0.04949747468305833
2023-11-23 17:58:09,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2480966.6666666665, ans=0.2
2023-11-23 17:58:10,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372150
2023-11-23 17:58:27,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2481033.3333333335, ans=0.0
2023-11-23 17:58:31,099 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11450, loss[loss=0.07712, simple_loss=0.1025, pruned_loss=0.01812, audio_tagging_loss=0.007738, over 15552.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09228, pruned_loss=0.01354, audio_tagging_loss=0.008894, over 3037878.14 frames. ], batch size: 57, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:58:39,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2481100.0, ans=0.125
2023-11-23 17:58:43,919 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.644e+01 8.130e+01 8.677e+01 9.441e+01 1.245e+02, threshold=1.735e+02, percent-clipped=0.0
2023-11-23 17:58:45,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2481166.6666666665, ans=0.0
2023-11-23 17:58:51,903 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 17:59:04,345 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.75 vs. limit=15.0
2023-11-23 17:59:12,564 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372200
2023-11-23 17:59:34,000 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11500, loss[loss=0.06462, simple_loss=0.08263, pruned_loss=0.01237, audio_tagging_loss=0.01093, over 14958.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09233, pruned_loss=0.01353, audio_tagging_loss=0.00898, over 3046281.77 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 17:59:46,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2481500.0, ans=0.0
2023-11-23 17:59:58,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2481566.6666666665, ans=0.0
2023-11-23 18:00:01,277 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.45 vs. limit=15.0
2023-11-23 18:00:14,400 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372250
2023-11-23 18:00:19,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2481633.3333333335, ans=0.025
2023-11-23 18:00:22,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2481700.0, ans=0.125
2023-11-23 18:00:35,661 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11550, loss[loss=0.06929, simple_loss=0.0976, pruned_loss=0.01211, audio_tagging_loss=0.008376, over 15558.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09234, pruned_loss=0.01352, audio_tagging_loss=0.008981, over 3052217.53 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 8.0
2023-11-23 18:00:48,169 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.328e+01 8.420e+01 9.006e+01 9.672e+01 1.345e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 18:00:53,275 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:00:59,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2481900.0, ans=0.125
2023-11-23 18:01:13,606 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:01:17,176 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372300
2023-11-23 18:01:20,012 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.92 vs. limit=15.0
2023-11-23 18:01:24,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2482033.3333333335, ans=0.05
2023-11-23 18:01:27,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2482033.3333333335, ans=0.2
2023-11-23 18:01:28,157 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.73 vs. limit=15.0
2023-11-23 18:01:28,450 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.61 vs. limit=10.0
2023-11-23 18:01:34,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2482033.3333333335, ans=0.0
2023-11-23 18:01:35,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2482033.3333333335, ans=0.1
2023-11-23 18:01:37,302 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11600, loss[loss=0.06219, simple_loss=0.07801, pruned_loss=0.01251, audio_tagging_loss=0.01068, over 14866.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09227, pruned_loss=0.01349, audio_tagging_loss=0.008976, over 3042780.95 frames. ], batch size: 59, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:01:37,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2482100.0, ans=0.1
2023-11-23 18:01:57,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2482166.6666666665, ans=0.125
2023-11-23 18:02:12,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=2482300.0, ans=0.2
2023-11-23 18:02:17,495 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372350
2023-11-23 18:02:31,189 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:02:38,551 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11650, loss[loss=0.06052, simple_loss=0.08345, pruned_loss=0.01001, audio_tagging_loss=0.008783, over 14835.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09287, pruned_loss=0.01362, audio_tagging_loss=0.009008, over 3046184.96 frames. ], batch size: 56, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:02:50,779 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.967e+01 8.302e+01 8.966e+01 9.575e+01 1.248e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 18:02:54,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2482500.0, ans=0.09899494936611666
2023-11-23 18:03:01,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2482566.6666666665, ans=0.1
2023-11-23 18:03:01,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2482566.6666666665, ans=0.2
2023-11-23 18:03:02,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2482566.6666666665, ans=0.125
2023-11-23 18:03:05,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2482566.6666666665, ans=0.1
2023-11-23 18:03:08,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2482566.6666666665, ans=0.125
2023-11-23 18:03:08,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2482566.6666666665, ans=0.0
2023-11-23 18:03:18,828 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372400
2023-11-23 18:03:18,985 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:03:40,992 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11700, loss[loss=0.05719, simple_loss=0.07308, pruned_loss=0.01131, audio_tagging_loss=0.00934, over 13973.00 frames. ], tot_loss[loss=0.06897, simple_loss=0.09246, pruned_loss=0.01367, audio_tagging_loss=0.009068, over 3044748.41 frames. ], batch size: 54, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:04:03,355 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.62 vs. limit=22.5
2023-11-23 18:04:22,552 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372450
2023-11-23 18:04:37,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2483033.3333333335, ans=0.0
2023-11-23 18:04:39,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2483033.3333333335, ans=0.0
2023-11-23 18:04:42,738 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11750, loss[loss=0.07061, simple_loss=0.1011, pruned_loss=0.0143, audio_tagging_loss=0.005736, over 14183.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09105, pruned_loss=0.01332, audio_tagging_loss=0.009163, over 3043292.45 frames. ], batch size: 54, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:04:47,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2483100.0, ans=0.125
2023-11-23 18:04:53,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2483166.6666666665, ans=0.1
2023-11-23 18:04:54,942 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.352e+01 9.061e+01 9.652e+01 1.174e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-23 18:05:07,892 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.75 vs. limit=22.5
2023-11-23 18:05:16,834 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-23 18:05:23,291 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372500
2023-11-23 18:05:25,515 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.49 vs. limit=15.0
2023-11-23 18:05:44,255 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11800, loss[loss=0.07022, simple_loss=0.09181, pruned_loss=0.0136, audio_tagging_loss=0.01071, over 14209.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09165, pruned_loss=0.01349, audio_tagging_loss=0.009176, over 3046006.62 frames. ], batch size: 55, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:05:48,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2483433.3333333335, ans=10.0
2023-11-23 18:05:53,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2483433.3333333335, ans=0.1
2023-11-23 18:06:10,117 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.22 vs. limit=15.0
2023-11-23 18:06:25,664 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372550
2023-11-23 18:06:25,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2483633.3333333335, ans=0.1
2023-11-23 18:06:38,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2483700.0, ans=0.125
2023-11-23 18:06:42,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2483700.0, ans=0.125
2023-11-23 18:06:46,839 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11850, loss[loss=0.07518, simple_loss=0.1017, pruned_loss=0.01784, audio_tagging_loss=0.006477, over 14168.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09087, pruned_loss=0.01339, audio_tagging_loss=0.009237, over 3041510.78 frames. ], batch size: 54, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:06:54,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2483766.6666666665, ans=0.2
2023-11-23 18:06:57,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2483766.6666666665, ans=0.125
2023-11-23 18:06:58,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2483833.3333333335, ans=0.125
2023-11-23 18:06:59,891 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.830e+01 8.390e+01 9.057e+01 9.896e+01 1.267e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 18:07:28,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372600
2023-11-23 18:07:33,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2483966.6666666665, ans=0.04949747468305833
2023-11-23 18:07:39,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2484033.3333333335, ans=0.2
2023-11-23 18:07:49,516 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11900, loss[loss=0.07158, simple_loss=0.08901, pruned_loss=0.01381, audio_tagging_loss=0.01327, over 16101.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.0906, pruned_loss=0.01328, audio_tagging_loss=0.009318, over 3040903.26 frames. ], batch size: 61, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:07:49,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2484100.0, ans=0.125
2023-11-23 18:07:51,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2484100.0, ans=0.1
2023-11-23 18:07:55,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2484100.0, ans=0.125
2023-11-23 18:08:11,295 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.13 vs. limit=15.0
2023-11-23 18:08:25,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2484300.0, ans=0.125
2023-11-23 18:08:30,679 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372650
2023-11-23 18:08:37,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2484300.0, ans=0.0
2023-11-23 18:08:51,648 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 11950, loss[loss=0.06541, simple_loss=0.0931, pruned_loss=0.01044, audio_tagging_loss=0.00842, over 15433.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09107, pruned_loss=0.01324, audio_tagging_loss=0.009295, over 3036701.89 frames. ], batch size: 60, lr: 2.17e-03, grad_scale: 16.0
2023-11-23 18:09:04,020 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.358e+01 9.149e+01 9.907e+01 1.513e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 18:09:04,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2484500.0, ans=0.1
2023-11-23 18:09:14,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2484500.0, ans=0.0
2023-11-23 18:09:25,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2484566.6666666665, ans=0.0
2023-11-23 18:09:31,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372700
2023-11-23 18:09:43,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2484700.0, ans=0.0
2023-11-23 18:09:50,721 INFO [train_asr.py:1221] (0/4) Epoch 31, batch 12000, loss[loss=0.07576, simple_loss=0.1014, pruned_loss=0.01712, audio_tagging_loss=0.007917, over 15118.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.091, pruned_loss=0.01331, audio_tagging_loss=0.009392, over 3042420.77 frames. ], batch size: 55, lr: 2.17e-03, grad_scale: 32.0
2023-11-23 18:09:50,724 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 18:10:31,555 INFO [train_asr.py:1253] (0/4) Epoch 31, validation: loss=0.05782, simple_loss=0.05111, pruned_loss=0.00519, audio_tagging_loss=0.02708, over 4681554.00 frames. 
2023-11-23 18:10:31,557 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 18:10:51,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.24 vs. limit=22.5
2023-11-23 18:10:59,724 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-31.pt
2023-11-23 18:11:32,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2484926.6666666665, ans=0.125
2023-11-23 18:11:33,931 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 0, loss[loss=0.07637, simple_loss=0.09142, pruned_loss=0.009722, audio_tagging_loss=0.02094, over 14229.00 frames. ], tot_loss[loss=0.07637, simple_loss=0.09142, pruned_loss=0.009722, audio_tagging_loss=0.02094, over 14229.00 frames. ], batch size: 55, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:11:33,934 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 18:12:07,797 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8073, 4.9181, 5.0732, 4.8898], device='cuda:0')
2023-11-23 18:12:09,602 INFO [train_asr.py:1253] (0/4) Epoch 32, validation: loss=0.05806, simple_loss=0.0511, pruned_loss=0.005184, audio_tagging_loss=0.02732, over 4681554.00 frames. 
2023-11-23 18:12:09,603 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 18:12:09,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2484926.6666666665, ans=0.125
2023-11-23 18:12:20,798 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372750
2023-11-23 18:12:24,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2484993.3333333335, ans=0.125
2023-11-23 18:12:26,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2484993.3333333335, ans=0.125
2023-11-23 18:12:32,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2484993.3333333335, ans=0.0
2023-11-23 18:12:32,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2484993.3333333335, ans=0.125
2023-11-23 18:12:32,604 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.37 vs. limit=15.0
2023-11-23 18:12:40,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2485060.0, ans=0.1
2023-11-23 18:12:49,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2485126.6666666665, ans=0.125
2023-11-23 18:12:52,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2485126.6666666665, ans=0.125
2023-11-23 18:12:52,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2485126.6666666665, ans=0.125
2023-11-23 18:12:54,212 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.111e+01 8.631e+01 9.416e+01 1.070e+02 1.612e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-23 18:13:11,376 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 50, loss[loss=0.08249, simple_loss=0.1047, pruned_loss=0.01608, audio_tagging_loss=0.01406, over 15883.00 frames. ], tot_loss[loss=0.07618, simple_loss=0.09072, pruned_loss=0.01341, audio_tagging_loss=0.0174, over 678911.06 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:13:15,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2485260.0, ans=0.2
2023-11-23 18:13:22,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372800
2023-11-23 18:13:32,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2485326.6666666665, ans=0.125
2023-11-23 18:13:37,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2485393.3333333335, ans=0.125
2023-11-23 18:13:53,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=23.94 vs. limit=15.0
2023-11-23 18:14:13,833 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 100, loss[loss=0.05524, simple_loss=0.06209, pruned_loss=0.007406, audio_tagging_loss=0.01679, over 15079.00 frames. ], tot_loss[loss=0.07516, simple_loss=0.09096, pruned_loss=0.01316, audio_tagging_loss=0.01651, over 1206595.42 frames. ], batch size: 56, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:14:25,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372850
2023-11-23 18:14:27,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2485660.0, ans=0.0
2023-11-23 18:14:28,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2485660.0, ans=0.1
2023-11-23 18:14:29,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.03 vs. limit=15.0
2023-11-23 18:14:39,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2485726.6666666665, ans=0.125
2023-11-23 18:14:45,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2485726.6666666665, ans=0.125
2023-11-23 18:14:59,367 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.698e+01 8.892e+01 9.428e+01 1.026e+02 1.429e+02, threshold=1.886e+02, percent-clipped=0.0
2023-11-23 18:15:00,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2485793.3333333335, ans=0.125
2023-11-23 18:15:09,953 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.07 vs. limit=15.0
2023-11-23 18:15:17,200 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 150, loss[loss=0.08078, simple_loss=0.1074, pruned_loss=0.01512, audio_tagging_loss=0.01194, over 14991.00 frames. ], tot_loss[loss=0.07441, simple_loss=0.09205, pruned_loss=0.01363, audio_tagging_loss=0.01476, over 1627430.27 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:15:21,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2485926.6666666665, ans=0.2
2023-11-23 18:15:28,467 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372900
2023-11-23 18:15:40,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=9.48 vs. limit=15.0
2023-11-23 18:15:57,217 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0
2023-11-23 18:16:18,937 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 200, loss[loss=0.05856, simple_loss=0.07546, pruned_loss=0.01095, audio_tagging_loss=0.009878, over 15123.00 frames. ], tot_loss[loss=0.07255, simple_loss=0.09106, pruned_loss=0.01382, audio_tagging_loss=0.0132, over 1943063.77 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:16:26,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2486260.0, ans=0.125
2023-11-23 18:16:30,298 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 372950
2023-11-23 18:16:46,945 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.72 vs. limit=10.0
2023-11-23 18:16:48,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2486393.3333333335, ans=0.5
2023-11-23 18:16:48,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2486393.3333333335, ans=0.125
2023-11-23 18:16:49,266 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.18 vs. limit=15.0
2023-11-23 18:17:04,694 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.397e+01 8.534e+01 9.090e+01 9.831e+01 1.209e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 18:17:07,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2486526.6666666665, ans=0.125
2023-11-23 18:17:20,652 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 250, loss[loss=0.06747, simple_loss=0.08609, pruned_loss=0.01422, audio_tagging_loss=0.0102, over 15665.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09248, pruned_loss=0.0139, audio_tagging_loss=0.01191, over 2186970.53 frames. ], batch size: 61, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:17:31,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373000
2023-11-23 18:18:02,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2486793.3333333335, ans=0.0
2023-11-23 18:18:09,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2486860.0, ans=0.2
2023-11-23 18:18:15,709 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:18:22,661 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 300, loss[loss=0.06821, simple_loss=0.1009, pruned_loss=0.0135, audio_tagging_loss=0.004249, over 14885.00 frames. ], tot_loss[loss=0.07096, simple_loss=0.09257, pruned_loss=0.0136, audio_tagging_loss=0.01107, over 2373418.53 frames. ], batch size: 54, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:18:24,683 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.63 vs. limit=15.0
2023-11-23 18:18:32,654 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.98 vs. limit=22.5
2023-11-23 18:18:34,645 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373050
2023-11-23 18:18:53,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2487060.0, ans=0.125
2023-11-23 18:19:08,698 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.767e+01 9.315e+01 9.987e+01 1.181e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-23 18:19:19,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.81 vs. limit=12.0
2023-11-23 18:19:24,691 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 350, loss[loss=0.06875, simple_loss=0.08994, pruned_loss=0.01536, audio_tagging_loss=0.00843, over 14256.00 frames. ], tot_loss[loss=0.07122, simple_loss=0.09361, pruned_loss=0.01387, audio_tagging_loss=0.01055, over 2526934.37 frames. ], batch size: 54, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:19:36,188 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373100
2023-11-23 18:19:39,408 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.43 vs. limit=12.0
2023-11-23 18:19:40,010 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:20:04,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2487460.0, ans=0.1
2023-11-23 18:20:07,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2487460.0, ans=10.0
2023-11-23 18:20:16,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2487526.6666666665, ans=0.0
2023-11-23 18:20:21,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2487526.6666666665, ans=0.0
2023-11-23 18:20:26,834 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 400, loss[loss=0.08347, simple_loss=0.1162, pruned_loss=0.01768, audio_tagging_loss=0.007678, over 16489.00 frames. ], tot_loss[loss=0.07044, simple_loss=0.09298, pruned_loss=0.01377, audio_tagging_loss=0.01017, over 2648550.60 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:20:37,975 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373150
2023-11-23 18:20:43,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.70 vs. limit=22.5
2023-11-23 18:20:49,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2487726.6666666665, ans=0.09899494936611666
2023-11-23 18:20:58,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2487726.6666666665, ans=0.2
2023-11-23 18:21:04,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2487793.3333333335, ans=0.2
2023-11-23 18:21:05,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2487793.3333333335, ans=0.035
2023-11-23 18:21:12,521 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.263e+01 8.510e+01 9.091e+01 9.716e+01 1.253e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 18:21:14,379 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.05 vs. limit=10.0
2023-11-23 18:21:16,702 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=15.0
2023-11-23 18:21:21,231 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:21:28,546 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 450, loss[loss=0.06119, simple_loss=0.07175, pruned_loss=0.01477, audio_tagging_loss=0.01055, over 15910.00 frames. ], tot_loss[loss=0.07047, simple_loss=0.09342, pruned_loss=0.01381, audio_tagging_loss=0.009945, over 2730489.45 frames. ], batch size: 62, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:21:40,009 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373200
2023-11-23 18:21:48,308 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.11 vs. limit=15.0
2023-11-23 18:21:59,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2488060.0, ans=0.0
2023-11-23 18:22:14,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2488126.6666666665, ans=0.125
2023-11-23 18:22:31,188 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 500, loss[loss=0.05534, simple_loss=0.06575, pruned_loss=0.01197, audio_tagging_loss=0.0105, over 16594.00 frames. ], tot_loss[loss=0.07019, simple_loss=0.09334, pruned_loss=0.0138, audio_tagging_loss=0.009712, over 2802330.49 frames. ], batch size: 63, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:22:35,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2488260.0, ans=10.0
2023-11-23 18:22:42,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373250
2023-11-23 18:22:56,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2488393.3333333335, ans=0.0
2023-11-23 18:23:00,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2488393.3333333335, ans=0.07
2023-11-23 18:23:04,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2488393.3333333335, ans=0.125
2023-11-23 18:23:07,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2488460.0, ans=0.125
2023-11-23 18:23:17,399 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.516e+01 9.137e+01 9.960e+01 1.270e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-23 18:23:25,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2488526.6666666665, ans=0.125
2023-11-23 18:23:34,148 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 550, loss[loss=0.06883, simple_loss=0.08935, pruned_loss=0.01497, audio_tagging_loss=0.009183, over 13670.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09212, pruned_loss=0.01359, audio_tagging_loss=0.009561, over 2852421.11 frames. ], batch size: 54, lr: 2.14e-03, grad_scale: 32.0
2023-11-23 18:23:44,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373300
2023-11-23 18:23:46,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2488660.0, ans=0.125
2023-11-23 18:23:50,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2488660.0, ans=0.0
2023-11-23 18:23:56,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2488660.0, ans=0.2
2023-11-23 18:24:14,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2488793.3333333335, ans=0.125
2023-11-23 18:24:15,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2488793.3333333335, ans=0.0
2023-11-23 18:24:17,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2488793.3333333335, ans=0.1
2023-11-23 18:24:20,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2488793.3333333335, ans=0.1
2023-11-23 18:24:22,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2488860.0, ans=0.0
2023-11-23 18:24:36,002 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 600, loss[loss=0.06012, simple_loss=0.08056, pruned_loss=0.01197, audio_tagging_loss=0.007864, over 14409.00 frames. ], tot_loss[loss=0.06969, simple_loss=0.09334, pruned_loss=0.0136, audio_tagging_loss=0.009419, over 2905134.24 frames. ], batch size: 54, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:24:38,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2488926.6666666665, ans=0.125
2023-11-23 18:24:46,746 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373350
2023-11-23 18:25:04,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2489060.0, ans=0.125
2023-11-23 18:25:05,602 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.60 vs. limit=12.0
2023-11-23 18:25:11,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2489060.0, ans=0.125
2023-11-23 18:25:11,379 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.02 vs. limit=22.5
2023-11-23 18:25:22,635 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.85 vs. limit=15.0
2023-11-23 18:25:23,229 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.157e+01 8.713e+01 9.332e+01 1.014e+02 1.298e+02, threshold=1.866e+02, percent-clipped=0.0
2023-11-23 18:25:38,136 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 650, loss[loss=0.05362, simple_loss=0.06752, pruned_loss=0.009854, audio_tagging_loss=0.01, over 15386.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09103, pruned_loss=0.01328, audio_tagging_loss=0.009441, over 2934421.31 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:25:49,564 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373400
2023-11-23 18:25:52,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.55 vs. limit=22.5
2023-11-23 18:25:53,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2489326.6666666665, ans=0.0
2023-11-23 18:26:13,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2489393.3333333335, ans=0.125
2023-11-23 18:26:14,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2489460.0, ans=0.0
2023-11-23 18:26:18,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2489460.0, ans=0.125
2023-11-23 18:26:21,249 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.55 vs. limit=10.0
2023-11-23 18:26:38,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2489526.6666666665, ans=0.2
2023-11-23 18:26:41,069 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 700, loss[loss=0.07561, simple_loss=0.1041, pruned_loss=0.01496, audio_tagging_loss=0.008584, over 17197.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09088, pruned_loss=0.01318, audio_tagging_loss=0.009454, over 2965412.84 frames. ], batch size: 62, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:26:46,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2489593.3333333335, ans=0.1
2023-11-23 18:26:51,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373450
2023-11-23 18:26:57,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2489660.0, ans=0.125
2023-11-23 18:27:10,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2489726.6666666665, ans=0.09899494936611666
2023-11-23 18:27:17,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2489793.3333333335, ans=0.1
2023-11-23 18:27:20,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2489793.3333333335, ans=0.125
2023-11-23 18:27:23,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2489793.3333333335, ans=0.025
2023-11-23 18:27:29,303 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.142e+01 8.089e+01 8.956e+01 9.594e+01 1.307e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 18:27:32,538 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.03 vs. limit=12.0
2023-11-23 18:27:42,838 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 750, loss[loss=0.05343, simple_loss=0.07315, pruned_loss=0.008606, audio_tagging_loss=0.008254, over 15178.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09173, pruned_loss=0.0133, audio_tagging_loss=0.009312, over 2991832.13 frames. ], batch size: 58, lr: 2.14e-03, grad_scale: 8.0
2023-11-23 18:27:53,790 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373500
2023-11-23 18:28:02,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2489993.3333333335, ans=0.0
2023-11-23 18:28:13,080 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.64 vs. limit=15.0
2023-11-23 18:28:15,819 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:28:19,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2490126.6666666665, ans=0.125
2023-11-23 18:28:29,462 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.09 vs. limit=15.0
2023-11-23 18:28:36,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2490193.3333333335, ans=0.125
2023-11-23 18:28:45,058 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 800, loss[loss=0.0673, simple_loss=0.08632, pruned_loss=0.01307, audio_tagging_loss=0.01107, over 13960.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09223, pruned_loss=0.01345, audio_tagging_loss=0.009285, over 3001761.21 frames. ], batch size: 54, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:28:46,859 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.55 vs. limit=15.0
2023-11-23 18:28:49,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2490260.0, ans=0.125
2023-11-23 18:28:55,845 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373550
2023-11-23 18:29:04,729 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.45 vs. limit=22.5
2023-11-23 18:29:13,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2490393.3333333335, ans=0.125
2023-11-23 18:29:18,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2490393.3333333335, ans=0.125
2023-11-23 18:29:19,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2490393.3333333335, ans=0.1
2023-11-23 18:29:28,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2490460.0, ans=0.125
2023-11-23 18:29:33,493 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.422e+01 8.967e+01 9.780e+01 1.208e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 18:29:40,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2490526.6666666665, ans=0.0
2023-11-23 18:29:47,000 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 850, loss[loss=0.07288, simple_loss=0.09619, pruned_loss=0.01389, audio_tagging_loss=0.0109, over 14829.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09148, pruned_loss=0.01328, audio_tagging_loss=0.009418, over 3016722.00 frames. ], batch size: 56, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:29:58,484 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373600
2023-11-23 18:30:02,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2490660.0, ans=0.2
2023-11-23 18:30:27,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.96 vs. limit=15.0
2023-11-23 18:30:29,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2490793.3333333335, ans=0.0
2023-11-23 18:30:39,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2490860.0, ans=0.0
2023-11-23 18:30:47,855 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.54 vs. limit=22.5
2023-11-23 18:30:49,690 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 900, loss[loss=0.06416, simple_loss=0.08444, pruned_loss=0.01213, audio_tagging_loss=0.009804, over 15379.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09184, pruned_loss=0.01337, audio_tagging_loss=0.009456, over 3023452.84 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:31:01,138 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373650
2023-11-23 18:31:13,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2491060.0, ans=0.1
2023-11-23 18:31:37,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2491126.6666666665, ans=0.0
2023-11-23 18:31:38,017 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.169e+01 8.291e+01 8.989e+01 9.605e+01 1.191e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 18:31:40,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2491193.3333333335, ans=0.125
2023-11-23 18:31:51,784 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 950, loss[loss=0.07199, simple_loss=0.1059, pruned_loss=0.009417, audio_tagging_loss=0.009629, over 15534.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09125, pruned_loss=0.01342, audio_tagging_loss=0.009478, over 3025018.57 frames. ], batch size: 57, lr: 2.14e-03, grad_scale: 16.0
2023-11-23 18:31:53,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2491260.0, ans=0.125
2023-11-23 18:32:02,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2491260.0, ans=0.125
2023-11-23 18:32:03,036 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373700
2023-11-23 18:32:06,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2491326.6666666665, ans=0.05
2023-11-23 18:32:07,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2491326.6666666665, ans=0.1
2023-11-23 18:32:47,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2491526.6666666665, ans=0.0
2023-11-23 18:32:51,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2491526.6666666665, ans=0.015
2023-11-23 18:32:53,765 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1000, loss[loss=0.06141, simple_loss=0.07907, pruned_loss=0.008599, audio_tagging_loss=0.01327, over 16997.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09056, pruned_loss=0.01334, audio_tagging_loss=0.009333, over 3032601.05 frames. ], batch size: 65, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:33:05,166 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373750
2023-11-23 18:33:06,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2491660.0, ans=0.125
2023-11-23 18:33:09,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff2.min_abs, batch_count=2491660.0, ans=0.1
2023-11-23 18:33:14,214 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.81 vs. limit=6.0
2023-11-23 18:33:17,181 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:33:20,543 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:33:24,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2491726.6666666665, ans=0.125
2023-11-23 18:33:35,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2491793.3333333335, ans=0.95
2023-11-23 18:33:42,322 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.226e+01 8.316e+01 8.954e+01 9.512e+01 1.152e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 18:33:56,553 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1050, loss[loss=0.04927, simple_loss=0.06045, pruned_loss=0.006099, audio_tagging_loss=0.01295, over 16363.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09006, pruned_loss=0.01323, audio_tagging_loss=0.009275, over 3037416.79 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:34:07,879 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373800
2023-11-23 18:34:09,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2491993.3333333335, ans=0.125
2023-11-23 18:34:09,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2491993.3333333335, ans=0.125
2023-11-23 18:34:25,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2492060.0, ans=0.0
2023-11-23 18:34:33,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2492126.6666666665, ans=0.125
2023-11-23 18:34:36,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2492126.6666666665, ans=0.0
2023-11-23 18:34:38,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2492126.6666666665, ans=0.1
2023-11-23 18:34:44,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2492126.6666666665, ans=0.125
2023-11-23 18:34:59,574 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1100, loss[loss=0.06976, simple_loss=0.09185, pruned_loss=0.01405, audio_tagging_loss=0.009786, over 14101.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09136, pruned_loss=0.01354, audio_tagging_loss=0.009192, over 3038614.94 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:35:02,540 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:35:08,118 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.04 vs. limit=15.0
2023-11-23 18:35:10,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373850
2023-11-23 18:35:41,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.35 vs. limit=12.0
2023-11-23 18:35:48,091 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.110e+01 8.335e+01 9.013e+01 9.723e+01 1.742e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-23 18:36:01,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2492593.3333333335, ans=0.2
2023-11-23 18:36:02,015 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1150, loss[loss=0.08076, simple_loss=0.109, pruned_loss=0.02038, audio_tagging_loss=0.005906, over 15861.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09145, pruned_loss=0.01362, audio_tagging_loss=0.00903, over 3038479.72 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:36:04,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2492593.3333333335, ans=0.125
2023-11-23 18:36:07,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2492593.3333333335, ans=0.0
2023-11-23 18:36:13,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373900
2023-11-23 18:36:41,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2492793.3333333335, ans=0.1
2023-11-23 18:36:50,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2492860.0, ans=0.125
2023-11-23 18:36:58,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2492860.0, ans=0.125
2023-11-23 18:36:58,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2492860.0, ans=0.0
2023-11-23 18:36:58,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2492860.0, ans=0.07
2023-11-23 18:37:04,185 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1200, loss[loss=0.06189, simple_loss=0.07675, pruned_loss=0.01574, audio_tagging_loss=0.00777, over 14337.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09148, pruned_loss=0.01361, audio_tagging_loss=0.008983, over 3037655.33 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:37:16,259 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 373950
2023-11-23 18:37:22,734 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.27 vs. limit=15.0
2023-11-23 18:37:45,318 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.82 vs. limit=6.0
2023-11-23 18:37:49,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2493126.6666666665, ans=0.125
2023-11-23 18:37:52,690 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.832e+01 8.465e+01 8.983e+01 9.576e+01 1.363e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 18:37:56,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2493193.3333333335, ans=0.1
2023-11-23 18:38:06,965 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1250, loss[loss=0.07283, simple_loss=0.1062, pruned_loss=0.01126, audio_tagging_loss=0.008489, over 15656.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09168, pruned_loss=0.01371, audio_tagging_loss=0.008898, over 3030998.79 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:38:15,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2493260.0, ans=10.0
2023-11-23 18:38:18,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374000
2023-11-23 18:38:24,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2493326.6666666665, ans=0.125
2023-11-23 18:38:32,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2493393.3333333335, ans=0.0
2023-11-23 18:38:43,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2493460.0, ans=0.125
2023-11-23 18:38:51,973 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.29 vs. limit=15.0
2023-11-23 18:39:00,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2493526.6666666665, ans=0.1
2023-11-23 18:39:07,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2493526.6666666665, ans=10.0
2023-11-23 18:39:09,330 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1300, loss[loss=0.05361, simple_loss=0.07864, pruned_loss=0.007032, audio_tagging_loss=0.00726, over 15159.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09155, pruned_loss=0.01364, audio_tagging_loss=0.008889, over 3033351.45 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:39:10,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2493593.3333333335, ans=0.1
2023-11-23 18:39:19,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2493593.3333333335, ans=0.125
2023-11-23 18:39:20,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374050
2023-11-23 18:39:29,291 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.75 vs. limit=22.5
2023-11-23 18:39:30,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2493660.0, ans=0.2
2023-11-23 18:39:43,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2493726.6666666665, ans=0.0
2023-11-23 18:39:59,215 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.307e+01 8.471e+01 8.997e+01 9.654e+01 1.238e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 18:40:11,614 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1350, loss[loss=0.07911, simple_loss=0.09996, pruned_loss=0.01864, audio_tagging_loss=0.0105, over 14710.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.0912, pruned_loss=0.01355, audio_tagging_loss=0.008925, over 3035713.30 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:40:17,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2493926.6666666665, ans=0.125
2023-11-23 18:40:22,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374100
2023-11-23 18:40:31,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2493993.3333333335, ans=0.125
2023-11-23 18:40:34,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2493993.3333333335, ans=0.1
2023-11-23 18:40:36,315 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.79 vs. limit=15.0
2023-11-23 18:40:43,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2494060.0, ans=0.025
2023-11-23 18:40:44,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2494060.0, ans=0.1
2023-11-23 18:40:52,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2494126.6666666665, ans=0.1
2023-11-23 18:40:55,037 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:40:57,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2494126.6666666665, ans=0.5
2023-11-23 18:41:01,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2494193.3333333335, ans=0.2
2023-11-23 18:41:04,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2494193.3333333335, ans=0.125
2023-11-23 18:41:13,476 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1400, loss[loss=0.07669, simple_loss=0.102, pruned_loss=0.01603, audio_tagging_loss=0.009632, over 15167.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.0917, pruned_loss=0.01369, audio_tagging_loss=0.009016, over 3039722.70 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:41:24,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2494260.0, ans=0.1
2023-11-23 18:41:25,521 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374150
2023-11-23 18:41:30,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2494326.6666666665, ans=0.125
2023-11-23 18:41:34,460 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.27 vs. limit=10.0
2023-11-23 18:41:49,252 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.14 vs. limit=22.5
2023-11-23 18:41:54,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2494460.0, ans=0.125
2023-11-23 18:41:55,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2494460.0, ans=0.1
2023-11-23 18:42:03,221 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.059e+01 8.437e+01 8.832e+01 9.783e+01 1.644e+02, threshold=1.766e+02, percent-clipped=0.0
2023-11-23 18:42:10,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2494526.6666666665, ans=0.2
2023-11-23 18:42:15,818 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1450, loss[loss=0.06144, simple_loss=0.07805, pruned_loss=0.0118, audio_tagging_loss=0.01062, over 15135.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09187, pruned_loss=0.0137, audio_tagging_loss=0.009069, over 3045206.13 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:42:27,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374200
2023-11-23 18:42:30,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2494660.0, ans=0.125
2023-11-23 18:42:35,547 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.78 vs. limit=10.0
2023-11-23 18:42:50,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2494726.6666666665, ans=0.1
2023-11-23 18:43:14,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2494860.0, ans=0.125
2023-11-23 18:43:16,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2494860.0, ans=0.05
2023-11-23 18:43:16,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2494860.0, ans=0.0
2023-11-23 18:43:18,568 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1500, loss[loss=0.06163, simple_loss=0.07978, pruned_loss=0.01271, audio_tagging_loss=0.009028, over 15154.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09208, pruned_loss=0.01371, audio_tagging_loss=0.009138, over 3039874.93 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:43:29,890 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374250
2023-11-23 18:43:35,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2494993.3333333335, ans=0.1
2023-11-23 18:43:53,451 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.97 vs. limit=15.0
2023-11-23 18:44:08,640 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.534e+01 8.386e+01 9.077e+01 9.732e+01 1.149e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 18:44:11,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.37 vs. limit=15.0
2023-11-23 18:44:15,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.whiten.whitening_limit, batch_count=2495193.3333333335, ans=12.0
2023-11-23 18:44:21,226 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1550, loss[loss=0.05808, simple_loss=0.06468, pruned_loss=0.01139, audio_tagging_loss=0.01435, over 16196.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09184, pruned_loss=0.01376, audio_tagging_loss=0.009247, over 3036707.10 frames. ], batch size: 64, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 18:44:32,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374300
2023-11-23 18:44:34,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2495326.6666666665, ans=0.0
2023-11-23 18:44:36,446 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.65 vs. limit=15.0
2023-11-23 18:45:07,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2495460.0, ans=0.0
2023-11-23 18:45:07,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2495460.0, ans=0.125
2023-11-23 18:45:24,454 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1600, loss[loss=0.05684, simple_loss=0.08147, pruned_loss=0.008527, audio_tagging_loss=0.007578, over 14692.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09139, pruned_loss=0.01354, audio_tagging_loss=0.009346, over 3031362.39 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:45:33,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2495593.3333333335, ans=0.0
2023-11-23 18:45:35,057 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374350
2023-11-23 18:45:45,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2495660.0, ans=0.2
2023-11-23 18:45:47,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2495726.6666666665, ans=0.125
2023-11-23 18:45:53,597 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.75 vs. limit=22.5
2023-11-23 18:45:56,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2495726.6666666665, ans=0.0
2023-11-23 18:46:00,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2495793.3333333335, ans=0.125
2023-11-23 18:46:00,650 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.23 vs. limit=15.0
2023-11-23 18:46:05,264 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.92 vs. limit=15.0
2023-11-23 18:46:13,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2495860.0, ans=0.1
2023-11-23 18:46:14,058 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.840e+01 8.412e+01 9.043e+01 9.774e+01 1.173e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 18:46:25,943 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1650, loss[loss=0.06947, simple_loss=0.08797, pruned_loss=0.01296, audio_tagging_loss=0.01252, over 15882.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09165, pruned_loss=0.01358, audio_tagging_loss=0.009332, over 3032574.24 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:46:37,333 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374400
2023-11-23 18:46:40,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.41 vs. limit=12.0
2023-11-23 18:46:56,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2496060.0, ans=0.125
2023-11-23 18:47:17,229 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.74 vs. limit=15.0
2023-11-23 18:47:20,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2496193.3333333335, ans=0.125
2023-11-23 18:47:28,989 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1700, loss[loss=0.07505, simple_loss=0.09827, pruned_loss=0.01673, audio_tagging_loss=0.009182, over 15499.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09106, pruned_loss=0.0136, audio_tagging_loss=0.009332, over 3044779.15 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:47:32,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2496260.0, ans=0.1
2023-11-23 18:47:39,731 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374450
2023-11-23 18:47:47,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2496326.6666666665, ans=0.0
2023-11-23 18:47:56,652 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 18:48:18,894 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.196e+01 8.417e+01 9.193e+01 9.971e+01 1.264e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 18:48:23,178 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.47 vs. limit=6.0
2023-11-23 18:48:30,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2496593.3333333335, ans=0.0
2023-11-23 18:48:31,090 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1750, loss[loss=0.06063, simple_loss=0.08074, pruned_loss=0.01138, audio_tagging_loss=0.008879, over 15140.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.0915, pruned_loss=0.01361, audio_tagging_loss=0.009337, over 3044528.92 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:48:42,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374500
2023-11-23 18:48:46,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2496660.0, ans=0.0
2023-11-23 18:48:56,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2496726.6666666665, ans=0.1
2023-11-23 18:48:58,016 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.50 vs. limit=15.0
2023-11-23 18:49:01,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2496726.6666666665, ans=0.125
2023-11-23 18:49:02,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2496726.6666666665, ans=0.1
2023-11-23 18:49:14,225 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.18 vs. limit=22.5
2023-11-23 18:49:33,580 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1800, loss[loss=0.06944, simple_loss=0.09389, pruned_loss=0.01392, audio_tagging_loss=0.008578, over 14315.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09178, pruned_loss=0.01371, audio_tagging_loss=0.009215, over 3040808.46 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:49:45,328 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374550
2023-11-23 18:49:47,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2496993.3333333335, ans=0.0
2023-11-23 18:49:59,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2497060.0, ans=0.125
2023-11-23 18:50:23,587 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.228e+01 8.763e+01 9.519e+01 1.190e+02, threshold=1.753e+02, percent-clipped=0.0
2023-11-23 18:50:29,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2497193.3333333335, ans=0.125
2023-11-23 18:50:35,997 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1850, loss[loss=0.04617, simple_loss=0.05684, pruned_loss=0.006125, audio_tagging_loss=0.01162, over 15804.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09206, pruned_loss=0.01383, audio_tagging_loss=0.009165, over 3035990.69 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:50:47,275 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374600
2023-11-23 18:51:02,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2497393.3333333335, ans=0.1
2023-11-23 18:51:38,311 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.33 vs. limit=6.0
2023-11-23 18:51:38,706 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1900, loss[loss=0.04698, simple_loss=0.06223, pruned_loss=0.007239, audio_tagging_loss=0.008628, over 15555.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09117, pruned_loss=0.01354, audio_tagging_loss=0.009087, over 3041716.65 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:51:50,235 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374650
2023-11-23 18:51:51,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2497660.0, ans=0.1
2023-11-23 18:51:57,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2497660.0, ans=0.1
2023-11-23 18:52:06,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2497726.6666666665, ans=0.07
2023-11-23 18:52:20,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2497793.3333333335, ans=0.125
2023-11-23 18:52:29,656 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.323e+01 8.306e+01 8.793e+01 9.456e+01 1.356e+02, threshold=1.759e+02, percent-clipped=0.0
2023-11-23 18:52:41,815 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 1950, loss[loss=0.05517, simple_loss=0.0645, pruned_loss=0.01153, audio_tagging_loss=0.01139, over 14770.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09054, pruned_loss=0.01335, audio_tagging_loss=0.009022, over 3044155.01 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:52:53,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374700
2023-11-23 18:52:59,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2497993.3333333335, ans=0.0
2023-11-23 18:53:18,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2498060.0, ans=0.5
2023-11-23 18:53:18,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2498060.0, ans=0.125
2023-11-23 18:53:38,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2498193.3333333335, ans=0.125
2023-11-23 18:53:38,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2498193.3333333335, ans=0.5
2023-11-23 18:53:45,420 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2000, loss[loss=0.08759, simple_loss=0.1147, pruned_loss=0.02186, audio_tagging_loss=0.008364, over 15688.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09104, pruned_loss=0.01341, audio_tagging_loss=0.008987, over 3045201.51 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:53:48,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2498260.0, ans=0.0
2023-11-23 18:53:50,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2498260.0, ans=0.0
2023-11-23 18:53:56,655 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374750
2023-11-23 18:53:59,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2498326.6666666665, ans=0.125
2023-11-23 18:54:09,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2498393.3333333335, ans=0.0
2023-11-23 18:54:16,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=2498393.3333333335, ans=0.05
2023-11-23 18:54:22,956 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.19 vs. limit=15.0
2023-11-23 18:54:35,237 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.533e+01 8.441e+01 9.066e+01 9.690e+01 1.160e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 18:54:48,069 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2050, loss[loss=0.1061, simple_loss=0.1452, pruned_loss=0.02665, audio_tagging_loss=0.006892, over 15669.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09157, pruned_loss=0.01353, audio_tagging_loss=0.009089, over 3046135.89 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:54:56,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff3.min_abs, batch_count=2498593.3333333335, ans=0.2
2023-11-23 18:54:57,560 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.09 vs. limit=15.0
2023-11-23 18:54:59,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374800
2023-11-23 18:55:02,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2498660.0, ans=0.125
2023-11-23 18:55:36,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2498793.3333333335, ans=0.125
2023-11-23 18:55:37,853 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.67 vs. limit=15.0
2023-11-23 18:55:50,657 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2100, loss[loss=0.06081, simple_loss=0.07874, pruned_loss=0.01103, audio_tagging_loss=0.01041, over 14715.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09168, pruned_loss=0.01348, audio_tagging_loss=0.009088, over 3045542.28 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:56:01,945 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374850
2023-11-23 18:56:07,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.56 vs. limit=15.0
2023-11-23 18:56:30,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2499126.6666666665, ans=0.0
2023-11-23 18:56:39,351 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.85 vs. limit=15.0
2023-11-23 18:56:39,981 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.878e+01 8.622e+01 9.048e+01 9.766e+01 1.225e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 18:56:52,563 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2150, loss[loss=0.08062, simple_loss=0.1085, pruned_loss=0.017, audio_tagging_loss=0.009368, over 16223.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09318, pruned_loss=0.0137, audio_tagging_loss=0.008888, over 3043667.96 frames. ], batch size: 60, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:56:52,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2499260.0, ans=0.125
2023-11-23 18:56:59,529 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.23 vs. limit=22.5
2023-11-23 18:57:04,356 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374900
2023-11-23 18:57:08,388 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.37 vs. limit=6.0
2023-11-23 18:57:25,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2499393.3333333335, ans=0.015
2023-11-23 18:57:26,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2499393.3333333335, ans=0.015
2023-11-23 18:57:29,796 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 18:57:32,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2499460.0, ans=0.035
2023-11-23 18:57:32,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2499460.0, ans=0.0
2023-11-23 18:57:36,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2499460.0, ans=0.125
2023-11-23 18:57:37,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2499460.0, ans=0.2
2023-11-23 18:57:54,961 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2200, loss[loss=0.05194, simple_loss=0.06321, pruned_loss=0.008036, audio_tagging_loss=0.0123, over 15603.00 frames. ], tot_loss[loss=0.07004, simple_loss=0.09439, pruned_loss=0.01395, audio_tagging_loss=0.008895, over 3049096.58 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:57:57,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2499593.3333333335, ans=0.125
2023-11-23 18:58:06,044 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 374950
2023-11-23 18:58:12,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2499660.0, ans=0.125
2023-11-23 18:58:14,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2499660.0, ans=0.125
2023-11-23 18:58:44,342 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.234e+01 8.395e+01 9.074e+01 9.645e+01 1.441e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 18:58:56,709 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2250, loss[loss=0.06944, simple_loss=0.09052, pruned_loss=0.01402, audio_tagging_loss=0.01016, over 15566.00 frames. ], tot_loss[loss=0.06964, simple_loss=0.09374, pruned_loss=0.01384, audio_tagging_loss=0.008926, over 3050167.26 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 18:59:07,951 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375000
2023-11-23 18:59:12,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2499993.3333333335, ans=0.0
2023-11-23 18:59:12,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2499993.3333333335, ans=0.125
2023-11-23 18:59:33,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2500126.6666666665, ans=0.125
2023-11-23 18:59:47,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2500193.3333333335, ans=0.1
2023-11-23 18:59:48,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.22 vs. limit=15.0
2023-11-23 18:59:58,411 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2300, loss[loss=0.06189, simple_loss=0.08615, pruned_loss=0.01054, audio_tagging_loss=0.008278, over 14991.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09338, pruned_loss=0.01383, audio_tagging_loss=0.00899, over 3043963.01 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:00:10,254 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375050
2023-11-23 19:00:26,787 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.15 vs. limit=10.0
2023-11-23 19:00:33,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2500393.3333333335, ans=0.125
2023-11-23 19:00:47,718 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.536e+01 8.606e+01 9.166e+01 9.861e+01 1.207e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 19:00:50,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2500526.6666666665, ans=0.1
2023-11-23 19:00:52,509 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:00:54,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2500526.6666666665, ans=0.0
2023-11-23 19:01:00,776 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2350, loss[loss=0.07302, simple_loss=0.09769, pruned_loss=0.01579, audio_tagging_loss=0.00838, over 14947.00 frames. ], tot_loss[loss=0.06976, simple_loss=0.09341, pruned_loss=0.01398, audio_tagging_loss=0.009078, over 3043028.38 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:01:03,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.60 vs. limit=15.0
2023-11-23 19:01:05,765 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:01:12,045 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375100
2023-11-23 19:01:18,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2500660.0, ans=0.125
2023-11-23 19:01:20,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2500660.0, ans=0.125
2023-11-23 19:01:25,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.39 vs. limit=15.0
2023-11-23 19:01:40,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2500793.3333333335, ans=0.0
2023-11-23 19:02:02,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2500926.6666666665, ans=0.125
2023-11-23 19:02:02,492 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.97 vs. limit=22.5
2023-11-23 19:02:03,155 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2400, loss[loss=0.07266, simple_loss=0.09407, pruned_loss=0.01821, audio_tagging_loss=0.007408, over 15345.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09265, pruned_loss=0.01403, audio_tagging_loss=0.009154, over 3039115.28 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:02:09,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2500926.6666666665, ans=0.0
2023-11-23 19:02:13,925 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375150
2023-11-23 19:02:29,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2501060.0, ans=0.125
2023-11-23 19:02:33,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2501060.0, ans=0.125
2023-11-23 19:02:41,186 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.39 vs. limit=15.0
2023-11-23 19:02:53,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2501193.3333333335, ans=0.125
2023-11-23 19:02:54,392 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.019e+01 8.466e+01 8.908e+01 9.614e+01 2.076e+02, threshold=1.782e+02, percent-clipped=1.0
2023-11-23 19:03:05,707 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2450, loss[loss=0.08035, simple_loss=0.1043, pruned_loss=0.0168, audio_tagging_loss=0.01141, over 15484.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09237, pruned_loss=0.01399, audio_tagging_loss=0.009263, over 3034528.94 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:03:09,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.whiten.whitening_limit, batch_count=2501260.0, ans=12.0
2023-11-23 19:03:16,927 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375200
2023-11-23 19:03:19,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2501326.6666666665, ans=0.0
2023-11-23 19:03:43,722 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.44 vs. limit=12.0
2023-11-23 19:04:06,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2501593.3333333335, ans=0.125
2023-11-23 19:04:07,620 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2500, loss[loss=0.05853, simple_loss=0.07178, pruned_loss=0.01255, audio_tagging_loss=0.01009, over 15832.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09193, pruned_loss=0.01376, audio_tagging_loss=0.00933, over 3038826.84 frames. ], batch size: 62, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:04:09,805 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.96 vs. limit=6.0
2023-11-23 19:04:19,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375250
2023-11-23 19:04:25,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2501660.0, ans=0.125
2023-11-23 19:04:31,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2501726.6666666665, ans=0.125
2023-11-23 19:04:39,865 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.22 vs. limit=10.0
2023-11-23 19:04:50,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2501793.3333333335, ans=0.0
2023-11-23 19:04:52,217 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=17.65 vs. limit=22.5
2023-11-23 19:04:58,326 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=12.0
2023-11-23 19:05:01,028 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.221e+01 8.437e+01 9.142e+01 9.814e+01 1.303e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 19:05:10,395 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2550, loss[loss=0.05783, simple_loss=0.07403, pruned_loss=0.0131, audio_tagging_loss=0.007723, over 14747.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09187, pruned_loss=0.0139, audio_tagging_loss=0.009291, over 3047097.92 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:05:21,393 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375300
2023-11-23 19:06:01,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2502193.3333333335, ans=0.0
2023-11-23 19:06:12,339 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2600, loss[loss=0.05584, simple_loss=0.06884, pruned_loss=0.01081, audio_tagging_loss=0.01061, over 15132.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09148, pruned_loss=0.01394, audio_tagging_loss=0.009165, over 3043972.48 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:06:23,681 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375350
2023-11-23 19:06:26,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2502326.6666666665, ans=0.125
2023-11-23 19:06:46,542 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.54 vs. limit=15.0
2023-11-23 19:07:06,772 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.389e+01 8.376e+01 8.898e+01 9.967e+01 2.098e+02, threshold=1.780e+02, percent-clipped=2.0
2023-11-23 19:07:15,725 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2650, loss[loss=0.07599, simple_loss=0.1079, pruned_loss=0.01533, audio_tagging_loss=0.006714, over 14515.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09174, pruned_loss=0.0139, audio_tagging_loss=0.009128, over 3045770.51 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:07:15,964 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:07:26,397 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375400
2023-11-23 19:07:51,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2502793.3333333335, ans=0.5
2023-11-23 19:07:52,344 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.56 vs. limit=15.0
2023-11-23 19:07:55,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2502793.3333333335, ans=0.0
2023-11-23 19:07:57,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2502793.3333333335, ans=0.0
2023-11-23 19:08:03,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2502793.3333333335, ans=0.0
2023-11-23 19:08:11,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2502860.0, ans=0.0
2023-11-23 19:08:17,547 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2700, loss[loss=0.05224, simple_loss=0.05983, pruned_loss=0.00992, audio_tagging_loss=0.01241, over 14923.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09108, pruned_loss=0.0137, audio_tagging_loss=0.009074, over 3046726.44 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:08:27,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2502926.6666666665, ans=0.125
2023-11-23 19:08:28,754 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375450
2023-11-23 19:08:37,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2502993.3333333335, ans=0.2
2023-11-23 19:09:03,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2503126.6666666665, ans=0.125
2023-11-23 19:09:04,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2503126.6666666665, ans=0.05
2023-11-23 19:09:04,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2503126.6666666665, ans=0.0
2023-11-23 19:09:10,600 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.867e+01 8.343e+01 9.045e+01 9.938e+01 1.315e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 19:09:12,460 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.31 vs. limit=15.0
2023-11-23 19:09:18,828 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2750, loss[loss=0.06392, simple_loss=0.08405, pruned_loss=0.01347, audio_tagging_loss=0.008422, over 15548.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09086, pruned_loss=0.01361, audio_tagging_loss=0.00907, over 3048305.91 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 8.0
2023-11-23 19:09:23,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2503260.0, ans=0.125
2023-11-23 19:09:30,072 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375500
2023-11-23 19:10:11,504 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:10:20,913 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2800, loss[loss=0.06056, simple_loss=0.08923, pruned_loss=0.00924, audio_tagging_loss=0.006706, over 15214.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09107, pruned_loss=0.01355, audio_tagging_loss=0.009026, over 3045081.37 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:10:26,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2503593.3333333335, ans=0.0
2023-11-23 19:10:31,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2503593.3333333335, ans=10.0
2023-11-23 19:10:32,238 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375550
2023-11-23 19:10:33,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2503660.0, ans=0.0
2023-11-23 19:10:51,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.74 vs. limit=10.0
2023-11-23 19:10:53,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2503726.6666666665, ans=0.125
2023-11-23 19:10:54,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2503726.6666666665, ans=0.125
2023-11-23 19:11:02,579 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.61 vs. limit=6.0
2023-11-23 19:11:13,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2503860.0, ans=0.125
2023-11-23 19:11:14,366 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.252e+01 8.866e+01 9.555e+01 1.297e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 19:11:14,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2503860.0, ans=0.0
2023-11-23 19:11:14,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2503860.0, ans=0.0
2023-11-23 19:11:21,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2503926.6666666665, ans=0.1
2023-11-23 19:11:22,780 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2850, loss[loss=0.05961, simple_loss=0.07966, pruned_loss=0.01081, audio_tagging_loss=0.008964, over 14829.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09072, pruned_loss=0.01341, audio_tagging_loss=0.008991, over 3045176.51 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:11:34,023 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375600
2023-11-23 19:11:34,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2503993.3333333335, ans=0.0
2023-11-23 19:11:48,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2504060.0, ans=0.125
2023-11-23 19:11:51,785 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.03 vs. limit=15.0
2023-11-23 19:12:22,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2504193.3333333335, ans=0.0
2023-11-23 19:12:26,399 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2900, loss[loss=0.06995, simple_loss=0.09294, pruned_loss=0.01279, audio_tagging_loss=0.01069, over 13357.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09098, pruned_loss=0.01365, audio_tagging_loss=0.008934, over 3038334.28 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:12:30,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2504260.0, ans=0.0
2023-11-23 19:12:37,695 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375650
2023-11-23 19:12:49,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2504326.6666666665, ans=0.1
2023-11-23 19:12:54,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=2504393.3333333335, ans=10.0
2023-11-23 19:12:57,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2504393.3333333335, ans=0.2
2023-11-23 19:13:09,150 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.00 vs. limit=15.0
2023-11-23 19:13:12,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2504460.0, ans=0.0
2023-11-23 19:13:18,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2504526.6666666665, ans=0.125
2023-11-23 19:13:20,157 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.135e+01 8.540e+01 9.133e+01 9.789e+01 1.546e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-23 19:13:28,632 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 2950, loss[loss=0.06331, simple_loss=0.08629, pruned_loss=0.01009, audio_tagging_loss=0.01007, over 16699.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09193, pruned_loss=0.01371, audio_tagging_loss=0.008848, over 3032237.81 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:13:29,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2504593.3333333335, ans=0.125
2023-11-23 19:13:34,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2504593.3333333335, ans=0.0
2023-11-23 19:13:35,798 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.06 vs. limit=22.5
2023-11-23 19:13:37,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2504593.3333333335, ans=0.0
2023-11-23 19:13:40,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375700
2023-11-23 19:13:45,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2504660.0, ans=0.125
2023-11-23 19:13:52,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2504726.6666666665, ans=0.0
2023-11-23 19:14:04,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2504726.6666666665, ans=0.125
2023-11-23 19:14:13,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2504793.3333333335, ans=0.0
2023-11-23 19:14:26,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2504860.0, ans=0.125
2023-11-23 19:14:26,928 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.03 vs. limit=22.5
2023-11-23 19:14:31,241 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3000, loss[loss=0.06679, simple_loss=0.09164, pruned_loss=0.01078, audio_tagging_loss=0.01019, over 14624.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09155, pruned_loss=0.0137, audio_tagging_loss=0.008939, over 3033376.65 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:14:31,244 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 19:15:09,805 INFO [train_asr.py:1253] (0/4) Epoch 32, validation: loss=0.05818, simple_loss=0.05104, pruned_loss=0.005158, audio_tagging_loss=0.0275, over 4681554.00 frames. 
2023-11-23 19:15:09,806 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 19:15:21,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375750
2023-11-23 19:15:42,954 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.95 vs. limit=6.0
2023-11-23 19:15:43,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2505060.0, ans=0.125
2023-11-23 19:15:53,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2505126.6666666665, ans=0.0
2023-11-23 19:15:59,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2505193.3333333335, ans=0.0
2023-11-23 19:16:03,445 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.656e+01 9.141e+01 9.887e+01 1.235e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 19:16:11,773 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3050, loss[loss=0.07665, simple_loss=0.1041, pruned_loss=0.01648, audio_tagging_loss=0.008136, over 15445.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09209, pruned_loss=0.0138, audio_tagging_loss=0.008896, over 3037592.65 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:16:23,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375800
2023-11-23 19:16:38,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2505393.3333333335, ans=0.05
2023-11-23 19:16:46,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2505393.3333333335, ans=0.2
2023-11-23 19:16:48,182 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:16:53,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2505460.0, ans=0.0
2023-11-23 19:17:07,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2505526.6666666665, ans=0.125
2023-11-23 19:17:13,718 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3100, loss[loss=0.06661, simple_loss=0.08834, pruned_loss=0.01401, audio_tagging_loss=0.008424, over 16380.00 frames. ], tot_loss[loss=0.06931, simple_loss=0.09279, pruned_loss=0.01395, audio_tagging_loss=0.008964, over 3036024.63 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:17:25,021 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375850
2023-11-23 19:17:26,773 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.50 vs. limit=15.0
2023-11-23 19:17:37,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2505660.0, ans=0.0
2023-11-23 19:17:39,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2505726.6666666665, ans=0.0
2023-11-23 19:17:59,217 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.19 vs. limit=6.0
2023-11-23 19:18:07,375 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.481e+01 9.058e+01 9.503e+01 1.492e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-23 19:18:15,799 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3150, loss[loss=0.05858, simple_loss=0.08299, pruned_loss=0.01014, audio_tagging_loss=0.006942, over 14418.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09236, pruned_loss=0.01375, audio_tagging_loss=0.009063, over 3038182.89 frames. ], batch size: 53, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:18:18,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2505926.6666666665, ans=0.125
2023-11-23 19:18:18,407 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:18:22,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2505926.6666666665, ans=0.125
2023-11-23 19:18:27,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375900
2023-11-23 19:18:29,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2505993.3333333335, ans=0.1
2023-11-23 19:18:33,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.38 vs. limit=22.5
2023-11-23 19:18:33,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2505993.3333333335, ans=0.05
2023-11-23 19:18:35,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2505993.3333333335, ans=0.125
2023-11-23 19:18:41,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2506060.0, ans=0.2
2023-11-23 19:18:42,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2506060.0, ans=0.0
2023-11-23 19:19:18,401 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3200, loss[loss=0.06283, simple_loss=0.08031, pruned_loss=0.01218, audio_tagging_loss=0.0105, over 15156.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09325, pruned_loss=0.0139, audio_tagging_loss=0.009125, over 3045379.89 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:19:22,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2506260.0, ans=0.125
2023-11-23 19:19:23,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2506260.0, ans=0.125
2023-11-23 19:19:29,717 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 375950
2023-11-23 19:19:47,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.95 vs. limit=10.0
2023-11-23 19:19:48,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2506393.3333333335, ans=0.125
2023-11-23 19:20:04,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2506460.0, ans=0.125
2023-11-23 19:20:13,187 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.564e+01 8.152e+01 8.777e+01 9.495e+01 2.540e+02, threshold=1.755e+02, percent-clipped=1.0
2023-11-23 19:20:18,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2506526.6666666665, ans=0.0
2023-11-23 19:20:20,389 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3250, loss[loss=0.07957, simple_loss=0.09501, pruned_loss=0.01893, audio_tagging_loss=0.01314, over 14911.00 frames. ], tot_loss[loss=0.06965, simple_loss=0.09317, pruned_loss=0.01386, audio_tagging_loss=0.009199, over 3046718.30 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:20:31,839 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376000
2023-11-23 19:20:33,324 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-376000.pt
2023-11-23 19:20:42,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2506660.0, ans=0.125
2023-11-23 19:20:50,668 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.45 vs. limit=15.0
2023-11-23 19:21:03,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2506793.3333333335, ans=0.125
2023-11-23 19:21:05,988 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.77 vs. limit=6.0
2023-11-23 19:21:12,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2506860.0, ans=0.09899494936611666
2023-11-23 19:21:26,147 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3300, loss[loss=0.05544, simple_loss=0.06507, pruned_loss=0.006377, audio_tagging_loss=0.01653, over 14515.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09264, pruned_loss=0.01375, audio_tagging_loss=0.009315, over 3045413.45 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:21:26,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=2506926.6666666665, ans=22.5
2023-11-23 19:21:37,349 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376050
2023-11-23 19:22:20,310 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.632e+01 8.685e+01 9.326e+01 1.032e+02 1.222e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-23 19:22:24,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2507193.3333333335, ans=0.0
2023-11-23 19:22:28,037 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3350, loss[loss=0.08058, simple_loss=0.1065, pruned_loss=0.01973, audio_tagging_loss=0.007573, over 15176.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09195, pruned_loss=0.01363, audio_tagging_loss=0.009337, over 3047404.15 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:22:33,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2507260.0, ans=0.125
2023-11-23 19:22:39,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376100
2023-11-23 19:23:10,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2507460.0, ans=0.0
2023-11-23 19:23:12,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2507460.0, ans=10.0
2023-11-23 19:23:27,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2507526.6666666665, ans=0.125
2023-11-23 19:23:27,750 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.15 vs. limit=15.0
2023-11-23 19:23:30,757 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3400, loss[loss=0.07396, simple_loss=0.1068, pruned_loss=0.01415, audio_tagging_loss=0.00639, over 16246.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09228, pruned_loss=0.01364, audio_tagging_loss=0.009107, over 3053759.01 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:23:32,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2507593.3333333335, ans=0.1
2023-11-23 19:23:41,847 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376150
2023-11-23 19:23:49,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.16 vs. limit=15.0
2023-11-23 19:24:01,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2507726.6666666665, ans=0.0
2023-11-23 19:24:08,413 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.86 vs. limit=22.5
2023-11-23 19:24:13,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2507793.3333333335, ans=0.0
2023-11-23 19:24:16,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2507793.3333333335, ans=0.2
2023-11-23 19:24:24,966 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.627e+01 8.340e+01 8.680e+01 9.645e+01 1.165e+02, threshold=1.736e+02, percent-clipped=0.0
2023-11-23 19:24:29,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2507860.0, ans=0.1
2023-11-23 19:24:32,686 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3450, loss[loss=0.0702, simple_loss=0.09382, pruned_loss=0.01318, audio_tagging_loss=0.01011, over 16457.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09176, pruned_loss=0.0136, audio_tagging_loss=0.008954, over 3052625.38 frames. ], batch size: 61, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:24:36,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2507926.6666666665, ans=0.0
2023-11-23 19:24:39,267 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.03 vs. limit=15.0
2023-11-23 19:24:43,413 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376200
2023-11-23 19:25:12,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2508126.6666666665, ans=0.125
2023-11-23 19:25:28,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2508193.3333333335, ans=0.0
2023-11-23 19:25:34,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2508260.0, ans=0.125
2023-11-23 19:25:35,036 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3500, loss[loss=0.06329, simple_loss=0.08132, pruned_loss=0.0147, audio_tagging_loss=0.007932, over 14780.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09156, pruned_loss=0.0136, audio_tagging_loss=0.008924, over 3042359.00 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:25:40,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2508260.0, ans=0.125
2023-11-23 19:25:47,225 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376250
2023-11-23 19:25:48,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2508326.6666666665, ans=0.0
2023-11-23 19:26:04,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2508393.3333333335, ans=0.125
2023-11-23 19:26:06,578 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:26:10,934 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.16 vs. limit=22.5
2023-11-23 19:26:24,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2508526.6666666665, ans=0.125
2023-11-23 19:26:29,277 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.96 vs. limit=12.0
2023-11-23 19:26:29,755 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.989e+01 8.248e+01 9.005e+01 9.777e+01 1.238e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 19:26:38,184 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3550, loss[loss=0.06433, simple_loss=0.08869, pruned_loss=0.01273, audio_tagging_loss=0.007257, over 16108.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09112, pruned_loss=0.01359, audio_tagging_loss=0.008965, over 3037465.32 frames. ], batch size: 60, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:26:49,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376300
2023-11-23 19:27:01,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2508660.0, ans=0.125
2023-11-23 19:27:13,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2508726.6666666665, ans=0.0
2023-11-23 19:27:27,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2508860.0, ans=0.0
2023-11-23 19:27:31,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2508860.0, ans=0.0
2023-11-23 19:27:40,974 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3600, loss[loss=0.07264, simple_loss=0.09506, pruned_loss=0.01777, audio_tagging_loss=0.007345, over 14674.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09124, pruned_loss=0.01355, audio_tagging_loss=0.008911, over 3040460.93 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:27:41,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2508926.6666666665, ans=0.0
2023-11-23 19:27:45,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2508926.6666666665, ans=0.125
2023-11-23 19:27:49,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2508926.6666666665, ans=0.1
2023-11-23 19:27:51,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376350
2023-11-23 19:27:59,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2508993.3333333335, ans=0.125
2023-11-23 19:28:25,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.53 vs. limit=15.0
2023-11-23 19:28:26,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2509126.6666666665, ans=0.0
2023-11-23 19:28:35,123 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.359e+01 9.090e+01 9.890e+01 1.396e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 19:28:41,399 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:28:42,874 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3650, loss[loss=0.06631, simple_loss=0.09462, pruned_loss=0.009247, audio_tagging_loss=0.009749, over 16341.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09141, pruned_loss=0.01348, audio_tagging_loss=0.00877, over 3038637.13 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:28:54,292 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376400
2023-11-23 19:29:08,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2509393.3333333335, ans=0.2
2023-11-23 19:29:13,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2509393.3333333335, ans=0.0
2023-11-23 19:29:44,985 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3700, loss[loss=0.05651, simple_loss=0.06651, pruned_loss=0.01167, audio_tagging_loss=0.01159, over 15840.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09037, pruned_loss=0.01349, audio_tagging_loss=0.008848, over 3036740.77 frames. ], batch size: 60, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:29:56,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376450
2023-11-23 19:30:10,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2509726.6666666665, ans=0.125
2023-11-23 19:30:32,830 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.08 vs. limit=6.0
2023-11-23 19:30:40,600 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.564e+01 8.395e+01 9.094e+01 9.680e+01 1.215e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-23 19:30:48,260 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3750, loss[loss=0.03848, simple_loss=0.0413, pruned_loss=0.005566, audio_tagging_loss=0.01226, over 14303.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.0911, pruned_loss=0.01351, audio_tagging_loss=0.008822, over 3042592.05 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:30:48,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2509926.6666666665, ans=0.125
2023-11-23 19:30:56,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2509926.6666666665, ans=0.125
2023-11-23 19:30:59,312 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376500
2023-11-23 19:31:09,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2509993.3333333335, ans=0.0
2023-11-23 19:31:30,170 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.45 vs. limit=12.0
2023-11-23 19:31:30,740 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:31:50,359 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3800, loss[loss=0.03848, simple_loss=0.03954, pruned_loss=0.004959, audio_tagging_loss=0.01375, over 14352.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09025, pruned_loss=0.01336, audio_tagging_loss=0.00907, over 3042996.56 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:31:55,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2510260.0, ans=0.125
2023-11-23 19:32:01,469 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376550
2023-11-23 19:32:08,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2510326.6666666665, ans=0.125
2023-11-23 19:32:14,602 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.75 vs. limit=15.0
2023-11-23 19:32:27,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2510460.0, ans=0.0
2023-11-23 19:32:32,251 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.38 vs. limit=15.0
2023-11-23 19:32:42,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2510526.6666666665, ans=0.5
2023-11-23 19:32:45,286 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.658e+01 9.226e+01 1.003e+02 1.868e+02, threshold=1.845e+02, percent-clipped=1.0
2023-11-23 19:32:49,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2510526.6666666665, ans=0.2
2023-11-23 19:32:52,904 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3850, loss[loss=0.1073, simple_loss=0.1538, pruned_loss=0.02501, audio_tagging_loss=0.005434, over 14565.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09057, pruned_loss=0.01343, audio_tagging_loss=0.009119, over 3036649.70 frames. ], batch size: 54, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:33:02,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2510593.3333333335, ans=0.125
2023-11-23 19:33:02,964 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.48 vs. limit=15.0
2023-11-23 19:33:03,574 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376600
2023-11-23 19:33:47,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2510860.0, ans=0.2
2023-11-23 19:33:54,687 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3900, loss[loss=0.06338, simple_loss=0.07556, pruned_loss=0.01212, audio_tagging_loss=0.01348, over 14233.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09052, pruned_loss=0.01342, audio_tagging_loss=0.009086, over 3035592.85 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:34:05,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376650
2023-11-23 19:34:28,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2511060.0, ans=0.2
2023-11-23 19:34:49,482 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.099e+01 8.305e+01 8.896e+01 9.680e+01 1.276e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 19:34:56,886 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 3950, loss[loss=0.08134, simple_loss=0.1045, pruned_loss=0.02103, audio_tagging_loss=0.008069, over 15915.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09048, pruned_loss=0.01341, audio_tagging_loss=0.009239, over 3040831.21 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:35:00,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2511260.0, ans=0.125
2023-11-23 19:35:08,292 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376700
2023-11-23 19:35:14,241 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.43 vs. limit=12.0
2023-11-23 19:35:34,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2511460.0, ans=0.125
2023-11-23 19:35:40,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2511460.0, ans=0.2
2023-11-23 19:35:43,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2511460.0, ans=0.0
2023-11-23 19:35:54,735 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.82 vs. limit=15.0
2023-11-23 19:35:59,036 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4000, loss[loss=0.07639, simple_loss=0.104, pruned_loss=0.01477, audio_tagging_loss=0.009609, over 15715.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09111, pruned_loss=0.01358, audio_tagging_loss=0.00923, over 3040425.75 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:36:10,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376750
2023-11-23 19:36:15,888 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.19 vs. limit=5.0
2023-11-23 19:36:31,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2511726.6666666665, ans=0.0
2023-11-23 19:36:32,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2511726.6666666665, ans=0.0
2023-11-23 19:36:53,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2511860.0, ans=0.125
2023-11-23 19:36:53,798 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.200e+01 8.509e+01 8.978e+01 9.607e+01 1.233e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-23 19:36:59,238 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.39 vs. limit=15.0
2023-11-23 19:37:00,848 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4050, loss[loss=0.07477, simple_loss=0.1009, pruned_loss=0.01632, audio_tagging_loss=0.00799, over 15391.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09193, pruned_loss=0.01363, audio_tagging_loss=0.009143, over 3047430.20 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:37:03,265 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:37:03,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2511926.6666666665, ans=0.125
2023-11-23 19:37:12,604 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376800
2023-11-23 19:37:18,166 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=14.80 vs. limit=22.5
2023-11-23 19:37:19,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2511993.3333333335, ans=0.125
2023-11-23 19:37:50,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2512193.3333333335, ans=0.0
2023-11-23 19:37:51,116 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.03 vs. limit=15.0
2023-11-23 19:37:54,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2512193.3333333335, ans=0.0
2023-11-23 19:38:03,255 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4100, loss[loss=0.06162, simple_loss=0.08568, pruned_loss=0.009287, audio_tagging_loss=0.009492, over 14536.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09194, pruned_loss=0.01358, audio_tagging_loss=0.009152, over 3052503.40 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:38:14,713 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376850
2023-11-23 19:38:17,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2512326.6666666665, ans=0.0
2023-11-23 19:38:31,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2512393.3333333335, ans=0.125
2023-11-23 19:38:50,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2512460.0, ans=0.2
2023-11-23 19:38:54,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2512526.6666666665, ans=0.1
2023-11-23 19:38:56,056 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.97 vs. limit=12.0
2023-11-23 19:38:59,476 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.223e+01 8.590e+01 9.258e+01 1.002e+02 1.554e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-23 19:39:05,552 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4150, loss[loss=0.07913, simple_loss=0.1077, pruned_loss=0.01717, audio_tagging_loss=0.008098, over 15339.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09289, pruned_loss=0.01385, audio_tagging_loss=0.008983, over 3047609.76 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:39:14,954 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.94 vs. limit=15.0
2023-11-23 19:39:16,183 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.87 vs. limit=15.0
2023-11-23 19:39:16,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376900
2023-11-23 19:39:19,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2512660.0, ans=0.2
2023-11-23 19:39:25,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2512660.0, ans=0.2
2023-11-23 19:39:30,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2512726.6666666665, ans=0.0
2023-11-23 19:39:42,219 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.82 vs. limit=15.0
2023-11-23 19:39:43,335 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=15.08 vs. limit=15.0
2023-11-23 19:39:49,385 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:39:50,766 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:40:07,879 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4200, loss[loss=0.07065, simple_loss=0.09858, pruned_loss=0.01323, audio_tagging_loss=0.008131, over 15537.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09275, pruned_loss=0.01397, audio_tagging_loss=0.008847, over 3036623.27 frames. ], batch size: 57, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:40:10,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2512926.6666666665, ans=0.125
2023-11-23 19:40:19,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 376950
2023-11-23 19:40:32,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2513060.0, ans=0.035
2023-11-23 19:40:35,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2513060.0, ans=0.0
2023-11-23 19:40:53,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2513126.6666666665, ans=0.0
2023-11-23 19:40:59,437 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.03 vs. limit=15.0
2023-11-23 19:41:04,172 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.429e+01 8.916e+01 9.542e+01 1.145e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 19:41:10,209 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4250, loss[loss=0.08003, simple_loss=0.1099, pruned_loss=0.01801, audio_tagging_loss=0.00706, over 14929.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09374, pruned_loss=0.01403, audio_tagging_loss=0.008786, over 3044519.49 frames. ], batch size: 55, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:41:12,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2513260.0, ans=0.125
2023-11-23 19:41:13,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2513260.0, ans=0.125
2023-11-23 19:41:14,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2513260.0, ans=0.125
2023-11-23 19:41:22,254 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377000
2023-11-23 19:41:44,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2513393.3333333335, ans=0.125
2023-11-23 19:41:58,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2513460.0, ans=0.125
2023-11-23 19:42:13,233 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4300, loss[loss=0.0838, simple_loss=0.123, pruned_loss=0.01726, audio_tagging_loss=0.005011, over 15468.00 frames. ], tot_loss[loss=0.07002, simple_loss=0.09435, pruned_loss=0.01409, audio_tagging_loss=0.008753, over 3048574.68 frames. ], batch size: 56, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:42:24,749 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377050
2023-11-23 19:42:41,066 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=12.0
2023-11-23 19:42:50,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2513793.3333333335, ans=0.07
2023-11-23 19:42:57,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2513793.3333333335, ans=0.1
2023-11-23 19:43:09,404 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.731e+01 8.254e+01 8.867e+01 9.585e+01 1.161e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 19:43:15,474 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4350, loss[loss=0.06552, simple_loss=0.09126, pruned_loss=0.01263, audio_tagging_loss=0.007259, over 15754.00 frames. ], tot_loss[loss=0.06995, simple_loss=0.09426, pruned_loss=0.01402, audio_tagging_loss=0.008801, over 3050429.66 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 16.0
2023-11-23 19:43:26,706 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377100
2023-11-23 19:43:28,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2513993.3333333335, ans=0.2
2023-11-23 19:43:28,378 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.89 vs. limit=15.0
2023-11-23 19:43:29,555 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.04 vs. limit=10.0
2023-11-23 19:43:52,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.53 vs. limit=6.0
2023-11-23 19:43:54,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2514126.6666666665, ans=0.1
2023-11-23 19:44:00,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2514126.6666666665, ans=0.0
2023-11-23 19:44:17,157 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4400, loss[loss=0.05813, simple_loss=0.07233, pruned_loss=0.01162, audio_tagging_loss=0.01034, over 15305.00 frames. ], tot_loss[loss=0.06986, simple_loss=0.09399, pruned_loss=0.01404, audio_tagging_loss=0.008828, over 3054997.13 frames. ], batch size: 59, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:44:28,437 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377150
2023-11-23 19:44:32,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2514326.6666666665, ans=0.0
2023-11-23 19:44:45,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2514393.3333333335, ans=0.125
2023-11-23 19:44:47,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2514393.3333333335, ans=0.025
2023-11-23 19:44:49,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.51 vs. limit=22.5
2023-11-23 19:44:50,527 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.43 vs. limit=5.0
2023-11-23 19:44:51,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2514393.3333333335, ans=0.125
2023-11-23 19:44:58,897 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.66 vs. limit=22.5
2023-11-23 19:45:13,008 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.858e+01 8.470e+01 9.052e+01 9.834e+01 1.276e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 19:45:20,346 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4450, loss[loss=0.0573, simple_loss=0.07406, pruned_loss=0.01092, audio_tagging_loss=0.009351, over 14937.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09324, pruned_loss=0.01395, audio_tagging_loss=0.008799, over 3055582.58 frames. ], batch size: 58, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:45:30,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2514593.3333333335, ans=0.125
2023-11-23 19:45:31,765 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377200
2023-11-23 19:45:39,400 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:45:52,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.27 vs. limit=15.0
2023-11-23 19:46:12,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2514860.0, ans=0.125
2023-11-23 19:46:23,437 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4500, loss[loss=0.06547, simple_loss=0.09501, pruned_loss=0.00968, audio_tagging_loss=0.008281, over 15154.00 frames. ], tot_loss[loss=0.06958, simple_loss=0.09369, pruned_loss=0.01398, audio_tagging_loss=0.008756, over 3054873.95 frames. ], batch size: 60, lr: 2.13e-03, grad_scale: 32.0
2023-11-23 19:46:29,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2514926.6666666665, ans=0.125
2023-11-23 19:46:34,097 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377250
2023-11-23 19:46:34,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2514993.3333333335, ans=0.2
2023-11-23 19:46:45,668 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.87 vs. limit=15.0
2023-11-23 19:46:58,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2515060.0, ans=0.0
2023-11-23 19:47:18,872 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.855e+01 8.414e+01 9.190e+01 9.821e+01 1.226e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-23 19:47:19,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2515193.3333333335, ans=0.125
2023-11-23 19:47:21,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2515193.3333333335, ans=0.0
2023-11-23 19:47:25,483 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4550, loss[loss=0.06646, simple_loss=0.08806, pruned_loss=0.01357, audio_tagging_loss=0.008868, over 16143.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09263, pruned_loss=0.01377, audio_tagging_loss=0.008812, over 3042717.07 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:47:28,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2515260.0, ans=0.125
2023-11-23 19:47:36,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377300
2023-11-23 19:47:41,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2515326.6666666665, ans=0.125
2023-11-23 19:47:43,612 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.02 vs. limit=6.0
2023-11-23 19:47:52,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2515393.3333333335, ans=0.125
2023-11-23 19:47:55,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2515393.3333333335, ans=0.025
2023-11-23 19:47:57,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2515393.3333333335, ans=0.0
2023-11-23 19:47:59,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2515393.3333333335, ans=0.0
2023-11-23 19:48:02,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2515460.0, ans=0.1
2023-11-23 19:48:11,903 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 19:48:16,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2515526.6666666665, ans=0.1
2023-11-23 19:48:28,010 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4600, loss[loss=0.06687, simple_loss=0.08449, pruned_loss=0.01294, audio_tagging_loss=0.01169, over 15408.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09233, pruned_loss=0.0139, audio_tagging_loss=0.008984, over 3045333.56 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:48:34,289 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.14 vs. limit=22.5
2023-11-23 19:48:39,061 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377350
2023-11-23 19:48:40,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2515660.0, ans=0.125
2023-11-23 19:48:42,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=2515660.0, ans=0.1
2023-11-23 19:48:42,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_na.min_abs, batch_count=2515660.0, ans=0.02
2023-11-23 19:48:44,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2515660.0, ans=0.0
2023-11-23 19:48:47,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2515660.0, ans=0.1
2023-11-23 19:49:07,942 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:49:11,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2515793.3333333335, ans=0.2
2023-11-23 19:49:13,069 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.88 vs. limit=15.0
2023-11-23 19:49:22,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2515860.0, ans=0.125
2023-11-23 19:49:23,553 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.494e+01 9.080e+01 9.829e+01 1.199e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 19:49:23,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2515860.0, ans=0.1
2023-11-23 19:49:27,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2515860.0, ans=0.125
2023-11-23 19:49:30,641 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4650, loss[loss=0.06349, simple_loss=0.08566, pruned_loss=0.009639, audio_tagging_loss=0.01102, over 14394.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.09301, pruned_loss=0.01408, audio_tagging_loss=0.009075, over 3047158.07 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:49:36,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2515926.6666666665, ans=0.125
2023-11-23 19:49:41,359 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377400
2023-11-23 19:50:19,326 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.17 vs. limit=22.5
2023-11-23 19:50:33,078 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4700, loss[loss=0.07482, simple_loss=0.1017, pruned_loss=0.01417, audio_tagging_loss=0.009776, over 14898.00 frames. ], tot_loss[loss=0.07018, simple_loss=0.09371, pruned_loss=0.01428, audio_tagging_loss=0.00904, over 3055380.22 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:50:43,744 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377450
2023-11-23 19:51:05,491 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.15 vs. limit=22.5
2023-11-23 19:51:11,798 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.27 vs. limit=12.0
2023-11-23 19:51:28,171 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.107e+01 8.521e+01 9.088e+01 9.578e+01 1.245e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 19:51:34,066 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4750, loss[loss=0.08453, simple_loss=0.1098, pruned_loss=0.0196, audio_tagging_loss=0.01005, over 16172.00 frames. ], tot_loss[loss=0.06923, simple_loss=0.09241, pruned_loss=0.01388, audio_tagging_loss=0.009147, over 3052506.19 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:51:34,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_na.min_abs, batch_count=2516593.3333333335, ans=0.02
2023-11-23 19:51:40,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2516593.3333333335, ans=0.125
2023-11-23 19:51:45,464 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377500
2023-11-23 19:51:46,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2516660.0, ans=0.1
2023-11-23 19:52:02,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2516726.6666666665, ans=0.0
2023-11-23 19:52:04,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2516726.6666666665, ans=0.125
2023-11-23 19:52:36,091 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4800, loss[loss=0.06545, simple_loss=0.08827, pruned_loss=0.0109, audio_tagging_loss=0.01041, over 15268.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09294, pruned_loss=0.01396, audio_tagging_loss=0.009193, over 3055941.56 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 19:52:38,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2516926.6666666665, ans=0.2
2023-11-23 19:52:38,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2516926.6666666665, ans=0.125
2023-11-23 19:52:44,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2516926.6666666665, ans=0.125
2023-11-23 19:52:47,943 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377550
2023-11-23 19:53:08,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2517060.0, ans=0.0
2023-11-23 19:53:33,630 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.424e+01 8.895e+01 9.657e+01 1.306e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 19:53:34,134 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.86 vs. limit=15.0
2023-11-23 19:53:38,426 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4850, loss[loss=0.06513, simple_loss=0.07756, pruned_loss=0.01412, audio_tagging_loss=0.01223, over 14673.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09203, pruned_loss=0.01361, audio_tagging_loss=0.009324, over 3044817.91 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:53:49,680 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377600
2023-11-23 19:53:54,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2517326.6666666665, ans=0.0
2023-11-23 19:53:54,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2517326.6666666665, ans=0.2
2023-11-23 19:53:55,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2517326.6666666665, ans=0.05
2023-11-23 19:54:40,502 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4900, loss[loss=0.07121, simple_loss=0.0933, pruned_loss=0.01451, audio_tagging_loss=0.01006, over 16741.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09182, pruned_loss=0.01369, audio_tagging_loss=0.009266, over 3040343.57 frames. ], batch size: 62, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:54:51,786 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377650
2023-11-23 19:55:25,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2517793.3333333335, ans=0.125
2023-11-23 19:55:27,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2517793.3333333335, ans=0.125
2023-11-23 19:55:38,173 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.83 vs. limit=15.0
2023-11-23 19:55:38,612 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.557e+01 8.293e+01 8.753e+01 9.560e+01 1.963e+02, threshold=1.751e+02, percent-clipped=1.0
2023-11-23 19:55:43,393 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 4950, loss[loss=0.09292, simple_loss=0.1255, pruned_loss=0.02566, audio_tagging_loss=0.004518, over 15478.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09213, pruned_loss=0.01391, audio_tagging_loss=0.009074, over 3035672.62 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:55:53,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2517926.6666666665, ans=0.125
2023-11-23 19:55:54,839 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377700
2023-11-23 19:56:11,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2518060.0, ans=0.125
2023-11-23 19:56:11,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.84 vs. limit=15.0
2023-11-23 19:56:45,965 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5000, loss[loss=0.06585, simple_loss=0.08558, pruned_loss=0.01122, audio_tagging_loss=0.01184, over 14852.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09134, pruned_loss=0.01359, audio_tagging_loss=0.009051, over 3039082.44 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:56:48,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2518260.0, ans=0.1
2023-11-23 19:56:50,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2518260.0, ans=0.1
2023-11-23 19:56:53,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2518260.0, ans=0.0
2023-11-23 19:56:58,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377750
2023-11-23 19:57:11,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2518393.3333333335, ans=0.125
2023-11-23 19:57:17,794 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 19:57:30,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2518460.0, ans=0.125
2023-11-23 19:57:43,975 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.445e+01 8.514e+01 9.045e+01 9.813e+01 1.165e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 19:57:45,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=2518526.6666666665, ans=15.0
2023-11-23 19:57:46,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2.whitening_limit, batch_count=2518526.6666666665, ans=15.0
2023-11-23 19:57:48,795 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5050, loss[loss=0.06714, simple_loss=0.08701, pruned_loss=0.01413, audio_tagging_loss=0.009499, over 14983.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09182, pruned_loss=0.01361, audio_tagging_loss=0.00908, over 3040409.68 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:57:52,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2518593.3333333335, ans=0.2
2023-11-23 19:58:00,122 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377800
2023-11-23 19:58:43,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2518860.0, ans=0.1
2023-11-23 19:58:46,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2518860.0, ans=0.125
2023-11-23 19:58:50,379 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5100, loss[loss=0.07713, simple_loss=0.1054, pruned_loss=0.01437, audio_tagging_loss=0.01007, over 15448.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09214, pruned_loss=0.01363, audio_tagging_loss=0.009024, over 3042305.06 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 19:59:01,756 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377850
2023-11-23 19:59:03,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2518993.3333333335, ans=0.125
2023-11-23 19:59:15,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2519060.0, ans=0.125
2023-11-23 19:59:23,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2519060.0, ans=0.125
2023-11-23 19:59:36,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2519126.6666666665, ans=0.1
2023-11-23 19:59:39,756 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=10.10 vs. limit=15.0
2023-11-23 19:59:47,574 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.550e+01 9.235e+01 1.014e+02 1.258e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-23 19:59:50,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2519193.3333333335, ans=0.125
2023-11-23 19:59:52,262 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5150, loss[loss=0.06384, simple_loss=0.08656, pruned_loss=0.013, audio_tagging_loss=0.00756, over 15429.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09203, pruned_loss=0.01373, audio_tagging_loss=0.008956, over 3038140.29 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:00:03,377 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377900
2023-11-23 20:00:37,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2519460.0, ans=0.0
2023-11-23 20:00:42,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2519526.6666666665, ans=0.125
2023-11-23 20:00:54,660 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5200, loss[loss=0.0992, simple_loss=0.1393, pruned_loss=0.02336, audio_tagging_loss=0.006186, over 14869.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09259, pruned_loss=0.01394, audio_tagging_loss=0.008915, over 3042065.82 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:00:57,756 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.87 vs. limit=15.0
2023-11-23 20:01:05,947 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 377950
2023-11-23 20:01:11,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2519660.0, ans=0.0
2023-11-23 20:01:28,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2519726.6666666665, ans=0.0
2023-11-23 20:01:34,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2519793.3333333335, ans=0.1
2023-11-23 20:01:35,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.27 vs. limit=15.0
2023-11-23 20:01:49,543 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.70 vs. limit=15.0
2023-11-23 20:01:51,893 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.115e+01 8.438e+01 9.195e+01 9.999e+01 1.195e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-23 20:01:56,763 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5250, loss[loss=0.0834, simple_loss=0.1164, pruned_loss=0.01612, audio_tagging_loss=0.009094, over 15677.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.0928, pruned_loss=0.01397, audio_tagging_loss=0.008839, over 3036632.84 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:01:56,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2519926.6666666665, ans=0.125
2023-11-23 20:02:05,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2519926.6666666665, ans=0.09899494936611666
2023-11-23 20:02:07,995 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378000
2023-11-23 20:02:09,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2519993.3333333335, ans=0.125
2023-11-23 20:02:09,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2519993.3333333335, ans=0.125
2023-11-23 20:02:32,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2520060.0, ans=0.125
2023-11-23 20:02:59,103 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5300, loss[loss=0.07409, simple_loss=0.0948, pruned_loss=0.01758, audio_tagging_loss=0.009109, over 15546.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09308, pruned_loss=0.01406, audio_tagging_loss=0.008786, over 3027996.11 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:03:06,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2520260.0, ans=0.0
2023-11-23 20:03:10,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378050
2023-11-23 20:03:31,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2520393.3333333335, ans=0.1
2023-11-23 20:03:33,945 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.71 vs. limit=10.0
2023-11-23 20:03:56,741 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.515e+01 8.458e+01 9.027e+01 9.985e+01 1.662e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 20:04:01,563 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5350, loss[loss=0.05504, simple_loss=0.07343, pruned_loss=0.008241, audio_tagging_loss=0.01008, over 16082.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.0925, pruned_loss=0.01379, audio_tagging_loss=0.008867, over 3030295.10 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:04:03,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.12 vs. limit=10.0
2023-11-23 20:04:13,483 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378100
2023-11-23 20:04:17,688 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.26 vs. limit=12.0
2023-11-23 20:04:23,535 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.00 vs. limit=15.0
2023-11-23 20:04:29,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2520726.6666666665, ans=0.0
2023-11-23 20:04:29,478 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=12.0
2023-11-23 20:05:04,839 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5400, loss[loss=0.08574, simple_loss=0.1294, pruned_loss=0.01602, audio_tagging_loss=0.005022, over 15807.00 frames. ], tot_loss[loss=0.06934, simple_loss=0.09294, pruned_loss=0.01397, audio_tagging_loss=0.008906, over 3029629.85 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:05:05,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2520926.6666666665, ans=0.125
2023-11-23 20:05:15,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378150
2023-11-23 20:05:15,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2520993.3333333335, ans=0.2
2023-11-23 20:05:24,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2520993.3333333335, ans=0.0
2023-11-23 20:05:34,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2521060.0, ans=0.0
2023-11-23 20:05:48,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2521126.6666666665, ans=0.0
2023-11-23 20:05:50,495 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.97 vs. limit=22.5
2023-11-23 20:05:52,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2521126.6666666665, ans=0.125
2023-11-23 20:05:59,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2521193.3333333335, ans=0.125
2023-11-23 20:06:01,387 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.266e+01 8.400e+01 9.032e+01 9.850e+01 1.216e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-23 20:06:06,749 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5450, loss[loss=0.05574, simple_loss=0.07439, pruned_loss=0.008489, audio_tagging_loss=0.01006, over 15303.00 frames. ], tot_loss[loss=0.07003, simple_loss=0.09385, pruned_loss=0.01416, audio_tagging_loss=0.00895, over 3040394.26 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:06:17,527 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378200
2023-11-23 20:06:23,694 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.93 vs. limit=15.0
2023-11-23 20:06:30,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2521393.3333333335, ans=0.1
2023-11-23 20:06:54,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2521460.0, ans=0.1
2023-11-23 20:07:09,666 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5500, loss[loss=0.07448, simple_loss=0.1041, pruned_loss=0.01627, audio_tagging_loss=0.006178, over 14769.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09287, pruned_loss=0.01388, audio_tagging_loss=0.009004, over 3040394.86 frames. ], batch size: 52, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:07:12,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.14 vs. limit=15.0
2023-11-23 20:07:15,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2521593.3333333335, ans=0.125
2023-11-23 20:07:20,542 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378250
2023-11-23 20:07:38,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2521726.6666666665, ans=0.0
2023-11-23 20:07:54,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2521793.3333333335, ans=0.0
2023-11-23 20:08:00,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2521860.0, ans=0.125
2023-11-23 20:08:06,547 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.084e+01 8.320e+01 8.896e+01 9.871e+01 1.353e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 20:08:06,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2521860.0, ans=0.0
2023-11-23 20:08:12,059 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5550, loss[loss=0.05416, simple_loss=0.07426, pruned_loss=0.007683, audio_tagging_loss=0.009351, over 14850.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09317, pruned_loss=0.01387, audio_tagging_loss=0.009067, over 3037897.97 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:08:23,514 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378300
2023-11-23 20:08:27,463 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.09 vs. limit=15.0
2023-11-23 20:08:56,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2522126.6666666665, ans=0.2
2023-11-23 20:09:13,832 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5600, loss[loss=0.07576, simple_loss=0.09482, pruned_loss=0.01668, audio_tagging_loss=0.01167, over 14766.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09301, pruned_loss=0.01387, audio_tagging_loss=0.009141, over 3034185.00 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:09:14,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2522260.0, ans=0.0
2023-11-23 20:09:25,243 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378350
2023-11-23 20:09:25,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2522326.6666666665, ans=0.125
2023-11-23 20:09:27,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2522326.6666666665, ans=0.125
2023-11-23 20:09:51,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2522460.0, ans=0.125
2023-11-23 20:09:57,320 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 20:10:08,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2522526.6666666665, ans=0.5
2023-11-23 20:10:11,709 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.330e+01 9.147e+01 9.848e+01 1.519e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-23 20:10:15,361 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5650, loss[loss=0.07351, simple_loss=0.1018, pruned_loss=0.01506, audio_tagging_loss=0.007573, over 14563.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09271, pruned_loss=0.01376, audio_tagging_loss=0.0092, over 3034366.28 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:10:26,523 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378400
2023-11-23 20:10:35,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2522660.0, ans=0.125
2023-11-23 20:10:40,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2522726.6666666665, ans=0.125
2023-11-23 20:11:17,842 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5700, loss[loss=0.08546, simple_loss=0.1087, pruned_loss=0.02101, audio_tagging_loss=0.01011, over 14434.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09193, pruned_loss=0.01365, audio_tagging_loss=0.009154, over 3040845.03 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:11:22,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2522926.6666666665, ans=0.0
2023-11-23 20:11:27,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2522926.6666666665, ans=0.07
2023-11-23 20:11:28,385 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378450
2023-11-23 20:11:30,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2522993.3333333335, ans=0.0
2023-11-23 20:11:33,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2522993.3333333335, ans=0.0
2023-11-23 20:11:52,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2523060.0, ans=0.125
2023-11-23 20:12:14,662 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.181e+01 8.343e+01 8.908e+01 9.732e+01 1.345e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 20:12:18,291 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5750, loss[loss=0.08115, simple_loss=0.1135, pruned_loss=0.01914, audio_tagging_loss=0.005281, over 15466.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09247, pruned_loss=0.01381, audio_tagging_loss=0.009064, over 3048907.46 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:12:29,530 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378500
2023-11-23 20:13:20,191 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5800, loss[loss=0.06363, simple_loss=0.07678, pruned_loss=0.01356, audio_tagging_loss=0.01168, over 15538.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09243, pruned_loss=0.01388, audio_tagging_loss=0.008996, over 3047205.21 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:13:25,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2523593.3333333335, ans=0.125
2023-11-23 20:13:31,213 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378550
2023-11-23 20:13:35,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2523660.0, ans=0.125
2023-11-23 20:13:36,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2523660.0, ans=0.025
2023-11-23 20:13:43,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2523726.6666666665, ans=0.125
2023-11-23 20:14:01,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2523793.3333333335, ans=0.2
2023-11-23 20:14:12,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2523860.0, ans=0.125
2023-11-23 20:14:19,416 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.431e+01 8.343e+01 9.004e+01 9.818e+01 1.259e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 20:14:19,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2523860.0, ans=0.0
2023-11-23 20:14:21,786 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5850, loss[loss=0.06827, simple_loss=0.09552, pruned_loss=0.01176, audio_tagging_loss=0.008748, over 15035.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09238, pruned_loss=0.01377, audio_tagging_loss=0.008947, over 3048429.53 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 8.0
2023-11-23 20:14:29,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2523926.6666666665, ans=0.0
2023-11-23 20:14:33,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378600
2023-11-23 20:15:24,464 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5900, loss[loss=0.0581, simple_loss=0.07298, pruned_loss=0.01281, audio_tagging_loss=0.008793, over 15453.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09286, pruned_loss=0.01387, audio_tagging_loss=0.008906, over 3051106.77 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 8.0
2023-11-23 20:15:32,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2524260.0, ans=0.05
2023-11-23 20:15:35,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378650
2023-11-23 20:15:39,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2524326.6666666665, ans=0.0
2023-11-23 20:16:18,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2524526.6666666665, ans=0.0
2023-11-23 20:16:18,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2524526.6666666665, ans=0.125
2023-11-23 20:16:23,875 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.199e+01 8.483e+01 8.986e+01 9.661e+01 2.610e+02, threshold=1.797e+02, percent-clipped=1.0
2023-11-23 20:16:26,893 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 5950, loss[loss=0.0609, simple_loss=0.07703, pruned_loss=0.01217, audio_tagging_loss=0.01022, over 15563.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09186, pruned_loss=0.01366, audio_tagging_loss=0.008939, over 3048049.18 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 8.0
2023-11-23 20:16:28,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2524593.3333333335, ans=0.0
2023-11-23 20:16:28,589 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=12.0
2023-11-23 20:16:33,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2524593.3333333335, ans=0.2
2023-11-23 20:16:38,048 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378700
2023-11-23 20:16:47,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2524660.0, ans=0.95
2023-11-23 20:16:56,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2524726.6666666665, ans=0.125
2023-11-23 20:17:28,288 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6000, loss[loss=0.05506, simple_loss=0.08074, pruned_loss=0.008252, audio_tagging_loss=0.006443, over 14801.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09137, pruned_loss=0.01353, audio_tagging_loss=0.008866, over 3040936.59 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:17:28,291 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 20:18:07,450 INFO [train_asr.py:1253] (0/4) Epoch 32, validation: loss=0.05807, simple_loss=0.05104, pruned_loss=0.005144, audio_tagging_loss=0.02741, over 4681554.00 frames. 
2023-11-23 20:18:07,451 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 20:18:18,763 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378750
2023-11-23 20:18:36,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2525060.0, ans=0.1
2023-11-23 20:18:39,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2525060.0, ans=0.125
2023-11-23 20:18:52,184 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 20:19:06,953 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.325e+01 9.027e+01 9.614e+01 1.325e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-23 20:19:10,003 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6050, loss[loss=0.05204, simple_loss=0.0683, pruned_loss=0.007611, audio_tagging_loss=0.01027, over 15274.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09153, pruned_loss=0.01357, audio_tagging_loss=0.008874, over 3039084.60 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:19:21,285 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378800
2023-11-23 20:19:25,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2525326.6666666665, ans=0.1
2023-11-23 20:19:30,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2525326.6666666665, ans=0.1
2023-11-23 20:20:12,625 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6100, loss[loss=0.06885, simple_loss=0.08894, pruned_loss=0.01216, audio_tagging_loss=0.01222, over 15633.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09083, pruned_loss=0.01335, audio_tagging_loss=0.008931, over 3042531.56 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:20:22,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.90 vs. limit=6.0
2023-11-23 20:20:24,170 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378850
2023-11-23 20:20:36,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2525726.6666666665, ans=0.0
2023-11-23 20:21:00,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2525793.3333333335, ans=0.2
2023-11-23 20:21:06,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2525860.0, ans=0.125
2023-11-23 20:21:12,674 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.468e+01 9.222e+01 1.005e+02 1.157e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-23 20:21:15,191 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6150, loss[loss=0.06254, simple_loss=0.08911, pruned_loss=0.01016, audio_tagging_loss=0.007826, over 16029.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09101, pruned_loss=0.01345, audio_tagging_loss=0.008962, over 3045361.16 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:21:26,620 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378900
2023-11-23 20:21:54,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0
2023-11-23 20:22:04,745 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:22:09,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2526193.3333333335, ans=0.0
2023-11-23 20:22:17,102 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6200, loss[loss=0.07564, simple_loss=0.1061, pruned_loss=0.01626, audio_tagging_loss=0.00634, over 14915.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.0901, pruned_loss=0.01339, audio_tagging_loss=0.009058, over 3035189.31 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:22:22,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2526260.0, ans=0.125
2023-11-23 20:22:29,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 378950
2023-11-23 20:22:45,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2526393.3333333335, ans=0.125
2023-11-23 20:22:55,915 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.03 vs. limit=6.0
2023-11-23 20:23:17,960 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.879e+01 8.457e+01 9.040e+01 9.939e+01 1.355e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-23 20:23:20,351 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6250, loss[loss=0.06439, simple_loss=0.08216, pruned_loss=0.01197, audio_tagging_loss=0.01135, over 14923.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.08933, pruned_loss=0.0134, audio_tagging_loss=0.00931, over 3038311.45 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:23:20,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2526593.3333333335, ans=0.1
2023-11-23 20:23:24,898 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.02 vs. limit=15.0
2023-11-23 20:23:25,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2526593.3333333335, ans=0.125
2023-11-23 20:23:31,671 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379000
2023-11-23 20:23:35,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2526660.0, ans=0.07
2023-11-23 20:23:39,741 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.19 vs. limit=15.0
2023-11-23 20:23:45,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2526726.6666666665, ans=0.125
2023-11-23 20:24:17,557 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.57 vs. limit=15.0
2023-11-23 20:24:22,791 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6300, loss[loss=0.06357, simple_loss=0.07496, pruned_loss=0.01367, audio_tagging_loss=0.01242, over 15395.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.0901, pruned_loss=0.01354, audio_tagging_loss=0.009298, over 3037022.74 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:24:30,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2526926.6666666665, ans=0.2
2023-11-23 20:24:34,257 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379050
2023-11-23 20:24:56,890 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.33 vs. limit=10.0
2023-11-23 20:25:01,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2527126.6666666665, ans=0.125
2023-11-23 20:25:11,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2527193.3333333335, ans=0.125
2023-11-23 20:25:14,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2527193.3333333335, ans=0.125
2023-11-23 20:25:19,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2527193.3333333335, ans=0.125
2023-11-23 20:25:21,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2527193.3333333335, ans=0.0
2023-11-23 20:25:21,994 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.078e+01 8.520e+01 9.067e+01 9.897e+01 1.384e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 20:25:24,465 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6350, loss[loss=0.07578, simple_loss=0.1088, pruned_loss=0.0144, audio_tagging_loss=0.006964, over 15168.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09086, pruned_loss=0.01368, audio_tagging_loss=0.009298, over 3043568.96 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:25:35,675 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379100
2023-11-23 20:25:42,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2527326.6666666665, ans=0.125
2023-11-23 20:25:47,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2527326.6666666665, ans=0.125
2023-11-23 20:26:14,942 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.12 vs. limit=15.0
2023-11-23 20:26:15,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2527526.6666666665, ans=0.2
2023-11-23 20:26:19,803 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.49 vs. limit=22.5
2023-11-23 20:26:24,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2527526.6666666665, ans=0.125
2023-11-23 20:26:27,042 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6400, loss[loss=0.07531, simple_loss=0.1107, pruned_loss=0.0138, audio_tagging_loss=0.006138, over 16266.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09144, pruned_loss=0.01371, audio_tagging_loss=0.009376, over 3039614.96 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:26:28,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2527593.3333333335, ans=0.125
2023-11-23 20:26:38,955 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379150
2023-11-23 20:26:51,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2527726.6666666665, ans=0.2
2023-11-23 20:27:02,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2527726.6666666665, ans=0.125
2023-11-23 20:27:04,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2527793.3333333335, ans=0.125
2023-11-23 20:27:27,506 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.271e+01 8.849e+01 9.635e+01 1.697e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-23 20:27:29,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2527926.6666666665, ans=0.0
2023-11-23 20:27:29,883 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6450, loss[loss=0.06429, simple_loss=0.0913, pruned_loss=0.01026, audio_tagging_loss=0.008383, over 15385.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09113, pruned_loss=0.01356, audio_tagging_loss=0.009395, over 3030998.69 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:27:41,503 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379200
2023-11-23 20:28:07,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.56 vs. limit=15.0
2023-11-23 20:28:17,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2528126.6666666665, ans=0.2
2023-11-23 20:28:31,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2528260.0, ans=0.125
2023-11-23 20:28:32,309 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6500, loss[loss=0.07487, simple_loss=0.09644, pruned_loss=0.0188, audio_tagging_loss=0.007853, over 14016.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09148, pruned_loss=0.01363, audio_tagging_loss=0.009419, over 3040623.23 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:28:34,986 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:28:36,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2528260.0, ans=0.2
2023-11-23 20:28:43,256 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379250
2023-11-23 20:29:01,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.71 vs. limit=6.0
2023-11-23 20:29:19,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2528460.0, ans=0.125
2023-11-23 20:29:32,128 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.058e+01 8.606e+01 9.202e+01 9.720e+01 1.328e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 20:29:35,071 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6550, loss[loss=0.03974, simple_loss=0.04563, pruned_loss=0.005148, audio_tagging_loss=0.01178, over 14444.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09166, pruned_loss=0.01371, audio_tagging_loss=0.00926, over 3040450.56 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:29:46,324 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379300
2023-11-23 20:29:48,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2528660.0, ans=0.2
2023-11-23 20:29:59,228 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.46 vs. limit=15.0
2023-11-23 20:30:04,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2528726.6666666665, ans=0.125
2023-11-23 20:30:05,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2528726.6666666665, ans=0.125
2023-11-23 20:30:07,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2528726.6666666665, ans=0.125
2023-11-23 20:30:12,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2528793.3333333335, ans=0.125
2023-11-23 20:30:13,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2528793.3333333335, ans=0.1
2023-11-23 20:30:37,591 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6600, loss[loss=0.07349, simple_loss=0.09485, pruned_loss=0.01488, audio_tagging_loss=0.01118, over 14912.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.0918, pruned_loss=0.0137, audio_tagging_loss=0.009126, over 3043045.81 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:30:37,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2528926.6666666665, ans=0.125
2023-11-23 20:30:44,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2528926.6666666665, ans=0.2
2023-11-23 20:30:48,444 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379350
2023-11-23 20:31:18,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2529126.6666666665, ans=0.0
2023-11-23 20:31:36,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2529193.3333333335, ans=0.125
2023-11-23 20:31:37,890 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.263e+01 8.994e+01 9.633e+01 1.189e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 20:31:40,331 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6650, loss[loss=0.06729, simple_loss=0.09215, pruned_loss=0.01478, audio_tagging_loss=0.006438, over 16568.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09055, pruned_loss=0.01348, audio_tagging_loss=0.009146, over 3039411.33 frames. ], batch size: 62, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:31:41,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2529260.0, ans=0.125
2023-11-23 20:31:48,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2529260.0, ans=0.0
2023-11-23 20:31:51,010 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379400
2023-11-23 20:31:56,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=2529326.6666666665, ans=10.0
2023-11-23 20:31:58,097 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.72 vs. limit=22.5
2023-11-23 20:32:42,237 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6700, loss[loss=0.06589, simple_loss=0.08697, pruned_loss=0.0132, audio_tagging_loss=0.009209, over 15720.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09071, pruned_loss=0.01341, audio_tagging_loss=0.009087, over 3042778.91 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:32:42,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2529593.3333333335, ans=0.0
2023-11-23 20:32:50,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2529593.3333333335, ans=0.125
2023-11-23 20:32:50,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2529593.3333333335, ans=0.125
2023-11-23 20:32:53,721 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379450
2023-11-23 20:32:58,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2529660.0, ans=0.07
2023-11-23 20:33:42,134 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.739e+01 8.296e+01 8.825e+01 9.532e+01 1.402e+02, threshold=1.765e+02, percent-clipped=0.0
2023-11-23 20:33:42,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2529860.0, ans=0.125
2023-11-23 20:33:45,229 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6750, loss[loss=0.07582, simple_loss=0.09459, pruned_loss=0.02055, audio_tagging_loss=0.007971, over 14475.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.08984, pruned_loss=0.01334, audio_tagging_loss=0.009061, over 3036434.61 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:33:45,636 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:33:45,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2529926.6666666665, ans=0.125
2023-11-23 20:33:56,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379500
2023-11-23 20:34:11,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2530060.0, ans=0.1
2023-11-23 20:34:19,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2530060.0, ans=0.125
2023-11-23 20:34:38,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.90 vs. limit=15.0
2023-11-23 20:34:41,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2530193.3333333335, ans=0.125
2023-11-23 20:34:47,427 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6800, loss[loss=0.06399, simple_loss=0.09141, pruned_loss=0.01073, audio_tagging_loss=0.007552, over 16473.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09049, pruned_loss=0.01351, audio_tagging_loss=0.00902, over 3033046.59 frames. ], batch size: 62, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:34:48,048 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.31 vs. limit=22.5
2023-11-23 20:34:50,811 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:34:58,810 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379550
2023-11-23 20:35:23,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2530460.0, ans=0.125
2023-11-23 20:35:23,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2530460.0, ans=0.0
2023-11-23 20:35:26,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2530460.0, ans=0.1
2023-11-23 20:35:38,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2530526.6666666665, ans=0.0
2023-11-23 20:35:48,140 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.108e+01 8.273e+01 8.968e+01 9.529e+01 1.211e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-23 20:35:49,320 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6850, loss[loss=0.07984, simple_loss=0.1096, pruned_loss=0.01585, audio_tagging_loss=0.009168, over 14678.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09102, pruned_loss=0.01345, audio_tagging_loss=0.009008, over 3031044.52 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:35:50,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2530593.3333333335, ans=0.125
2023-11-23 20:35:52,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2530593.3333333335, ans=0.0
2023-11-23 20:35:58,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2530593.3333333335, ans=0.125
2023-11-23 20:36:00,743 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379600
2023-11-23 20:36:06,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2530660.0, ans=0.125
2023-11-23 20:36:10,316 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.25 vs. limit=15.0
2023-11-23 20:36:23,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2530726.6666666665, ans=0.04949747468305833
2023-11-23 20:36:24,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2530726.6666666665, ans=0.2
2023-11-23 20:36:41,595 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.51 vs. limit=22.5
2023-11-23 20:36:51,608 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6900, loss[loss=0.06529, simple_loss=0.08517, pruned_loss=0.0141, audio_tagging_loss=0.008606, over 15576.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09011, pruned_loss=0.01317, audio_tagging_loss=0.00904, over 3033562.78 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:37:02,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379650
2023-11-23 20:37:13,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2530993.3333333335, ans=0.125
2023-11-23 20:37:38,826 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 20:37:49,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2531193.3333333335, ans=0.125
2023-11-23 20:37:52,177 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.634e+01 8.329e+01 8.996e+01 9.675e+01 1.152e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 20:37:53,389 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 6950, loss[loss=0.06514, simple_loss=0.08871, pruned_loss=0.01449, audio_tagging_loss=0.006293, over 16557.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09125, pruned_loss=0.01348, audio_tagging_loss=0.008859, over 3038659.85 frames. ], batch size: 62, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:37:59,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2531260.0, ans=0.125
2023-11-23 20:38:00,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2531260.0, ans=0.0
2023-11-23 20:38:04,704 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379700
2023-11-23 20:38:04,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2531326.6666666665, ans=0.0
2023-11-23 20:38:13,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2531326.6666666665, ans=10.0
2023-11-23 20:38:19,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2531393.3333333335, ans=0.125
2023-11-23 20:38:21,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2531393.3333333335, ans=0.125
2023-11-23 20:38:33,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2531460.0, ans=0.125
2023-11-23 20:38:47,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2531526.6666666665, ans=0.025
2023-11-23 20:38:53,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2531526.6666666665, ans=0.0
2023-11-23 20:38:53,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2531526.6666666665, ans=0.5
2023-11-23 20:38:55,293 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7000, loss[loss=0.07073, simple_loss=0.09197, pruned_loss=0.01499, audio_tagging_loss=0.009756, over 16151.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09094, pruned_loss=0.01341, audio_tagging_loss=0.008999, over 3033348.72 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:39:06,364 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379750
2023-11-23 20:39:16,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2531660.0, ans=0.125
2023-11-23 20:39:27,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2531726.6666666665, ans=0.1
2023-11-23 20:39:51,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2531860.0, ans=0.0
2023-11-23 20:39:53,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2531860.0, ans=0.2
2023-11-23 20:39:55,966 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.253e+01 8.867e+01 9.647e+01 1.261e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 20:39:57,223 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7050, loss[loss=0.07449, simple_loss=0.1033, pruned_loss=0.0149, audio_tagging_loss=0.007973, over 14812.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09125, pruned_loss=0.01338, audio_tagging_loss=0.009006, over 3037507.94 frames. ], batch size: 53, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:40:03,163 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.28 vs. limit=12.0
2023-11-23 20:40:08,503 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379800
2023-11-23 20:40:25,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2532060.0, ans=0.1
2023-11-23 20:40:32,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2532060.0, ans=0.125
2023-11-23 20:40:36,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2532126.6666666665, ans=0.1
2023-11-23 20:40:40,723 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.36 vs. limit=6.0
2023-11-23 20:40:44,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2532126.6666666665, ans=0.125
2023-11-23 20:40:47,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2532193.3333333335, ans=0.1
2023-11-23 20:40:58,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2532260.0, ans=0.125
2023-11-23 20:40:59,240 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7100, loss[loss=0.06054, simple_loss=0.08324, pruned_loss=0.01072, audio_tagging_loss=0.008194, over 15899.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09115, pruned_loss=0.01361, audio_tagging_loss=0.009118, over 3040033.50 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:41:01,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2532260.0, ans=0.125
2023-11-23 20:41:08,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2532260.0, ans=0.1
2023-11-23 20:41:11,198 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379850
2023-11-23 20:41:13,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2532326.6666666665, ans=0.2
2023-11-23 20:41:19,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2532326.6666666665, ans=0.1
2023-11-23 20:41:53,714 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.22 vs. limit=12.0
2023-11-23 20:42:01,915 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.887e+01 8.490e+01 9.157e+01 9.882e+01 1.439e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-23 20:42:01,959 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7150, loss[loss=0.07396, simple_loss=0.09762, pruned_loss=0.0137, audio_tagging_loss=0.01145, over 15018.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09073, pruned_loss=0.0135, audio_tagging_loss=0.009127, over 3046053.72 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:42:13,328 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379900
2023-11-23 20:42:15,174 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.81 vs. limit=22.5
2023-11-23 20:42:33,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2532726.6666666665, ans=0.125
2023-11-23 20:42:56,952 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:43:01,165 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2532860.0, ans=0.2
2023-11-23 20:43:04,349 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7200, loss[loss=0.06284, simple_loss=0.08552, pruned_loss=0.0116, audio_tagging_loss=0.008477, over 15416.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09063, pruned_loss=0.01334, audio_tagging_loss=0.009116, over 3043908.96 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:43:13,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2532926.6666666665, ans=0.125
2023-11-23 20:43:15,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 379950
2023-11-23 20:43:20,928 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.24 vs. limit=12.0
2023-11-23 20:43:22,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2532993.3333333335, ans=0.125
2023-11-23 20:43:30,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2533060.0, ans=0.0
2023-11-23 20:43:34,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2533060.0, ans=0.0
2023-11-23 20:44:05,867 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7250, loss[loss=0.07448, simple_loss=0.09879, pruned_loss=0.01403, audio_tagging_loss=0.01105, over 15524.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09081, pruned_loss=0.01344, audio_tagging_loss=0.009194, over 3048348.55 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:44:06,966 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.235e+01 8.202e+01 8.739e+01 9.383e+01 1.704e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-23 20:44:17,325 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380000
2023-11-23 20:44:18,778 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-380000.pt
2023-11-23 20:44:21,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2533326.6666666665, ans=0.125
2023-11-23 20:45:10,601 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7300, loss[loss=0.0632, simple_loss=0.08655, pruned_loss=0.01258, audio_tagging_loss=0.007346, over 15883.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09138, pruned_loss=0.01337, audio_tagging_loss=0.009171, over 3041106.78 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:45:17,665 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.16 vs. limit=15.0
2023-11-23 20:45:20,120 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.69 vs. limit=15.0
2023-11-23 20:45:22,452 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380050
2023-11-23 20:45:25,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.68 vs. limit=12.0
2023-11-23 20:46:00,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2533860.0, ans=0.125
2023-11-23 20:46:01,276 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.01 vs. limit=15.0
2023-11-23 20:46:14,076 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7350, loss[loss=0.06766, simple_loss=0.09067, pruned_loss=0.01572, audio_tagging_loss=0.006602, over 15004.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09058, pruned_loss=0.01329, audio_tagging_loss=0.009077, over 3038668.76 frames. ], batch size: 55, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:46:15,159 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.654e+01 8.465e+01 9.022e+01 1.001e+02 1.675e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-23 20:46:15,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2533926.6666666665, ans=0.1
2023-11-23 20:46:21,772 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.53 vs. limit=22.5
2023-11-23 20:46:25,452 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380100
2023-11-23 20:46:33,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2533993.3333333335, ans=0.0
2023-11-23 20:46:39,804 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.63 vs. limit=22.5
2023-11-23 20:46:44,249 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.16 vs. limit=12.0
2023-11-23 20:46:46,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2534060.0, ans=0.125
2023-11-23 20:46:48,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2534060.0, ans=0.125
2023-11-23 20:46:48,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2534060.0, ans=0.1
2023-11-23 20:47:01,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2534126.6666666665, ans=0.2
2023-11-23 20:47:07,796 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.28 vs. limit=15.0
2023-11-23 20:47:14,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2534193.3333333335, ans=0.125
2023-11-23 20:47:16,156 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7400, loss[loss=0.05165, simple_loss=0.07884, pruned_loss=0.006318, audio_tagging_loss=0.005915, over 16057.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09159, pruned_loss=0.01342, audio_tagging_loss=0.008908, over 3036289.02 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:47:27,577 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380150
2023-11-23 20:47:27,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2534326.6666666665, ans=0.0
2023-11-23 20:47:32,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2534326.6666666665, ans=0.5
2023-11-23 20:47:47,143 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.38 vs. limit=15.0
2023-11-23 20:48:01,797 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.10 vs. limit=6.0
2023-11-23 20:48:18,446 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7450, loss[loss=0.06914, simple_loss=0.09286, pruned_loss=0.01339, audio_tagging_loss=0.009315, over 15670.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09261, pruned_loss=0.01382, audio_tagging_loss=0.008864, over 3039874.78 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:48:19,598 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 8.435e+01 9.167e+01 9.796e+01 1.283e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 20:48:21,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=14.09 vs. limit=15.0
2023-11-23 20:48:22,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2534593.3333333335, ans=0.0
2023-11-23 20:48:29,288 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380200
2023-11-23 20:48:29,797 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.24 vs. limit=12.0
2023-11-23 20:49:10,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2534860.0, ans=0.0
2023-11-23 20:49:21,000 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7500, loss[loss=0.08317, simple_loss=0.1131, pruned_loss=0.01932, audio_tagging_loss=0.007282, over 16183.00 frames. ], tot_loss[loss=0.06971, simple_loss=0.09375, pruned_loss=0.01404, audio_tagging_loss=0.008795, over 3040965.84 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:49:27,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2534926.6666666665, ans=0.125
2023-11-23 20:49:29,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2534926.6666666665, ans=0.125
2023-11-23 20:49:32,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380250
2023-11-23 20:49:37,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff2.min_abs, batch_count=2534993.3333333335, ans=0.1
2023-11-23 20:50:23,269 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7550, loss[loss=0.08045, simple_loss=0.1009, pruned_loss=0.02214, audio_tagging_loss=0.007853, over 15683.00 frames. ], tot_loss[loss=0.06927, simple_loss=0.09301, pruned_loss=0.01395, audio_tagging_loss=0.00881, over 3050052.52 frames. ], batch size: 60, lr: 2.12e-03, grad_scale: 16.0
2023-11-23 20:50:24,462 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.259e+01 8.879e+01 9.874e+01 1.226e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-23 20:50:27,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2535260.0, ans=0.125
2023-11-23 20:50:31,144 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.62 vs. limit=12.0
2023-11-23 20:50:33,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2535260.0, ans=0.1
2023-11-23 20:50:34,025 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380300
2023-11-23 20:50:38,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2535326.6666666665, ans=0.125
2023-11-23 20:50:49,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2535393.3333333335, ans=0.5
2023-11-23 20:51:03,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2535460.0, ans=0.1
2023-11-23 20:51:12,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2535526.6666666665, ans=0.125
2023-11-23 20:51:23,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2535526.6666666665, ans=0.125
2023-11-23 20:51:25,742 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7600, loss[loss=0.06796, simple_loss=0.0983, pruned_loss=0.01139, audio_tagging_loss=0.00742, over 14920.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09163, pruned_loss=0.01375, audio_tagging_loss=0.008911, over 3047211.87 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:51:27,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.60 vs. limit=10.0
2023-11-23 20:51:36,513 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380350
2023-11-23 20:51:54,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=2535726.6666666665, ans=10.0
2023-11-23 20:52:13,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2535793.3333333335, ans=0.125
2023-11-23 20:52:27,489 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7650, loss[loss=0.07133, simple_loss=0.09244, pruned_loss=0.01595, audio_tagging_loss=0.00916, over 14933.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09038, pruned_loss=0.01365, audio_tagging_loss=0.008844, over 3043543.55 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:52:29,181 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.957e+01 8.327e+01 8.912e+01 9.610e+01 1.311e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-23 20:52:30,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2535926.6666666665, ans=0.5
2023-11-23 20:52:32,151 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.95 vs. limit=15.0
2023-11-23 20:52:33,314 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.66 vs. limit=15.0
2023-11-23 20:52:39,396 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380400
2023-11-23 20:52:57,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2536060.0, ans=0.0
2023-11-23 20:53:07,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2536126.6666666665, ans=0.1
2023-11-23 20:53:11,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2536126.6666666665, ans=0.125
2023-11-23 20:53:31,362 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7700, loss[loss=0.05671, simple_loss=0.0732, pruned_loss=0.007255, audio_tagging_loss=0.01285, over 14113.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09044, pruned_loss=0.01348, audio_tagging_loss=0.008936, over 3043142.96 frames. ], batch size: 54, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:53:38,785 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 20:53:42,064 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380450
2023-11-23 20:53:53,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2536326.6666666665, ans=0.125
2023-11-23 20:53:53,761 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=12.81 vs. limit=22.5
2023-11-23 20:53:56,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2536393.3333333335, ans=0.125
2023-11-23 20:54:32,991 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7750, loss[loss=0.05864, simple_loss=0.07896, pruned_loss=0.009433, audio_tagging_loss=0.009721, over 15374.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.08954, pruned_loss=0.01324, audio_tagging_loss=0.009023, over 3048077.49 frames. ], batch size: 59, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:54:34,649 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.446e+01 9.126e+01 9.765e+01 1.172e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 20:54:44,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380500
2023-11-23 20:54:52,575 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.81 vs. limit=22.5
2023-11-23 20:54:53,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2536660.0, ans=0.125
2023-11-23 20:54:55,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2536660.0, ans=0.0
2023-11-23 20:55:08,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2536726.6666666665, ans=0.125
2023-11-23 20:55:23,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2536860.0, ans=0.0
2023-11-23 20:55:34,689 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7800, loss[loss=0.06165, simple_loss=0.0836, pruned_loss=0.01197, audio_tagging_loss=0.007877, over 15190.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09006, pruned_loss=0.01336, audio_tagging_loss=0.009074, over 3041337.67 frames. ], batch size: 57, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:55:42,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2536926.6666666665, ans=0.2
2023-11-23 20:55:43,077 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.67 vs. limit=15.0
2023-11-23 20:55:46,027 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380550
2023-11-23 20:56:00,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2537060.0, ans=0.04949747468305833
2023-11-23 20:56:01,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2537060.0, ans=0.125
2023-11-23 20:56:27,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2537193.3333333335, ans=0.0
2023-11-23 20:56:37,565 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7850, loss[loss=0.05923, simple_loss=0.07361, pruned_loss=0.01394, audio_tagging_loss=0.008485, over 15091.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09107, pruned_loss=0.01354, audio_tagging_loss=0.009096, over 3043953.39 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:56:38,737 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.770e+01 8.376e+01 9.070e+01 9.715e+01 1.480e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-23 20:56:45,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2537260.0, ans=0.125
2023-11-23 20:56:49,113 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380600
2023-11-23 20:57:02,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2537393.3333333335, ans=0.125
2023-11-23 20:57:21,065 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.16 vs. limit=15.0
2023-11-23 20:57:31,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2537526.6666666665, ans=0.125
2023-11-23 20:57:40,339 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7900, loss[loss=0.08159, simple_loss=0.1159, pruned_loss=0.0174, audio_tagging_loss=0.006236, over 14502.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.0916, pruned_loss=0.0137, audio_tagging_loss=0.009145, over 3052768.05 frames. ], batch size: 52, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:57:45,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2537593.3333333335, ans=0.2
2023-11-23 20:57:51,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380650
2023-11-23 20:58:42,641 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 7950, loss[loss=0.08518, simple_loss=0.1133, pruned_loss=0.01742, audio_tagging_loss=0.01113, over 15940.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09177, pruned_loss=0.0137, audio_tagging_loss=0.009221, over 3052659.34 frames. ], batch size: 61, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:58:43,749 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.654e+01 8.414e+01 9.153e+01 9.684e+01 1.303e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-23 20:58:49,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2537926.6666666665, ans=0.0
2023-11-23 20:58:54,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380700
2023-11-23 20:58:57,534 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 20:59:14,514 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.49 vs. limit=15.0
2023-11-23 20:59:15,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2538060.0, ans=0.0
2023-11-23 20:59:44,784 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8000, loss[loss=0.0417, simple_loss=0.04536, pruned_loss=0.005925, audio_tagging_loss=0.01309, over 14521.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09152, pruned_loss=0.0136, audio_tagging_loss=0.00927, over 3050615.09 frames. ], batch size: 56, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 20:59:56,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380750
2023-11-23 21:00:10,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2538393.3333333335, ans=0.1
2023-11-23 21:00:16,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.70 vs. limit=22.5
2023-11-23 21:00:46,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2538593.3333333335, ans=0.1
2023-11-23 21:00:47,837 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8050, loss[loss=0.07349, simple_loss=0.09885, pruned_loss=0.01356, audio_tagging_loss=0.01051, over 14752.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09223, pruned_loss=0.01374, audio_tagging_loss=0.00931, over 3048879.59 frames. ], batch size: 58, lr: 2.12e-03, grad_scale: 32.0
2023-11-23 21:00:48,950 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.054e+01 8.478e+01 9.043e+01 9.667e+01 1.192e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 21:00:59,109 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380800
2023-11-23 21:01:09,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2538660.0, ans=0.125
2023-11-23 21:01:14,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2538726.6666666665, ans=0.95
2023-11-23 21:01:18,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2538726.6666666665, ans=0.125
2023-11-23 21:01:29,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2538793.3333333335, ans=0.125
2023-11-23 21:01:37,225 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2538860.0, ans=0.0
2023-11-23 21:01:43,896 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.77 vs. limit=15.0
2023-11-23 21:01:50,368 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8100, loss[loss=0.06664, simple_loss=0.09969, pruned_loss=0.01053, audio_tagging_loss=0.006269, over 15042.00 frames. ], tot_loss[loss=0.0698, simple_loss=0.09331, pruned_loss=0.01398, audio_tagging_loss=0.009155, over 3050970.67 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:01:51,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2538926.6666666665, ans=0.125
2023-11-23 21:02:01,486 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380850
2023-11-23 21:02:26,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2539126.6666666665, ans=0.09899494936611666
2023-11-23 21:02:31,118 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.64 vs. limit=15.0
2023-11-23 21:02:45,410 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.41 vs. limit=22.5
2023-11-23 21:02:52,179 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8150, loss[loss=0.06236, simple_loss=0.07535, pruned_loss=0.01534, audio_tagging_loss=0.009348, over 14798.00 frames. ], tot_loss[loss=0.0696, simple_loss=0.09347, pruned_loss=0.01383, audio_tagging_loss=0.009031, over 3055750.29 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:02:54,491 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.341e+01 9.005e+01 9.405e+01 1.221e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 21:02:55,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2539260.0, ans=0.125
2023-11-23 21:02:58,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2539260.0, ans=0.2
2023-11-23 21:03:03,699 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380900
2023-11-23 21:03:09,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2539326.6666666665, ans=0.2
2023-11-23 21:03:23,186 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.95 vs. limit=15.0
2023-11-23 21:03:24,029 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:03:26,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2539393.3333333335, ans=0.125
2023-11-23 21:03:47,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2539526.6666666665, ans=0.0
2023-11-23 21:03:54,167 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8200, loss[loss=0.07353, simple_loss=0.1045, pruned_loss=0.01257, audio_tagging_loss=0.008715, over 14563.00 frames. ], tot_loss[loss=0.06956, simple_loss=0.09356, pruned_loss=0.01388, audio_tagging_loss=0.008898, over 3048669.21 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:03:54,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2539593.3333333335, ans=0.1
2023-11-23 21:03:55,312 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:03:57,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2539593.3333333335, ans=0.1
2023-11-23 21:04:05,902 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 380950
2023-11-23 21:04:06,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2539660.0, ans=0.0
2023-11-23 21:04:20,945 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=15.0
2023-11-23 21:04:29,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2539726.6666666665, ans=0.125
2023-11-23 21:04:56,991 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8250, loss[loss=0.07161, simple_loss=0.0961, pruned_loss=0.01543, audio_tagging_loss=0.008128, over 15220.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09301, pruned_loss=0.01383, audio_tagging_loss=0.008919, over 3037952.16 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:04:59,273 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.259e+01 8.237e+01 8.988e+01 9.644e+01 1.224e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-23 21:05:05,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2539926.6666666665, ans=0.1
2023-11-23 21:05:07,906 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381000
2023-11-23 21:05:25,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2540060.0, ans=0.0
2023-11-23 21:05:36,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2540126.6666666665, ans=0.0
2023-11-23 21:05:39,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2540126.6666666665, ans=0.125
2023-11-23 21:05:59,295 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8300, loss[loss=0.0608, simple_loss=0.07679, pruned_loss=0.0108, audio_tagging_loss=0.0116, over 14424.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09294, pruned_loss=0.01374, audio_tagging_loss=0.008856, over 3037637.32 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:06:00,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2540260.0, ans=0.125
2023-11-23 21:06:02,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2540260.0, ans=0.125
2023-11-23 21:06:09,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2540260.0, ans=0.0
2023-11-23 21:06:10,715 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381050
2023-11-23 21:06:12,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2540326.6666666665, ans=0.125
2023-11-23 21:06:17,282 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.55 vs. limit=22.5
2023-11-23 21:06:21,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2540326.6666666665, ans=0.1
2023-11-23 21:06:54,192 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.29 vs. limit=15.0
2023-11-23 21:06:57,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2540526.6666666665, ans=0.125
2023-11-23 21:07:01,061 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8350, loss[loss=0.06886, simple_loss=0.09439, pruned_loss=0.01391, audio_tagging_loss=0.007752, over 15252.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09255, pruned_loss=0.01367, audio_tagging_loss=0.00884, over 3036093.66 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:07:03,381 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.724e+01 8.451e+01 9.185e+01 9.824e+01 1.570e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 21:07:03,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2540593.3333333335, ans=0.125
2023-11-23 21:07:07,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2540593.3333333335, ans=0.1
2023-11-23 21:07:11,709 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381100
2023-11-23 21:07:13,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2540660.0, ans=0.125
2023-11-23 21:07:51,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2540860.0, ans=0.0
2023-11-23 21:08:02,556 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8400, loss[loss=0.06019, simple_loss=0.0805, pruned_loss=0.01181, audio_tagging_loss=0.00813, over 14957.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09123, pruned_loss=0.01344, audio_tagging_loss=0.008881, over 3043246.26 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:08:11,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2540926.6666666665, ans=0.125
2023-11-23 21:08:13,825 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381150
2023-11-23 21:08:15,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2540993.3333333335, ans=0.125
2023-11-23 21:08:19,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2540993.3333333335, ans=0.2
2023-11-23 21:08:20,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2540993.3333333335, ans=0.1
2023-11-23 21:08:22,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.91 vs. limit=15.0
2023-11-23 21:08:27,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2541060.0, ans=0.125
2023-11-23 21:08:58,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2541193.3333333335, ans=0.0
2023-11-23 21:09:04,791 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8450, loss[loss=0.08665, simple_loss=0.1155, pruned_loss=0.01981, audio_tagging_loss=0.009093, over 16099.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09152, pruned_loss=0.01356, audio_tagging_loss=0.008948, over 3041438.25 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:09:05,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2541260.0, ans=0.0
2023-11-23 21:09:08,218 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.427e+01 8.936e+01 9.652e+01 1.220e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 21:09:08,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2541260.0, ans=0.1
2023-11-23 21:09:15,919 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381200
2023-11-23 21:09:29,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2541393.3333333335, ans=0.125
2023-11-23 21:09:37,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2541393.3333333335, ans=0.125
2023-11-23 21:09:46,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2541460.0, ans=0.2
2023-11-23 21:10:07,123 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8500, loss[loss=0.06282, simple_loss=0.0862, pruned_loss=0.01212, audio_tagging_loss=0.0076, over 15652.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09147, pruned_loss=0.01352, audio_tagging_loss=0.00891, over 3044784.32 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:10:11,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2541593.3333333335, ans=0.2
2023-11-23 21:10:11,362 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.81 vs. limit=15.0
2023-11-23 21:10:14,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2541593.3333333335, ans=0.0
2023-11-23 21:10:18,023 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381250
2023-11-23 21:10:26,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2541660.0, ans=0.125
2023-11-23 21:10:27,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.05 vs. limit=15.0
2023-11-23 21:10:32,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2541726.6666666665, ans=0.0
2023-11-23 21:10:51,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2541793.3333333335, ans=0.125
2023-11-23 21:10:51,691 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.06 vs. limit=15.0
2023-11-23 21:10:57,488 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.82 vs. limit=15.0
2023-11-23 21:11:09,326 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8550, loss[loss=0.06558, simple_loss=0.08969, pruned_loss=0.01218, audio_tagging_loss=0.008555, over 15486.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09105, pruned_loss=0.0133, audio_tagging_loss=0.008999, over 3044727.04 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:11:11,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2541926.6666666665, ans=0.2
2023-11-23 21:11:12,874 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.291e+01 8.400e+01 9.293e+01 9.776e+01 1.237e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-23 21:11:14,647 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.11 vs. limit=22.5
2023-11-23 21:11:20,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381300
2023-11-23 21:11:52,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2542126.6666666665, ans=0.125
2023-11-23 21:12:06,820 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-23 21:12:11,443 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8600, loss[loss=0.07234, simple_loss=0.08921, pruned_loss=0.01563, audio_tagging_loss=0.0121, over 15486.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09116, pruned_loss=0.01341, audio_tagging_loss=0.00907, over 3048508.08 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:12:19,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2542260.0, ans=0.125
2023-11-23 21:12:20,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2542260.0, ans=0.0
2023-11-23 21:12:20,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2542260.0, ans=0.125
2023-11-23 21:12:22,659 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381350
2023-11-23 21:12:28,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2542326.6666666665, ans=0.0
2023-11-23 21:12:35,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2542393.3333333335, ans=0.0
2023-11-23 21:12:36,523 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=13.93 vs. limit=15.0
2023-11-23 21:12:59,025 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0
2023-11-23 21:13:01,441 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.60 vs. limit=15.0
2023-11-23 21:13:13,096 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8650, loss[loss=0.07693, simple_loss=0.103, pruned_loss=0.01458, audio_tagging_loss=0.01085, over 15078.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09045, pruned_loss=0.01322, audio_tagging_loss=0.009171, over 3044112.42 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:13:16,582 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.050e+01 8.560e+01 9.209e+01 9.798e+01 1.197e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-23 21:13:19,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2542593.3333333335, ans=0.125
2023-11-23 21:13:23,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381400
2023-11-23 21:13:34,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.63 vs. limit=15.0
2023-11-23 21:13:57,061 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:14:07,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2542860.0, ans=0.0
2023-11-23 21:14:10,253 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.32 vs. limit=22.5
2023-11-23 21:14:15,431 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8700, loss[loss=0.09056, simple_loss=0.1144, pruned_loss=0.02121, audio_tagging_loss=0.01216, over 15527.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09044, pruned_loss=0.0132, audio_tagging_loss=0.009293, over 3048287.66 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:14:26,570 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381450
2023-11-23 21:14:35,763 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.09 vs. limit=10.0
2023-11-23 21:14:47,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2543060.0, ans=0.95
2023-11-23 21:14:56,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2543126.6666666665, ans=0.125
2023-11-23 21:15:17,582 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8750, loss[loss=0.08128, simple_loss=0.09862, pruned_loss=0.0214, audio_tagging_loss=0.01057, over 13700.00 frames. ], tot_loss[loss=0.06889, simple_loss=0.09233, pruned_loss=0.01354, audio_tagging_loss=0.009193, over 3046603.40 frames. ], batch size: 52, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:15:21,702 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.016e+01 8.453e+01 8.955e+01 9.752e+01 1.523e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-23 21:15:21,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2543260.0, ans=0.0
2023-11-23 21:15:27,810 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.42 vs. limit=6.0
2023-11-23 21:15:29,510 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381500
2023-11-23 21:15:56,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2543460.0, ans=0.125
2023-11-23 21:16:03,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2543460.0, ans=0.04949747468305833
2023-11-23 21:16:05,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2543460.0, ans=0.125
2023-11-23 21:16:10,790 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.51 vs. limit=15.0
2023-11-23 21:16:15,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2543526.6666666665, ans=0.2
2023-11-23 21:16:17,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2543526.6666666665, ans=0.125
2023-11-23 21:16:20,097 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8800, loss[loss=0.07543, simple_loss=0.1029, pruned_loss=0.015, audio_tagging_loss=0.008978, over 16370.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09293, pruned_loss=0.01358, audio_tagging_loss=0.009257, over 3047607.62 frames. ], batch size: 61, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:16:25,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2543593.3333333335, ans=0.0
2023-11-23 21:16:30,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2543593.3333333335, ans=0.125
2023-11-23 21:16:31,272 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381550
2023-11-23 21:17:14,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2543860.0, ans=0.05
2023-11-23 21:17:22,304 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8850, loss[loss=0.06391, simple_loss=0.08362, pruned_loss=0.0129, audio_tagging_loss=0.009194, over 14356.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09257, pruned_loss=0.0136, audio_tagging_loss=0.009302, over 3056490.50 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:17:23,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2543926.6666666665, ans=0.0
2023-11-23 21:17:25,948 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.726e+01 8.519e+01 9.142e+01 9.754e+01 1.117e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 21:17:33,927 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381600
2023-11-23 21:17:35,413 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:17:43,403 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.54 vs. limit=15.0
2023-11-23 21:17:48,609 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.46 vs. limit=22.5
2023-11-23 21:17:53,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2544060.0, ans=0.2
2023-11-23 21:18:06,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2544126.6666666665, ans=0.0
2023-11-23 21:18:25,684 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8900, loss[loss=0.05063, simple_loss=0.06531, pruned_loss=0.006717, audio_tagging_loss=0.01126, over 16014.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09301, pruned_loss=0.01383, audio_tagging_loss=0.009019, over 3051214.73 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:18:36,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381650
2023-11-23 21:18:53,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2544393.3333333335, ans=0.1
2023-11-23 21:18:55,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2544393.3333333335, ans=0.125
2023-11-23 21:19:00,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2544393.3333333335, ans=0.0
2023-11-23 21:19:27,559 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 8950, loss[loss=0.06668, simple_loss=0.09203, pruned_loss=0.01343, audio_tagging_loss=0.00723, over 15226.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09311, pruned_loss=0.0138, audio_tagging_loss=0.008851, over 3053182.97 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:19:32,784 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.301e+01 8.310e+01 8.895e+01 9.557e+01 1.131e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 21:19:36,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2544593.3333333335, ans=0.1
2023-11-23 21:19:38,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2544593.3333333335, ans=0.125
2023-11-23 21:19:39,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381700
2023-11-23 21:19:41,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2544660.0, ans=0.1
2023-11-23 21:19:43,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2544660.0, ans=0.125
2023-11-23 21:19:54,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2544726.6666666665, ans=0.125
2023-11-23 21:19:59,450 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.18 vs. limit=22.5
2023-11-23 21:20:01,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.18 vs. limit=15.0
2023-11-23 21:20:17,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2544860.0, ans=0.125
2023-11-23 21:20:18,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2544860.0, ans=0.07
2023-11-23 21:20:29,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2544926.6666666665, ans=0.125
2023-11-23 21:20:30,425 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9000, loss[loss=0.05056, simple_loss=0.07386, pruned_loss=0.007545, audio_tagging_loss=0.006088, over 14431.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09333, pruned_loss=0.01377, audio_tagging_loss=0.008818, over 3058656.55 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:20:30,428 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 21:20:53,316 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.2926, 4.9913, 4.6820, 5.1593], device='cuda:0')
2023-11-23 21:21:10,445 INFO [train_asr.py:1253] (0/4) Epoch 32, validation: loss=0.05909, simple_loss=0.05099, pruned_loss=0.005136, audio_tagging_loss=0.02845, over 4681554.00 frames. 
2023-11-23 21:21:10,446 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 21:21:14,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2544926.6666666665, ans=0.125
2023-11-23 21:21:21,706 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381750
2023-11-23 21:21:24,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2544993.3333333335, ans=0.0
2023-11-23 21:22:12,920 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9050, loss[loss=0.08373, simple_loss=0.1131, pruned_loss=0.01958, audio_tagging_loss=0.007587, over 16613.00 frames. ], tot_loss[loss=0.06928, simple_loss=0.09358, pruned_loss=0.01377, audio_tagging_loss=0.008719, over 3059443.96 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:22:17,519 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.661e+01 9.152e+01 9.951e+01 1.252e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 21:22:23,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2545260.0, ans=0.125
2023-11-23 21:22:24,247 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381800
2023-11-23 21:22:24,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2545326.6666666665, ans=0.0
2023-11-23 21:22:34,742 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.61 vs. limit=15.0
2023-11-23 21:22:36,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2545393.3333333335, ans=0.125
2023-11-23 21:23:15,523 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9100, loss[loss=0.0579, simple_loss=0.07985, pruned_loss=0.007685, audio_tagging_loss=0.01029, over 16120.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09294, pruned_loss=0.01358, audio_tagging_loss=0.008708, over 3061572.80 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:23:16,216 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.46 vs. limit=10.0
2023-11-23 21:23:26,909 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381850
2023-11-23 21:24:13,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2545860.0, ans=0.0
2023-11-23 21:24:17,817 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9150, loss[loss=0.06293, simple_loss=0.08627, pruned_loss=0.01003, audio_tagging_loss=0.009774, over 14482.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09229, pruned_loss=0.01358, audio_tagging_loss=0.008796, over 3058197.46 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:24:23,235 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.696e+01 8.321e+01 8.921e+01 9.557e+01 1.166e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 21:24:28,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2545926.6666666665, ans=0.125
2023-11-23 21:24:29,251 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381900
2023-11-23 21:24:35,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2545993.3333333335, ans=0.125
2023-11-23 21:25:20,078 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9200, loss[loss=0.06495, simple_loss=0.09228, pruned_loss=0.01142, audio_tagging_loss=0.007395, over 14794.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09226, pruned_loss=0.01355, audio_tagging_loss=0.008785, over 3063878.24 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:25:31,538 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 381950
2023-11-23 21:25:38,790 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.51 vs. limit=15.0
2023-11-23 21:25:51,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2546393.3333333335, ans=0.0
2023-11-23 21:26:07,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2546460.0, ans=0.0
2023-11-23 21:26:11,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2546526.6666666665, ans=0.125
2023-11-23 21:26:22,414 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9250, loss[loss=0.09554, simple_loss=0.1171, pruned_loss=0.02524, audio_tagging_loss=0.01175, over 16318.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.0928, pruned_loss=0.01375, audio_tagging_loss=0.008836, over 3064690.51 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:26:23,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2546593.3333333335, ans=0.125
2023-11-23 21:26:28,947 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.719e+01 8.321e+01 8.948e+01 9.831e+01 1.146e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-23 21:26:33,956 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382000
2023-11-23 21:26:37,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2546660.0, ans=0.0
2023-11-23 21:26:40,875 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:26:40,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2546660.0, ans=0.125
2023-11-23 21:26:47,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2546726.6666666665, ans=0.0
2023-11-23 21:27:18,547 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.67 vs. limit=15.0
2023-11-23 21:27:19,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2546860.0, ans=0.125
2023-11-23 21:27:25,742 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9300, loss[loss=0.06147, simple_loss=0.08563, pruned_loss=0.0106, audio_tagging_loss=0.008054, over 15603.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09286, pruned_loss=0.01373, audio_tagging_loss=0.008958, over 3073185.01 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:27:32,317 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.85 vs. limit=15.0
2023-11-23 21:27:36,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382050
2023-11-23 21:27:46,337 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.15 vs. limit=15.0
2023-11-23 21:27:50,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2547060.0, ans=0.125
2023-11-23 21:27:51,826 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.08 vs. limit=15.0
2023-11-23 21:28:19,438 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.16 vs. limit=22.5
2023-11-23 21:28:26,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2547260.0, ans=0.2
2023-11-23 21:28:27,245 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9350, loss[loss=0.08209, simple_loss=0.1074, pruned_loss=0.01787, audio_tagging_loss=0.0105, over 15210.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09286, pruned_loss=0.01372, audio_tagging_loss=0.00902, over 3069308.35 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:28:33,553 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.368e+01 8.391e+01 8.927e+01 9.865e+01 1.174e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 21:28:38,382 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382100
2023-11-23 21:28:48,499 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.95 vs. limit=22.5
2023-11-23 21:28:57,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2547393.3333333335, ans=0.07
2023-11-23 21:29:12,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2547460.0, ans=0.95
2023-11-23 21:29:14,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2547460.0, ans=0.0
2023-11-23 21:29:21,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2547526.6666666665, ans=0.125
2023-11-23 21:29:23,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2547526.6666666665, ans=0.2
2023-11-23 21:29:29,306 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9400, loss[loss=0.04985, simple_loss=0.05426, pruned_loss=0.01116, audio_tagging_loss=0.01156, over 15983.00 frames. ], tot_loss[loss=0.06952, simple_loss=0.09318, pruned_loss=0.01388, audio_tagging_loss=0.009044, over 3061969.07 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:29:36,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2547593.3333333335, ans=0.0
2023-11-23 21:29:40,649 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382150
2023-11-23 21:29:44,890 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.98 vs. limit=22.5
2023-11-23 21:30:06,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.86 vs. limit=15.0
2023-11-23 21:30:06,857 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.64 vs. limit=22.5
2023-11-23 21:30:18,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2547860.0, ans=0.125
2023-11-23 21:30:29,128 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:30:31,082 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.28 vs. limit=22.5
2023-11-23 21:30:31,446 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9450, loss[loss=0.07154, simple_loss=0.09008, pruned_loss=0.01595, audio_tagging_loss=0.01056, over 14901.00 frames. ], tot_loss[loss=0.06944, simple_loss=0.09286, pruned_loss=0.01394, audio_tagging_loss=0.009072, over 3062848.92 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:30:37,932 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.084e+01 8.414e+01 9.183e+01 9.854e+01 1.204e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 21:30:43,344 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382200
2023-11-23 21:30:55,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2547993.3333333335, ans=0.125
2023-11-23 21:31:03,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2548060.0, ans=0.1
2023-11-23 21:31:18,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2548126.6666666665, ans=0.2
2023-11-23 21:31:19,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2548126.6666666665, ans=0.125
2023-11-23 21:31:28,531 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.31 vs. limit=15.0
2023-11-23 21:31:34,402 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9500, loss[loss=0.08287, simple_loss=0.112, pruned_loss=0.0187, audio_tagging_loss=0.008169, over 15042.00 frames. ], tot_loss[loss=0.06933, simple_loss=0.09249, pruned_loss=0.01392, audio_tagging_loss=0.009164, over 3052619.19 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:31:45,740 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382250
2023-11-23 21:31:51,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2548326.6666666665, ans=0.2
2023-11-23 21:32:03,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2548393.3333333335, ans=0.125
2023-11-23 21:32:06,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2548393.3333333335, ans=0.125
2023-11-23 21:32:10,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2548460.0, ans=0.0
2023-11-23 21:32:12,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2548460.0, ans=0.125
2023-11-23 21:32:15,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2548460.0, ans=6.0
2023-11-23 21:32:30,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2548526.6666666665, ans=0.125
2023-11-23 21:32:31,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2548526.6666666665, ans=0.2
2023-11-23 21:32:36,316 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9550, loss[loss=0.07028, simple_loss=0.0998, pruned_loss=0.01088, audio_tagging_loss=0.009494, over 14121.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09234, pruned_loss=0.0139, audio_tagging_loss=0.009231, over 3046385.65 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:32:36,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2548593.3333333335, ans=0.125
2023-11-23 21:32:36,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2548593.3333333335, ans=0.1
2023-11-23 21:32:42,279 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.598e+01 9.160e+01 9.932e+01 1.326e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-23 21:32:47,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382300
2023-11-23 21:33:10,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=7.55 vs. limit=12.0
2023-11-23 21:33:37,963 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9600, loss[loss=0.06038, simple_loss=0.07482, pruned_loss=0.01168, audio_tagging_loss=0.01129, over 16420.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09147, pruned_loss=0.01376, audio_tagging_loss=0.009419, over 3057198.69 frames. ], batch size: 62, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:33:47,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2548926.6666666665, ans=0.1
2023-11-23 21:33:50,061 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382350
2023-11-23 21:33:55,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2548993.3333333335, ans=0.1
2023-11-23 21:34:17,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2549126.6666666665, ans=0.0
2023-11-23 21:34:41,433 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9650, loss[loss=0.06198, simple_loss=0.0914, pruned_loss=0.01052, audio_tagging_loss=0.005761, over 14486.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09037, pruned_loss=0.0135, audio_tagging_loss=0.009385, over 3050609.92 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:34:45,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2549260.0, ans=0.125
2023-11-23 21:34:47,305 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.285e+01 8.434e+01 9.003e+01 9.684e+01 1.226e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 21:34:52,754 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382400
2023-11-23 21:35:22,273 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.94 vs. limit=15.0
2023-11-23 21:35:30,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2549526.6666666665, ans=0.0
2023-11-23 21:35:44,311 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9700, loss[loss=0.07853, simple_loss=0.1066, pruned_loss=0.01672, audio_tagging_loss=0.008497, over 15300.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09022, pruned_loss=0.01351, audio_tagging_loss=0.009243, over 3048879.18 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:35:55,005 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382450
2023-11-23 21:36:29,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2549793.3333333335, ans=0.125
2023-11-23 21:36:33,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2549860.0, ans=0.0
2023-11-23 21:36:35,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2549860.0, ans=0.125
2023-11-23 21:36:45,384 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9750, loss[loss=0.0878, simple_loss=0.1265, pruned_loss=0.01935, audio_tagging_loss=0.005185, over 14715.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09151, pruned_loss=0.01366, audio_tagging_loss=0.009019, over 3043504.78 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:36:46,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2549926.6666666665, ans=0.0
2023-11-23 21:36:51,913 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.706e+01 8.362e+01 8.986e+01 9.723e+01 1.186e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 21:36:56,690 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382500
2023-11-23 21:37:05,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2549993.3333333335, ans=0.125
2023-11-23 21:37:22,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2550126.6666666665, ans=0.0
2023-11-23 21:37:26,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2550126.6666666665, ans=0.1
2023-11-23 21:37:26,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2550126.6666666665, ans=0.0
2023-11-23 21:37:27,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2550126.6666666665, ans=0.05
2023-11-23 21:37:33,987 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.90 vs. limit=15.0
2023-11-23 21:37:47,730 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9800, loss[loss=0.0657, simple_loss=0.08709, pruned_loss=0.01208, audio_tagging_loss=0.01008, over 14755.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09181, pruned_loss=0.01375, audio_tagging_loss=0.009064, over 3041838.32 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:37:59,708 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382550
2023-11-23 21:38:18,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2550393.3333333335, ans=0.125
2023-11-23 21:38:33,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2550460.0, ans=0.125
2023-11-23 21:38:37,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2550526.6666666665, ans=10.0
2023-11-23 21:38:37,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2550526.6666666665, ans=0.125
2023-11-23 21:38:43,020 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:38:44,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2550526.6666666665, ans=0.0
2023-11-23 21:38:44,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2550526.6666666665, ans=0.125
2023-11-23 21:38:48,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2550526.6666666665, ans=0.0
2023-11-23 21:38:50,715 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9850, loss[loss=0.06318, simple_loss=0.0808, pruned_loss=0.01461, audio_tagging_loss=0.008168, over 14925.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09255, pruned_loss=0.01391, audio_tagging_loss=0.008883, over 3042097.36 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:38:56,531 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.500e+01 9.067e+01 9.984e+01 1.563e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-23 21:38:57,454 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.38 vs. limit=15.0
2023-11-23 21:39:01,352 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382600
2023-11-23 21:39:17,286 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.14 vs. limit=15.0
2023-11-23 21:39:37,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2550793.3333333335, ans=0.04949747468305833
2023-11-23 21:39:46,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.57 vs. limit=15.0
2023-11-23 21:39:52,829 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9900, loss[loss=0.08695, simple_loss=0.1141, pruned_loss=0.02072, audio_tagging_loss=0.009172, over 15422.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09232, pruned_loss=0.01379, audio_tagging_loss=0.008835, over 3041116.71 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:40:01,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2550926.6666666665, ans=0.025
2023-11-23 21:40:04,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382650
2023-11-23 21:40:08,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2550993.3333333335, ans=0.0
2023-11-23 21:40:53,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2551193.3333333335, ans=0.125
2023-11-23 21:40:55,637 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 9950, loss[loss=0.08183, simple_loss=0.1084, pruned_loss=0.02053, audio_tagging_loss=0.007087, over 14935.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09195, pruned_loss=0.01373, audio_tagging_loss=0.008881, over 3040695.43 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:41:03,345 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.388e+01 8.223e+01 9.080e+01 9.967e+01 1.331e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 21:41:07,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382700
2023-11-23 21:41:15,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2551326.6666666665, ans=0.125
2023-11-23 21:41:17,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2551326.6666666665, ans=0.125
2023-11-23 21:41:18,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.51 vs. limit=15.0
2023-11-23 21:41:20,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2551393.3333333335, ans=0.125
2023-11-23 21:41:41,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2551460.0, ans=0.125
2023-11-23 21:41:42,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2551460.0, ans=0.125
2023-11-23 21:41:48,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2551526.6666666665, ans=0.04949747468305833
2023-11-23 21:41:50,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2551526.6666666665, ans=0.125
2023-11-23 21:41:59,250 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10000, loss[loss=0.06364, simple_loss=0.08625, pruned_loss=0.01143, audio_tagging_loss=0.009081, over 14893.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09087, pruned_loss=0.01354, audio_tagging_loss=0.008857, over 3045471.85 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:42:02,167 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.73 vs. limit=15.0
2023-11-23 21:42:04,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2551593.3333333335, ans=0.05
2023-11-23 21:42:09,935 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382750
2023-11-23 21:42:28,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2551726.6666666665, ans=0.2
2023-11-23 21:42:55,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2551860.0, ans=0.125
2023-11-23 21:43:00,964 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10050, loss[loss=0.08125, simple_loss=0.1068, pruned_loss=0.01977, audio_tagging_loss=0.008064, over 15538.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09193, pruned_loss=0.01368, audio_tagging_loss=0.008872, over 3052588.41 frames. ], batch size: 59, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:43:02,963 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.37 vs. limit=15.0
2023-11-23 21:43:08,057 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.171e+01 8.368e+01 8.922e+01 9.527e+01 1.170e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 21:43:11,940 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382800
2023-11-23 21:43:20,781 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.37 vs. limit=15.0
2023-11-23 21:43:37,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2552060.0, ans=0.1
2023-11-23 21:43:59,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2552193.3333333335, ans=0.1
2023-11-23 21:44:03,496 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10100, loss[loss=0.07619, simple_loss=0.1043, pruned_loss=0.01458, audio_tagging_loss=0.009457, over 17071.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09165, pruned_loss=0.01345, audio_tagging_loss=0.008899, over 3059407.94 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:44:14,777 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382850
2023-11-23 21:44:20,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=2552326.6666666665, ans=0.2
2023-11-23 21:44:35,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2552393.3333333335, ans=0.0
2023-11-23 21:44:36,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2552393.3333333335, ans=0.2
2023-11-23 21:44:37,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2552393.3333333335, ans=0.2
2023-11-23 21:44:49,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2552460.0, ans=0.0
2023-11-23 21:44:53,149 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:44:59,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2552526.6666666665, ans=0.0
2023-11-23 21:45:06,044 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10150, loss[loss=0.0874, simple_loss=0.1086, pruned_loss=0.02289, audio_tagging_loss=0.0102, over 15015.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09165, pruned_loss=0.01358, audio_tagging_loss=0.009011, over 3047291.74 frames. ], batch size: 56, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:45:07,744 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.76 vs. limit=15.0
2023-11-23 21:45:11,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2552593.3333333335, ans=0.125
2023-11-23 21:45:14,977 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.624e+01 8.658e+01 9.213e+01 1.003e+02 1.366e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-23 21:45:17,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382900
2023-11-23 21:45:18,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.09 vs. limit=15.0
2023-11-23 21:45:34,774 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:45:36,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2552726.6666666665, ans=0.125
2023-11-23 21:45:38,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2552726.6666666665, ans=0.125
2023-11-23 21:45:48,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2552793.3333333335, ans=0.125
2023-11-23 21:45:50,141 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.85 vs. limit=15.0
2023-11-23 21:45:55,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2552860.0, ans=0.125
2023-11-23 21:46:08,464 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10200, loss[loss=0.06935, simple_loss=0.0956, pruned_loss=0.01311, audio_tagging_loss=0.008441, over 14671.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09159, pruned_loss=0.01361, audio_tagging_loss=0.009115, over 3044297.00 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:46:10,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2552926.6666666665, ans=0.125
2023-11-23 21:46:10,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2552926.6666666665, ans=0.125
2023-11-23 21:46:14,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2552926.6666666665, ans=0.125
2023-11-23 21:46:15,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2552926.6666666665, ans=0.2
2023-11-23 21:46:19,161 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 382950
2023-11-23 21:46:26,283 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.45 vs. limit=15.0
2023-11-23 21:46:30,766 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 21:46:56,149 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.11 vs. limit=15.0
2023-11-23 21:47:10,205 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10250, loss[loss=0.08398, simple_loss=0.1195, pruned_loss=0.01786, audio_tagging_loss=0.006368, over 16523.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09295, pruned_loss=0.0139, audio_tagging_loss=0.009125, over 3044566.14 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:47:13,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2553260.0, ans=0.125
2023-11-23 21:47:18,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2553260.0, ans=0.125
2023-11-23 21:47:18,656 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.52 vs. limit=15.0
2023-11-23 21:47:18,953 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.042e+01 8.706e+01 9.282e+01 1.010e+02 1.139e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-23 21:47:21,403 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383000
2023-11-23 21:47:27,155 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.63 vs. limit=15.0
2023-11-23 21:47:29,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2553326.6666666665, ans=0.2
2023-11-23 21:47:46,882 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.57 vs. limit=12.0
2023-11-23 21:47:52,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2553460.0, ans=0.1
2023-11-23 21:47:54,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2553460.0, ans=0.0
2023-11-23 21:47:59,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2553526.6666666665, ans=0.125
2023-11-23 21:48:12,193 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10300, loss[loss=0.06427, simple_loss=0.08397, pruned_loss=0.0102, audio_tagging_loss=0.01209, over 14798.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09229, pruned_loss=0.01378, audio_tagging_loss=0.009182, over 3052992.46 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:48:23,514 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383050
2023-11-23 21:48:33,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2553660.0, ans=0.125
2023-11-23 21:48:34,850 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:48:36,320 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.09 vs. limit=22.5
2023-11-23 21:48:38,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2553726.6666666665, ans=0.09899494936611666
2023-11-23 21:49:12,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2553860.0, ans=0.1
2023-11-23 21:49:14,278 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10350, loss[loss=0.0671, simple_loss=0.09249, pruned_loss=0.01029, audio_tagging_loss=0.01057, over 15025.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09179, pruned_loss=0.01362, audio_tagging_loss=0.00922, over 3049338.06 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:49:18,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2553926.6666666665, ans=0.0
2023-11-23 21:49:23,130 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.250e+01 8.433e+01 8.904e+01 9.438e+01 1.477e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-23 21:49:25,649 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383100
2023-11-23 21:50:16,713 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10400, loss[loss=0.06588, simple_loss=0.08209, pruned_loss=0.01388, audio_tagging_loss=0.01095, over 13793.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09193, pruned_loss=0.01353, audio_tagging_loss=0.009276, over 3054833.07 frames. ], batch size: 53, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:50:28,801 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383150
2023-11-23 21:50:30,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2554326.6666666665, ans=0.125
2023-11-23 21:50:46,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2554393.3333333335, ans=0.2
2023-11-23 21:50:52,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2554393.3333333335, ans=0.0
2023-11-23 21:51:09,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2554526.6666666665, ans=0.125
2023-11-23 21:51:19,719 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10450, loss[loss=0.06927, simple_loss=0.09095, pruned_loss=0.01085, audio_tagging_loss=0.01294, over 14699.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09099, pruned_loss=0.01337, audio_tagging_loss=0.009303, over 3058499.28 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:51:21,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2554593.3333333335, ans=0.0
2023-11-23 21:51:28,556 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.962e+01 8.248e+01 8.875e+01 9.733e+01 1.236e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 21:51:31,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383200
2023-11-23 21:51:34,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2554660.0, ans=0.0
2023-11-23 21:51:44,879 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0
2023-11-23 21:51:49,499 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.90 vs. limit=6.0
2023-11-23 21:51:51,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2554726.6666666665, ans=0.0
2023-11-23 21:51:51,721 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.49 vs. limit=15.0
2023-11-23 21:52:15,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2554860.0, ans=0.07
2023-11-23 21:52:16,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2554860.0, ans=0.04949747468305833
2023-11-23 21:52:22,447 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10500, loss[loss=0.07546, simple_loss=0.0971, pruned_loss=0.01829, audio_tagging_loss=0.008619, over 15421.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09119, pruned_loss=0.01346, audio_tagging_loss=0.009192, over 3056051.49 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:52:25,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2554926.6666666665, ans=0.2
2023-11-23 21:52:27,827 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.11 vs. limit=22.5
2023-11-23 21:52:29,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2554926.6666666665, ans=0.0
2023-11-23 21:52:33,225 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.46 vs. limit=22.5
2023-11-23 21:52:33,853 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383250
2023-11-23 21:52:56,211 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:52:56,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2555060.0, ans=0.0
2023-11-23 21:53:14,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2555193.3333333335, ans=0.125
2023-11-23 21:53:24,429 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10550, loss[loss=0.05629, simple_loss=0.07591, pruned_loss=0.007908, audio_tagging_loss=0.01042, over 15949.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09103, pruned_loss=0.01335, audio_tagging_loss=0.009068, over 3051489.81 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:53:24,939 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=17.58 vs. limit=22.5
2023-11-23 21:53:31,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2555260.0, ans=0.2
2023-11-23 21:53:34,351 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.069e+01 8.236e+01 8.886e+01 9.649e+01 1.811e+02, threshold=1.777e+02, percent-clipped=1.0
2023-11-23 21:53:36,275 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383300
2023-11-23 21:53:37,714 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:53:42,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2555326.6666666665, ans=0.1
2023-11-23 21:53:48,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2555393.3333333335, ans=0.125
2023-11-23 21:53:57,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2555393.3333333335, ans=0.04949747468305833
2023-11-23 21:54:07,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2555460.0, ans=0.125
2023-11-23 21:54:07,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2555460.0, ans=0.2
2023-11-23 21:54:16,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2555526.6666666665, ans=0.1
2023-11-23 21:54:26,952 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10600, loss[loss=0.0884, simple_loss=0.1182, pruned_loss=0.02234, audio_tagging_loss=0.006952, over 15756.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09092, pruned_loss=0.01321, audio_tagging_loss=0.008979, over 3058287.96 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:54:38,497 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383350
2023-11-23 21:54:40,491 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.01 vs. limit=15.0
2023-11-23 21:54:59,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2555726.6666666665, ans=0.0
2023-11-23 21:55:19,177 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.72 vs. limit=15.0
2023-11-23 21:55:29,648 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10650, loss[loss=0.06167, simple_loss=0.08609, pruned_loss=0.009292, audio_tagging_loss=0.009328, over 14524.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09147, pruned_loss=0.01338, audio_tagging_loss=0.008923, over 3059664.57 frames. ], batch size: 55, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:55:38,962 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.266e+01 8.595e+01 9.053e+01 9.824e+01 1.350e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-23 21:55:40,279 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383400
2023-11-23 21:55:45,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2555993.3333333335, ans=0.1
2023-11-23 21:55:53,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2556060.0, ans=0.0
2023-11-23 21:55:54,787 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2556060.0, ans=0.0
2023-11-23 21:56:12,676 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 21:56:20,258 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.04 vs. limit=15.0
2023-11-23 21:56:25,963 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.79 vs. limit=15.0
2023-11-23 21:56:31,446 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10700, loss[loss=0.07296, simple_loss=0.1074, pruned_loss=0.01337, audio_tagging_loss=0.005907, over 15716.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09141, pruned_loss=0.01337, audio_tagging_loss=0.008956, over 3059819.23 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:56:43,595 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383450
2023-11-23 21:56:56,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2556393.3333333335, ans=0.2
2023-11-23 21:57:14,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2556460.0, ans=0.125
2023-11-23 21:57:19,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2556460.0, ans=0.0
2023-11-23 21:57:31,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2556526.6666666665, ans=0.2
2023-11-23 21:57:35,206 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10750, loss[loss=0.07189, simple_loss=0.09719, pruned_loss=0.01302, audio_tagging_loss=0.01027, over 15622.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09154, pruned_loss=0.01336, audio_tagging_loss=0.008965, over 3060908.12 frames. ], batch size: 59, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:57:37,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2556593.3333333335, ans=0.0
2023-11-23 21:57:43,151 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.42 vs. limit=22.5
2023-11-23 21:57:45,137 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.476e+01 9.127e+01 1.007e+02 1.402e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 21:57:46,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383500
2023-11-23 21:57:57,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2556660.0, ans=0.1
2023-11-23 21:58:06,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2556726.6666666665, ans=0.0
2023-11-23 21:58:09,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2556726.6666666665, ans=0.1
2023-11-23 21:58:18,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2556793.3333333335, ans=0.125
2023-11-23 21:58:22,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2556793.3333333335, ans=0.125
2023-11-23 21:58:37,465 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10800, loss[loss=0.06717, simple_loss=0.09773, pruned_loss=0.01307, audio_tagging_loss=0.005229, over 15156.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09163, pruned_loss=0.01335, audio_tagging_loss=0.008921, over 3059604.83 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 21:58:42,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2556926.6666666665, ans=0.125
2023-11-23 21:58:48,252 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383550
2023-11-23 21:58:53,313 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.99 vs. limit=15.0
2023-11-23 21:58:57,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2556993.3333333335, ans=0.035
2023-11-23 21:59:03,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2557060.0, ans=0.125
2023-11-23 21:59:05,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2557060.0, ans=0.125
2023-11-23 21:59:21,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2557126.6666666665, ans=0.125
2023-11-23 21:59:37,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2557193.3333333335, ans=15.0
2023-11-23 21:59:38,846 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10850, loss[loss=0.07485, simple_loss=0.1011, pruned_loss=0.0171, audio_tagging_loss=0.007211, over 15587.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09065, pruned_loss=0.01325, audio_tagging_loss=0.009072, over 3053084.27 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 21:59:49,892 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.971e+01 8.543e+01 9.235e+01 1.001e+02 1.289e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-23 21:59:50,066 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383600
2023-11-23 21:59:52,283 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.36 vs. limit=22.5
2023-11-23 22:00:36,666 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:00:41,470 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10900, loss[loss=0.07542, simple_loss=0.09568, pruned_loss=0.01916, audio_tagging_loss=0.008418, over 13799.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09097, pruned_loss=0.01343, audio_tagging_loss=0.009066, over 3043743.64 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:00:53,319 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383650
2023-11-23 22:01:03,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2557660.0, ans=0.125
2023-11-23 22:01:09,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2557726.6666666665, ans=0.125
2023-11-23 22:01:30,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2557860.0, ans=0.0
2023-11-23 22:01:31,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2557860.0, ans=0.125
2023-11-23 22:01:35,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2557860.0, ans=0.125
2023-11-23 22:01:44,020 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 10950, loss[loss=0.07033, simple_loss=0.09576, pruned_loss=0.01413, audio_tagging_loss=0.008327, over 15956.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.0913, pruned_loss=0.0134, audio_tagging_loss=0.008959, over 3048556.19 frames. ], batch size: 59, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:01:44,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2557926.6666666665, ans=0.125
2023-11-23 22:01:54,644 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.274e+01 8.562e+01 9.166e+01 9.897e+01 1.361e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-23 22:01:54,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383700
2023-11-23 22:02:06,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2558060.0, ans=0.0
2023-11-23 22:02:11,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2558060.0, ans=0.125
2023-11-23 22:02:30,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2558126.6666666665, ans=0.125
2023-11-23 22:02:31,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2558126.6666666665, ans=0.1
2023-11-23 22:02:36,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2558193.3333333335, ans=0.125
2023-11-23 22:02:37,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2558193.3333333335, ans=0.125
2023-11-23 22:02:44,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2558260.0, ans=0.0
2023-11-23 22:02:45,435 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11000, loss[loss=0.09266, simple_loss=0.1306, pruned_loss=0.02138, audio_tagging_loss=0.005982, over 15504.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09176, pruned_loss=0.01339, audio_tagging_loss=0.00905, over 3048861.12 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:02:46,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2558260.0, ans=0.125
2023-11-23 22:02:46,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2558260.0, ans=0.1
2023-11-23 22:02:51,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2558260.0, ans=0.0
2023-11-23 22:02:53,619 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:02:56,600 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383750
2023-11-23 22:03:04,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2558326.6666666665, ans=0.125
2023-11-23 22:03:34,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2558526.6666666665, ans=0.125
2023-11-23 22:03:47,002 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11050, loss[loss=0.05572, simple_loss=0.0649, pruned_loss=0.01036, audio_tagging_loss=0.01291, over 15766.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09231, pruned_loss=0.01367, audio_tagging_loss=0.009076, over 3058545.79 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:03:58,693 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.213e+01 8.515e+01 9.099e+01 9.905e+01 1.187e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-23 22:03:58,861 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383800
2023-11-23 22:04:03,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2558660.0, ans=0.09899494936611666
2023-11-23 22:04:03,923 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.40 vs. limit=15.0
2023-11-23 22:04:07,176 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:04:27,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2558793.3333333335, ans=15.0
2023-11-23 22:04:28,561 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.92 vs. limit=15.0
2023-11-23 22:04:33,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2558793.3333333335, ans=0.125
2023-11-23 22:04:35,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2558860.0, ans=0.0
2023-11-23 22:04:45,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2558860.0, ans=0.09899494936611666
2023-11-23 22:04:50,544 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11100, loss[loss=0.07996, simple_loss=0.1084, pruned_loss=0.01561, audio_tagging_loss=0.01014, over 15761.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09169, pruned_loss=0.01351, audio_tagging_loss=0.009155, over 3063367.74 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:04:53,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2558926.6666666665, ans=0.125
2023-11-23 22:04:55,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2558926.6666666665, ans=0.07
2023-11-23 22:05:01,196 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383850
2023-11-23 22:05:40,725 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.33 vs. limit=10.0
2023-11-23 22:05:51,956 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11150, loss[loss=0.04742, simple_loss=0.05204, pruned_loss=0.008276, audio_tagging_loss=0.01312, over 14940.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.09225, pruned_loss=0.01369, audio_tagging_loss=0.009191, over 3065481.11 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:05:53,906 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.51 vs. limit=6.0
2023-11-23 22:06:02,624 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.354e+01 8.926e+01 9.504e+01 1.198e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 22:06:02,787 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383900
2023-11-23 22:06:08,097 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:06:09,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_abs, batch_count=2559326.6666666665, ans=0.5
2023-11-23 22:06:33,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2559460.0, ans=0.125
2023-11-23 22:06:34,470 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.06 vs. limit=6.0
2023-11-23 22:06:53,541 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11200, loss[loss=0.08329, simple_loss=0.1054, pruned_loss=0.02247, audio_tagging_loss=0.008137, over 15143.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09219, pruned_loss=0.01378, audio_tagging_loss=0.009186, over 3064007.80 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:06:58,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2559593.3333333335, ans=0.125
2023-11-23 22:07:00,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2559593.3333333335, ans=0.125
2023-11-23 22:07:02,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2559593.3333333335, ans=0.0
2023-11-23 22:07:04,934 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 383950
2023-11-23 22:07:06,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2559660.0, ans=0.125
2023-11-23 22:07:16,106 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.44 vs. limit=15.0
2023-11-23 22:07:20,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2559726.6666666665, ans=0.1
2023-11-23 22:07:21,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2559726.6666666665, ans=0.0
2023-11-23 22:07:28,153 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.45 vs. limit=15.0
2023-11-23 22:07:32,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2559793.3333333335, ans=0.125
2023-11-23 22:07:50,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2559860.0, ans=0.125
2023-11-23 22:07:55,768 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11250, loss[loss=0.04568, simple_loss=0.05761, pruned_loss=0.007323, audio_tagging_loss=0.009555, over 15462.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09124, pruned_loss=0.01372, audio_tagging_loss=0.009164, over 3055865.00 frames. ], batch size: 60, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:08:05,872 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.65 vs. limit=15.0
2023-11-23 22:08:06,084 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.98 vs. limit=15.0
2023-11-23 22:08:07,490 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.817e+01 8.428e+01 9.033e+01 9.633e+01 1.168e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-23 22:08:07,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384000
2023-11-23 22:08:09,083 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-384000.pt
2023-11-23 22:08:13,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2559993.3333333335, ans=0.125
2023-11-23 22:08:20,635 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.28 vs. limit=15.0
2023-11-23 22:08:22,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2559993.3333333335, ans=0.125
2023-11-23 22:08:23,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2560060.0, ans=0.125
2023-11-23 22:08:55,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2560193.3333333335, ans=0.125
2023-11-23 22:09:02,412 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11300, loss[loss=0.06194, simple_loss=0.07869, pruned_loss=0.01319, audio_tagging_loss=0.009401, over 14630.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09101, pruned_loss=0.01377, audio_tagging_loss=0.009147, over 3050023.19 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:09:13,320 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384050
2023-11-23 22:09:20,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2560326.6666666665, ans=0.125
2023-11-23 22:09:23,978 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.90 vs. limit=15.0
2023-11-23 22:09:43,366 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.26 vs. limit=15.0
2023-11-23 22:10:03,846 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.27 vs. limit=15.0
2023-11-23 22:10:04,297 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11350, loss[loss=0.07615, simple_loss=0.1089, pruned_loss=0.01483, audio_tagging_loss=0.006864, over 15253.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09176, pruned_loss=0.01385, audio_tagging_loss=0.00907, over 3057494.36 frames. ], batch size: 54, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:10:16,254 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.699e+01 8.206e+01 9.003e+01 9.537e+01 1.148e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-23 22:10:16,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384100
2023-11-23 22:10:20,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2560660.0, ans=0.125
2023-11-23 22:10:26,868 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2560660.0, ans=0.0
2023-11-23 22:10:29,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2560726.6666666665, ans=0.125
2023-11-23 22:10:42,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2560793.3333333335, ans=0.0
2023-11-23 22:10:43,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2560793.3333333335, ans=0.125
2023-11-23 22:10:45,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2560793.3333333335, ans=0.125
2023-11-23 22:10:57,897 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.39 vs. limit=12.0
2023-11-23 22:11:02,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2560860.0, ans=0.125
2023-11-23 22:11:05,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2560860.0, ans=0.125
2023-11-23 22:11:07,783 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11400, loss[loss=0.07012, simple_loss=0.09652, pruned_loss=0.01411, audio_tagging_loss=0.007747, over 16903.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09219, pruned_loss=0.0139, audio_tagging_loss=0.008917, over 3048053.89 frames. ], batch size: 66, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:11:16,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2560926.6666666665, ans=0.0
2023-11-23 22:11:19,642 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384150
2023-11-23 22:11:49,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2561126.6666666665, ans=0.0
2023-11-23 22:11:51,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2561126.6666666665, ans=0.125
2023-11-23 22:12:00,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2561193.3333333335, ans=0.035
2023-11-23 22:12:05,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2561193.3333333335, ans=0.125
2023-11-23 22:12:06,820 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.75 vs. limit=15.0
2023-11-23 22:12:09,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2561260.0, ans=0.125
2023-11-23 22:12:10,327 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11450, loss[loss=0.05616, simple_loss=0.07917, pruned_loss=0.008881, audio_tagging_loss=0.007694, over 15026.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09324, pruned_loss=0.01401, audio_tagging_loss=0.008848, over 3048114.50 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:12:11,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2561260.0, ans=0.125
2023-11-23 22:12:16,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2561260.0, ans=0.2
2023-11-23 22:12:21,212 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384200
2023-11-23 22:12:22,238 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.686e+01 8.398e+01 8.984e+01 9.548e+01 1.185e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-23 22:12:36,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2561393.3333333335, ans=0.025
2023-11-23 22:12:40,771 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.89 vs. limit=15.0
2023-11-23 22:12:47,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2561460.0, ans=0.125
2023-11-23 22:12:54,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2561460.0, ans=0.125
2023-11-23 22:12:57,600 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.19 vs. limit=15.0
2023-11-23 22:13:06,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2561526.6666666665, ans=0.125
2023-11-23 22:13:11,994 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11500, loss[loss=0.0536, simple_loss=0.06592, pruned_loss=0.01074, audio_tagging_loss=0.009898, over 14857.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09203, pruned_loss=0.01372, audio_tagging_loss=0.008945, over 3046466.32 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:13:23,223 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384250
2023-11-23 22:13:29,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2561660.0, ans=0.1
2023-11-23 22:13:32,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.66 vs. limit=12.0
2023-11-23 22:13:49,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2561793.3333333335, ans=0.2
2023-11-23 22:14:13,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2561926.6666666665, ans=0.125
2023-11-23 22:14:14,458 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11550, loss[loss=0.06736, simple_loss=0.09041, pruned_loss=0.01276, audio_tagging_loss=0.009396, over 15300.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09205, pruned_loss=0.01368, audio_tagging_loss=0.008981, over 3047758.41 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 16.0
2023-11-23 22:14:14,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2561926.6666666665, ans=0.125
2023-11-23 22:14:15,917 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:14:21,962 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:14:22,481 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.62 vs. limit=15.0
2023-11-23 22:14:25,760 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384300
2023-11-23 22:14:26,805 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.811e+01 8.220e+01 8.896e+01 9.591e+01 1.198e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-23 22:14:48,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2562060.0, ans=0.1
2023-11-23 22:14:50,984 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:15:08,742 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:15:16,228 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11600, loss[loss=0.07733, simple_loss=0.1122, pruned_loss=0.01292, audio_tagging_loss=0.008287, over 15722.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09268, pruned_loss=0.01382, audio_tagging_loss=0.008907, over 3044517.94 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:15:27,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384350
2023-11-23 22:15:41,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2562393.3333333335, ans=0.1
2023-11-23 22:16:09,143 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.77 vs. limit=15.0
2023-11-23 22:16:18,759 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11650, loss[loss=0.07724, simple_loss=0.1086, pruned_loss=0.01606, audio_tagging_loss=0.006901, over 16489.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09288, pruned_loss=0.01385, audio_tagging_loss=0.008898, over 3050439.56 frames. ], batch size: 58, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:16:30,026 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384400
2023-11-23 22:16:31,040 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.399e+01 8.387e+01 8.924e+01 9.715e+01 1.157e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-23 22:16:33,201 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.92 vs. limit=15.0
2023-11-23 22:16:38,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2562660.0, ans=0.125
2023-11-23 22:16:43,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2562726.6666666665, ans=0.1
2023-11-23 22:17:15,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2562860.0, ans=0.1
2023-11-23 22:17:22,178 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11700, loss[loss=0.0742, simple_loss=0.1086, pruned_loss=0.01347, audio_tagging_loss=0.006422, over 15584.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09183, pruned_loss=0.01367, audio_tagging_loss=0.009021, over 3048422.07 frames. ], batch size: 57, lr: 2.11e-03, grad_scale: 32.0
2023-11-23 22:17:33,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384450
2023-11-23 22:17:37,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2562993.3333333335, ans=0.125
2023-11-23 22:17:38,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2562993.3333333335, ans=0.0
2023-11-23 22:17:43,418 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.47 vs. limit=15.0
2023-11-23 22:17:54,758 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.18 vs. limit=15.0
2023-11-23 22:18:18,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2563193.3333333335, ans=0.125
2023-11-23 22:18:18,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2563193.3333333335, ans=0.1
2023-11-23 22:18:24,604 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11750, loss[loss=0.04894, simple_loss=0.06905, pruned_loss=0.007461, audio_tagging_loss=0.006955, over 15125.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09169, pruned_loss=0.0136, audio_tagging_loss=0.009002, over 3056683.63 frames. ], batch size: 57, lr: 2.10e-03, grad_scale: 32.0
2023-11-23 22:18:26,376 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.10 vs. limit=15.0
2023-11-23 22:18:35,393 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384500
2023-11-23 22:18:36,427 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.506e+01 8.915e+01 9.648e+01 1.548e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-23 22:18:53,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2563393.3333333335, ans=0.125
2023-11-23 22:19:26,096 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11800, loss[loss=0.07602, simple_loss=0.1053, pruned_loss=0.01603, audio_tagging_loss=0.007338, over 15435.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09123, pruned_loss=0.01378, audio_tagging_loss=0.009067, over 3050320.76 frames. ], batch size: 55, lr: 2.10e-03, grad_scale: 32.0
2023-11-23 22:19:27,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2563593.3333333335, ans=0.2
2023-11-23 22:19:28,535 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.38 vs. limit=15.0
2023-11-23 22:19:30,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2563593.3333333335, ans=0.125
2023-11-23 22:19:34,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2563593.3333333335, ans=0.04949747468305833
2023-11-23 22:19:37,930 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384550
2023-11-23 22:19:38,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.63 vs. limit=6.0
2023-11-23 22:20:00,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2563726.6666666665, ans=0.1
2023-11-23 22:20:12,536 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.56 vs. limit=15.0
2023-11-23 22:20:28,404 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11850, loss[loss=0.07761, simple_loss=0.104, pruned_loss=0.01734, audio_tagging_loss=0.008292, over 16208.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09137, pruned_loss=0.01382, audio_tagging_loss=0.009161, over 3046797.66 frames. ], batch size: 60, lr: 2.10e-03, grad_scale: 32.0
2023-11-23 22:20:34,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2563926.6666666665, ans=0.05
2023-11-23 22:20:38,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2563926.6666666665, ans=0.2
2023-11-23 22:20:39,736 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384600
2023-11-23 22:20:39,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2563993.3333333335, ans=0.125
2023-11-23 22:20:41,327 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.029e+01 8.384e+01 8.994e+01 9.780e+01 1.224e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 22:20:57,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2564060.0, ans=0.125
2023-11-23 22:21:21,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2564193.3333333335, ans=0.2
2023-11-23 22:21:31,123 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11900, loss[loss=0.08503, simple_loss=0.1152, pruned_loss=0.018, audio_tagging_loss=0.009427, over 15244.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09114, pruned_loss=0.01359, audio_tagging_loss=0.009276, over 3043994.57 frames. ], batch size: 55, lr: 2.10e-03, grad_scale: 16.0
2023-11-23 22:21:41,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384650
2023-11-23 22:21:46,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2564326.6666666665, ans=0.125
2023-11-23 22:21:48,437 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.81 vs. limit=15.0
2023-11-23 22:21:52,036 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.66 vs. limit=15.0
2023-11-23 22:21:56,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2564393.3333333335, ans=0.2
2023-11-23 22:22:03,493 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2564393.3333333335, ans=0.125
2023-11-23 22:22:06,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2564393.3333333335, ans=0.0
2023-11-23 22:22:10,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2564460.0, ans=0.2
2023-11-23 22:22:32,791 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 11950, loss[loss=0.07063, simple_loss=0.08856, pruned_loss=0.01568, audio_tagging_loss=0.01067, over 14848.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09114, pruned_loss=0.01366, audio_tagging_loss=0.00938, over 3041848.82 frames. ], batch size: 56, lr: 2.10e-03, grad_scale: 16.0
2023-11-23 22:22:36,803 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.06 vs. limit=15.0
2023-11-23 22:22:43,927 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384700
2023-11-23 22:22:46,155 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.381e+01 9.044e+01 9.786e+01 1.119e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-23 22:23:06,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2564726.6666666665, ans=0.04949747468305833
2023-11-23 22:23:06,672 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.60 vs. limit=15.0
2023-11-23 22:23:28,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2564860.0, ans=0.0
2023-11-23 22:23:32,159 INFO [train_asr.py:1221] (0/4) Epoch 32, batch 12000, loss[loss=0.07775, simple_loss=0.1088, pruned_loss=0.01366, audio_tagging_loss=0.009682, over 15286.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09169, pruned_loss=0.01374, audio_tagging_loss=0.009414, over 3047038.03 frames. ], batch size: 57, lr: 2.10e-03, grad_scale: 32.0
2023-11-23 22:23:32,162 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 22:24:14,280 INFO [train_asr.py:1253] (0/4) Epoch 32, validation: loss=0.05848, simple_loss=0.0511, pruned_loss=0.005239, audio_tagging_loss=0.02769, over 4681554.00 frames. 
2023-11-23 22:24:14,281 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 22:24:15,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2564926.6666666665, ans=0.125
2023-11-23 22:24:20,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2564926.6666666665, ans=0.0
2023-11-23 22:24:24,965 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384750
2023-11-23 22:24:26,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2564993.3333333335, ans=0.0
2023-11-23 22:24:43,039 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-32.pt
2023-11-23 22:25:13,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2565086.6666666665, ans=0.1
2023-11-23 22:25:15,140 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 0, loss[loss=0.08127, simple_loss=0.09033, pruned_loss=0.01587, audio_tagging_loss=0.02024, over 16087.00 frames. ], tot_loss[loss=0.08127, simple_loss=0.09033, pruned_loss=0.01587, audio_tagging_loss=0.02024, over 16087.00 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:25:15,143 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 22:25:50,707 INFO [train_asr.py:1253] (0/4) Epoch 33, validation: loss=0.05781, simple_loss=0.05104, pruned_loss=0.005203, audio_tagging_loss=0.02709, over 4681554.00 frames. 
2023-11-23 22:25:50,708 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 22:26:10,601 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.00 vs. limit=15.0
2023-11-23 22:26:10,632 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0
2023-11-23 22:26:19,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2565220.0, ans=0.125
2023-11-23 22:26:27,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2565286.6666666665, ans=0.0
2023-11-23 22:26:34,531 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384800
2023-11-23 22:26:36,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2565286.6666666665, ans=0.1
2023-11-23 22:26:38,155 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.158e+01 8.944e+01 9.729e+01 1.034e+02 1.354e+02, threshold=1.946e+02, percent-clipped=0.0
2023-11-23 22:26:43,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2565353.3333333335, ans=0.125
2023-11-23 22:26:52,496 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 50, loss[loss=0.06971, simple_loss=0.08013, pruned_loss=0.01041, audio_tagging_loss=0.01923, over 15189.00 frames. ], tot_loss[loss=0.07553, simple_loss=0.08967, pruned_loss=0.013, audio_tagging_loss=0.0177, over 691043.77 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:26:55,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2565420.0, ans=0.0
2023-11-23 22:27:10,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2565486.6666666665, ans=0.0
2023-11-23 22:27:22,261 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:27:36,582 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384850
2023-11-23 22:27:49,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2565686.6666666665, ans=0.125
2023-11-23 22:27:51,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2565686.6666666665, ans=0.1
2023-11-23 22:27:52,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2565686.6666666665, ans=0.1
2023-11-23 22:27:55,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2565753.3333333335, ans=0.125
2023-11-23 22:27:56,911 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 100, loss[loss=0.06673, simple_loss=0.0852, pruned_loss=0.01054, audio_tagging_loss=0.01359, over 14890.00 frames. ], tot_loss[loss=0.0746, simple_loss=0.08967, pruned_loss=0.01299, audio_tagging_loss=0.01678, over 1213455.12 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:28:00,111 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.17 vs. limit=15.0
2023-11-23 22:28:31,736 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.22 vs. limit=15.0
2023-11-23 22:28:38,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384900
2023-11-23 22:28:43,526 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.457e+01 8.971e+01 9.700e+01 1.041e+02 1.375e+02, threshold=1.940e+02, percent-clipped=0.0
2023-11-23 22:28:53,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2566020.0, ans=6.0
2023-11-23 22:28:57,731 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 150, loss[loss=0.05734, simple_loss=0.07535, pruned_loss=0.006623, audio_tagging_loss=0.01305, over 13828.00 frames. ], tot_loss[loss=0.07364, simple_loss=0.09045, pruned_loss=0.01328, audio_tagging_loss=0.01513, over 1622862.55 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:28:58,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2566086.6666666665, ans=0.1
2023-11-23 22:29:05,400 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.09 vs. limit=10.0
2023-11-23 22:29:05,451 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.93 vs. limit=15.0
2023-11-23 22:29:14,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2566153.3333333335, ans=0.125
2023-11-23 22:29:19,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.42 vs. limit=22.5
2023-11-23 22:29:28,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2566220.0, ans=0.2
2023-11-23 22:29:31,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2566220.0, ans=0.125
2023-11-23 22:29:39,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2566286.6666666665, ans=0.09899494936611666
2023-11-23 22:29:41,537 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 384950
2023-11-23 22:29:52,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2566353.3333333335, ans=0.125
2023-11-23 22:29:59,071 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 200, loss[loss=0.06018, simple_loss=0.07898, pruned_loss=0.0101, audio_tagging_loss=0.01059, over 15158.00 frames. ], tot_loss[loss=0.07337, simple_loss=0.0924, pruned_loss=0.01388, audio_tagging_loss=0.01329, over 1936816.96 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:30:02,937 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:30:20,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2566486.6666666665, ans=0.0
2023-11-23 22:30:32,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2566553.3333333335, ans=0.1
2023-11-23 22:30:37,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2566620.0, ans=0.125
2023-11-23 22:30:37,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2566620.0, ans=0.125
2023-11-23 22:30:42,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385000
2023-11-23 22:30:46,068 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.668e+01 9.124e+01 1.011e+02 1.346e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-23 22:30:55,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2566686.6666666665, ans=0.125
2023-11-23 22:30:59,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2566753.3333333335, ans=0.0
2023-11-23 22:30:59,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2566753.3333333335, ans=0.05
2023-11-23 22:31:01,367 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 250, loss[loss=0.06808, simple_loss=0.09165, pruned_loss=0.01283, audio_tagging_loss=0.009426, over 15242.00 frames. ], tot_loss[loss=0.07205, simple_loss=0.09244, pruned_loss=0.01392, audio_tagging_loss=0.01191, over 2181502.20 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:31:04,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2566753.3333333335, ans=10.0
2023-11-23 22:31:09,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2566753.3333333335, ans=15.0
2023-11-23 22:31:18,482 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.11 vs. limit=22.5
2023-11-23 22:31:21,982 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.94 vs. limit=15.0
2023-11-23 22:31:43,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2566953.3333333335, ans=0.125
2023-11-23 22:31:44,968 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385050
2023-11-23 22:31:57,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2567020.0, ans=0.125
2023-11-23 22:32:04,241 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 300, loss[loss=0.0779, simple_loss=0.09994, pruned_loss=0.01797, audio_tagging_loss=0.00996, over 15272.00 frames. ], tot_loss[loss=0.07063, simple_loss=0.09173, pruned_loss=0.01361, audio_tagging_loss=0.01115, over 2372266.19 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:32:09,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2567086.6666666665, ans=0.1
2023-11-23 22:32:16,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2567153.3333333335, ans=0.125
2023-11-23 22:32:28,614 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=14.69 vs. limit=15.0
2023-11-23 22:32:30,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2567220.0, ans=0.2
2023-11-23 22:32:47,986 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385100
2023-11-23 22:32:51,312 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.686e+01 8.804e+01 9.384e+01 1.010e+02 1.261e+02, threshold=1.877e+02, percent-clipped=0.0
2023-11-23 22:32:53,970 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:33:05,398 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 350, loss[loss=0.0635, simple_loss=0.07975, pruned_loss=0.01307, audio_tagging_loss=0.01056, over 15340.00 frames. ], tot_loss[loss=0.06998, simple_loss=0.09177, pruned_loss=0.01355, audio_tagging_loss=0.01054, over 2520931.61 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:33:15,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2567420.0, ans=0.05
2023-11-23 22:33:49,649 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385150
2023-11-23 22:33:53,392 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:34:00,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2567686.6666666665, ans=0.125
2023-11-23 22:34:07,552 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 400, loss[loss=0.07109, simple_loss=0.09761, pruned_loss=0.01343, audio_tagging_loss=0.008853, over 15613.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09129, pruned_loss=0.01354, audio_tagging_loss=0.01021, over 2637944.94 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:34:11,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2567753.3333333335, ans=0.0
2023-11-23 22:34:22,931 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.01 vs. limit=12.0
2023-11-23 22:34:32,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2567886.6666666665, ans=0.125
2023-11-23 22:34:44,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2567953.3333333335, ans=0.1
2023-11-23 22:34:48,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2567953.3333333335, ans=0.125
2023-11-23 22:34:51,236 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385200
2023-11-23 22:34:51,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2567953.3333333335, ans=0.125
2023-11-23 22:34:54,936 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.173e+01 8.325e+01 8.873e+01 9.611e+01 1.385e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-23 22:35:11,091 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 450, loss[loss=0.07684, simple_loss=0.1088, pruned_loss=0.01434, audio_tagging_loss=0.008102, over 16182.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09241, pruned_loss=0.01358, audio_tagging_loss=0.009891, over 2733091.18 frames. ], batch size: 60, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:35:32,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2568153.3333333335, ans=0.0
2023-11-23 22:35:34,260 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.89 vs. limit=15.0
2023-11-23 22:35:37,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2568220.0, ans=0.125
2023-11-23 22:35:42,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2568220.0, ans=0.125
2023-11-23 22:35:43,047 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.29 vs. limit=15.0
2023-11-23 22:35:53,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2568286.6666666665, ans=0.125
2023-11-23 22:35:54,331 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385250
2023-11-23 22:35:56,291 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0
2023-11-23 22:35:57,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2568286.6666666665, ans=0.125
2023-11-23 22:35:59,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2568353.3333333335, ans=0.0
2023-11-23 22:36:12,792 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 500, loss[loss=0.06245, simple_loss=0.07951, pruned_loss=0.01179, audio_tagging_loss=0.0109, over 15781.00 frames. ], tot_loss[loss=0.06913, simple_loss=0.09194, pruned_loss=0.01346, audio_tagging_loss=0.009694, over 2811632.39 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:36:28,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2568486.6666666665, ans=0.125
2023-11-23 22:36:57,253 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385300
2023-11-23 22:37:00,578 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.320e+01 8.409e+01 9.142e+01 9.799e+01 1.482e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 22:37:06,153 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.56 vs. limit=22.5
2023-11-23 22:37:10,484 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:37:15,578 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 550, loss[loss=0.0593, simple_loss=0.07168, pruned_loss=0.01293, audio_tagging_loss=0.01053, over 15476.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09212, pruned_loss=0.01343, audio_tagging_loss=0.009574, over 2864261.15 frames. ], batch size: 59, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:37:32,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2568820.0, ans=0.125
2023-11-23 22:37:58,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385350
2023-11-23 22:37:58,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2568953.3333333335, ans=0.0
2023-11-23 22:38:04,485 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.87 vs. limit=15.0
2023-11-23 22:38:05,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2569020.0, ans=0.0
2023-11-23 22:38:07,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2569020.0, ans=0.09899494936611666
2023-11-23 22:38:11,473 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.08 vs. limit=12.0
2023-11-23 22:38:17,973 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 600, loss[loss=0.07308, simple_loss=0.1001, pruned_loss=0.01482, audio_tagging_loss=0.008181, over 15790.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09116, pruned_loss=0.01324, audio_tagging_loss=0.009462, over 2908962.07 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:38:34,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2569153.3333333335, ans=0.0
2023-11-23 22:39:00,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2569286.6666666665, ans=0.125
2023-11-23 22:39:01,709 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385400
2023-11-23 22:39:06,580 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.846e+01 8.472e+01 9.172e+01 9.611e+01 1.258e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 22:39:08,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2569353.3333333335, ans=0.125
2023-11-23 22:39:09,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2569353.3333333335, ans=0.1
2023-11-23 22:39:18,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2569353.3333333335, ans=0.125
2023-11-23 22:39:20,481 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 650, loss[loss=0.08134, simple_loss=0.1197, pruned_loss=0.01627, audio_tagging_loss=0.005205, over 17040.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09083, pruned_loss=0.01316, audio_tagging_loss=0.009458, over 2947550.67 frames. ], batch size: 61, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:39:21,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2569420.0, ans=0.04949747468305833
2023-11-23 22:39:32,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.22 vs. limit=15.0
2023-11-23 22:39:51,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2569553.3333333335, ans=0.125
2023-11-23 22:40:03,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2569620.0, ans=0.125
2023-11-23 22:40:03,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2569620.0, ans=0.125
2023-11-23 22:40:04,688 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385450
2023-11-23 22:40:09,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2569686.6666666665, ans=0.125
2023-11-23 22:40:22,210 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 700, loss[loss=0.08758, simple_loss=0.1227, pruned_loss=0.01766, audio_tagging_loss=0.008572, over 15454.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09063, pruned_loss=0.0131, audio_tagging_loss=0.009441, over 2965867.91 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:40:22,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2569753.3333333335, ans=0.125
2023-11-23 22:40:43,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2569820.0, ans=0.1
2023-11-23 22:41:06,042 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385500
2023-11-23 22:41:11,046 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.418e+01 8.479e+01 9.169e+01 9.632e+01 1.119e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-23 22:41:16,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2570020.0, ans=0.07
2023-11-23 22:41:25,259 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 750, loss[loss=0.07782, simple_loss=0.1073, pruned_loss=0.01571, audio_tagging_loss=0.008448, over 15910.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.0916, pruned_loss=0.0132, audio_tagging_loss=0.009272, over 2986081.10 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:41:30,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2570086.6666666665, ans=0.04949747468305833
2023-11-23 22:41:53,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2570220.0, ans=10.0
2023-11-23 22:42:09,690 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385550
2023-11-23 22:42:27,203 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 800, loss[loss=0.06893, simple_loss=0.09408, pruned_loss=0.0138, audio_tagging_loss=0.008096, over 14790.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09195, pruned_loss=0.01347, audio_tagging_loss=0.009287, over 2999717.77 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:42:43,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2570486.6666666665, ans=0.0
2023-11-23 22:42:51,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2570553.3333333335, ans=0.0
2023-11-23 22:42:57,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2570553.3333333335, ans=0.125
2023-11-23 22:43:11,306 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385600
2023-11-23 22:43:16,328 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.424e+01 8.735e+01 9.399e+01 1.001e+02 1.314e+02, threshold=1.880e+02, percent-clipped=0.0
2023-11-23 22:43:21,237 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.17 vs. limit=15.0
2023-11-23 22:43:30,066 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 850, loss[loss=0.0771, simple_loss=0.1069, pruned_loss=0.01583, audio_tagging_loss=0.007819, over 15932.00 frames. ], tot_loss[loss=0.06911, simple_loss=0.09241, pruned_loss=0.01357, audio_tagging_loss=0.00934, over 3014330.04 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:43:38,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2570753.3333333335, ans=0.1
2023-11-23 22:43:46,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2570820.0, ans=0.2
2023-11-23 22:43:48,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2570820.0, ans=0.1
2023-11-23 22:43:51,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2570820.0, ans=0.125
2023-11-23 22:44:14,388 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385650
2023-11-23 22:44:20,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2571020.0, ans=0.1
2023-11-23 22:44:23,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2571020.0, ans=0.125
2023-11-23 22:44:28,164 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.24 vs. limit=6.0
2023-11-23 22:44:33,796 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 900, loss[loss=0.07158, simple_loss=0.0923, pruned_loss=0.01241, audio_tagging_loss=0.01302, over 16084.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.09231, pruned_loss=0.01355, audio_tagging_loss=0.009431, over 3019093.77 frames. ], batch size: 63, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:44:34,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2571086.6666666665, ans=0.2
2023-11-23 22:44:41,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2571086.6666666665, ans=0.1
2023-11-23 22:44:45,349 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.43 vs. limit=10.0
2023-11-23 22:44:51,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2571153.3333333335, ans=0.125
2023-11-23 22:45:13,583 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.77 vs. limit=15.0
2023-11-23 22:45:17,168 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385700
2023-11-23 22:45:22,414 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.319e+01 9.000e+01 1.000e+02 1.242e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-23 22:45:35,459 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 950, loss[loss=0.0735, simple_loss=0.1009, pruned_loss=0.01419, audio_tagging_loss=0.00885, over 15105.00 frames. ], tot_loss[loss=0.06914, simple_loss=0.0923, pruned_loss=0.01364, audio_tagging_loss=0.009344, over 3022960.98 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:45:47,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2571486.6666666665, ans=0.125
2023-11-23 22:46:02,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2571553.3333333335, ans=0.0
2023-11-23 22:46:19,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385750
2023-11-23 22:46:33,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2571686.6666666665, ans=0.035
2023-11-23 22:46:37,430 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1000, loss[loss=0.06777, simple_loss=0.08924, pruned_loss=0.01468, audio_tagging_loss=0.008472, over 14169.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.09203, pruned_loss=0.0136, audio_tagging_loss=0.009252, over 3025218.27 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:46:41,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2571753.3333333335, ans=0.125
2023-11-23 22:46:51,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2571820.0, ans=0.125
2023-11-23 22:47:04,039 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:47:07,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2571886.6666666665, ans=0.125
2023-11-23 22:47:20,828 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385800
2023-11-23 22:47:27,681 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.934e+01 8.356e+01 8.922e+01 9.562e+01 1.199e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 22:47:40,648 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1050, loss[loss=0.0603, simple_loss=0.07768, pruned_loss=0.01092, audio_tagging_loss=0.01054, over 15024.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09196, pruned_loss=0.01366, audio_tagging_loss=0.009158, over 3031372.18 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:47:56,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2572153.3333333335, ans=0.2
2023-11-23 22:48:22,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2572286.6666666665, ans=0.2
2023-11-23 22:48:24,475 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385850
2023-11-23 22:48:42,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2572420.0, ans=0.0
2023-11-23 22:48:43,145 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1100, loss[loss=0.06495, simple_loss=0.08456, pruned_loss=0.01314, audio_tagging_loss=0.009537, over 14048.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09214, pruned_loss=0.0137, audio_tagging_loss=0.009017, over 3028791.94 frames. ], batch size: 53, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:48:45,574 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:49:00,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2572486.6666666665, ans=0.2
2023-11-23 22:49:26,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2572620.0, ans=0.125
2023-11-23 22:49:27,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385900
2023-11-23 22:49:33,459 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.142e+01 8.297e+01 8.837e+01 9.387e+01 1.175e+02, threshold=1.767e+02, percent-clipped=0.0
2023-11-23 22:49:42,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2572686.6666666665, ans=0.125
2023-11-23 22:49:45,371 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1150, loss[loss=0.0616, simple_loss=0.07854, pruned_loss=0.009475, audio_tagging_loss=0.01286, over 16180.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09212, pruned_loss=0.01367, audio_tagging_loss=0.008977, over 3034036.11 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:49:52,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2572753.3333333335, ans=0.0
2023-11-23 22:50:00,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2572820.0, ans=0.125
2023-11-23 22:50:12,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.60 vs. limit=15.0
2023-11-23 22:50:23,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2572953.3333333335, ans=0.125
2023-11-23 22:50:29,328 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 385950
2023-11-23 22:50:42,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2573020.0, ans=0.125
2023-11-23 22:50:47,806 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1200, loss[loss=0.06955, simple_loss=0.09341, pruned_loss=0.01471, audio_tagging_loss=0.008134, over 15534.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09233, pruned_loss=0.01357, audio_tagging_loss=0.008911, over 3033766.41 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:50:50,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2573086.6666666665, ans=0.1
2023-11-23 22:51:00,221 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.06 vs. limit=15.0
2023-11-23 22:51:02,182 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:51:13,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2573220.0, ans=0.125
2023-11-23 22:51:13,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2573220.0, ans=0.0
2023-11-23 22:51:15,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2573220.0, ans=0.125
2023-11-23 22:51:20,153 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.11 vs. limit=15.0
2023-11-23 22:51:31,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386000
2023-11-23 22:51:37,943 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.86 vs. limit=10.0
2023-11-23 22:51:38,145 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.994e+01 8.408e+01 9.187e+01 9.835e+01 1.432e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-23 22:51:43,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2573353.3333333335, ans=0.125
2023-11-23 22:51:50,531 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1250, loss[loss=0.06838, simple_loss=0.09473, pruned_loss=0.01447, audio_tagging_loss=0.006548, over 15260.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.0913, pruned_loss=0.01334, audio_tagging_loss=0.008926, over 3037649.24 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:52:22,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2573553.3333333335, ans=0.125
2023-11-23 22:52:33,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2573620.0, ans=0.125
2023-11-23 22:52:35,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386050
2023-11-23 22:52:43,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2573686.6666666665, ans=0.5
2023-11-23 22:52:52,993 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1300, loss[loss=0.05995, simple_loss=0.07405, pruned_loss=0.01384, audio_tagging_loss=0.009077, over 16797.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09069, pruned_loss=0.01336, audio_tagging_loss=0.008903, over 3034693.53 frames. ], batch size: 64, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:53:12,765 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:53:36,840 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386100
2023-11-23 22:53:42,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.06 vs. limit=15.0
2023-11-23 22:53:42,572 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.029e+01 8.267e+01 8.921e+01 9.505e+01 1.791e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-23 22:53:55,060 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1350, loss[loss=0.06127, simple_loss=0.08076, pruned_loss=0.01358, audio_tagging_loss=0.007312, over 16030.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09205, pruned_loss=0.01368, audio_tagging_loss=0.008857, over 3034810.47 frames. ], batch size: 59, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:54:03,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2574086.6666666665, ans=0.125
2023-11-23 22:54:06,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2574086.6666666665, ans=0.125
2023-11-23 22:54:28,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2574220.0, ans=0.05
2023-11-23 22:54:29,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2574220.0, ans=0.125
2023-11-23 22:54:34,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2574286.6666666665, ans=0.1
2023-11-23 22:54:35,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2574286.6666666665, ans=0.125
2023-11-23 22:54:38,560 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386150
2023-11-23 22:54:39,681 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 22:54:41,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2574286.6666666665, ans=0.125
2023-11-23 22:54:57,767 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1400, loss[loss=0.05912, simple_loss=0.07948, pruned_loss=0.009512, audio_tagging_loss=0.009867, over 14483.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.0907, pruned_loss=0.01341, audio_tagging_loss=0.008953, over 3028593.45 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:55:09,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2574486.6666666665, ans=0.125
2023-11-23 22:55:35,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2574620.0, ans=0.0
2023-11-23 22:55:40,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2574620.0, ans=0.1
2023-11-23 22:55:41,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386200
2023-11-23 22:55:48,296 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.180e+01 8.327e+01 9.200e+01 9.811e+01 1.262e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-23 22:55:52,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2574686.6666666665, ans=0.125
2023-11-23 22:55:58,977 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1450, loss[loss=0.07301, simple_loss=0.09993, pruned_loss=0.01491, audio_tagging_loss=0.008137, over 14592.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09058, pruned_loss=0.01333, audio_tagging_loss=0.008953, over 3036575.12 frames. ], batch size: 54, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:56:12,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2574820.0, ans=0.125
2023-11-23 22:56:19,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2574820.0, ans=0.2
2023-11-23 22:56:25,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2574886.6666666665, ans=0.2
2023-11-23 22:56:31,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2574886.6666666665, ans=0.125
2023-11-23 22:56:42,109 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386250
2023-11-23 22:56:46,266 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.53 vs. limit=15.0
2023-11-23 22:57:00,666 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1500, loss[loss=0.07563, simple_loss=0.1036, pruned_loss=0.01409, audio_tagging_loss=0.009737, over 15745.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09076, pruned_loss=0.01339, audio_tagging_loss=0.00911, over 3042025.90 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:57:03,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2575086.6666666665, ans=0.2
2023-11-23 22:57:10,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2575086.6666666665, ans=0.125
2023-11-23 22:57:19,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2575153.3333333335, ans=0.125
2023-11-23 22:57:25,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2575220.0, ans=0.0
2023-11-23 22:57:39,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2575286.6666666665, ans=0.125
2023-11-23 22:57:44,177 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386300
2023-11-23 22:57:45,614 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 22:57:51,768 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.322e+01 9.192e+01 9.802e+01 2.224e+02, threshold=1.838e+02, percent-clipped=1.0
2023-11-23 22:57:57,848 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.70 vs. limit=15.0
2023-11-23 22:58:02,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2575353.3333333335, ans=0.0
2023-11-23 22:58:04,262 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1550, loss[loss=0.0819, simple_loss=0.1086, pruned_loss=0.01643, audio_tagging_loss=0.01115, over 15867.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09065, pruned_loss=0.01351, audio_tagging_loss=0.00919, over 3041619.57 frames. ], batch size: 59, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 22:58:28,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2575553.3333333335, ans=0.2
2023-11-23 22:58:40,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2575620.0, ans=0.125
2023-11-23 22:58:47,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386350
2023-11-23 22:58:52,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2575686.6666666665, ans=0.125
2023-11-23 22:59:01,941 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.04 vs. limit=12.0
2023-11-23 22:59:02,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2575686.6666666665, ans=0.125
2023-11-23 22:59:05,877 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1600, loss[loss=0.05898, simple_loss=0.07406, pruned_loss=0.0116, audio_tagging_loss=0.01034, over 15123.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09121, pruned_loss=0.01364, audio_tagging_loss=0.009168, over 3039007.11 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 22:59:10,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2575753.3333333335, ans=0.125
2023-11-23 22:59:49,998 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386400
2023-11-23 22:59:52,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2575953.3333333335, ans=0.0
2023-11-23 22:59:57,364 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.070e+01 8.368e+01 9.062e+01 9.713e+01 1.291e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-23 23:00:08,294 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1650, loss[loss=0.06993, simple_loss=0.09494, pruned_loss=0.01107, audio_tagging_loss=0.01138, over 13583.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09058, pruned_loss=0.01363, audio_tagging_loss=0.009192, over 3043643.53 frames. ], batch size: 52, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:00:36,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2576220.0, ans=0.1
2023-11-23 23:00:38,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2576220.0, ans=0.1
2023-11-23 23:00:43,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2576220.0, ans=0.0
2023-11-23 23:00:52,493 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386450
2023-11-23 23:00:52,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2576286.6666666665, ans=0.125
2023-11-23 23:00:57,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2576353.3333333335, ans=0.1
2023-11-23 23:01:01,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2576353.3333333335, ans=0.125
2023-11-23 23:01:08,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2576353.3333333335, ans=0.2
2023-11-23 23:01:08,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2576353.3333333335, ans=0.1
2023-11-23 23:01:12,036 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1700, loss[loss=0.0941, simple_loss=0.1244, pruned_loss=0.01936, audio_tagging_loss=0.01254, over 15831.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.0917, pruned_loss=0.01368, audio_tagging_loss=0.009203, over 3052580.76 frames. ], batch size: 59, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:01:36,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2576553.3333333335, ans=0.1
2023-11-23 23:01:45,180 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.22 vs. limit=15.0
2023-11-23 23:01:55,356 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386500
2023-11-23 23:02:03,012 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.350e+01 8.996e+01 9.717e+01 1.219e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-23 23:02:14,546 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1750, loss[loss=0.08094, simple_loss=0.1108, pruned_loss=0.01776, audio_tagging_loss=0.007792, over 14984.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09096, pruned_loss=0.01342, audio_tagging_loss=0.00912, over 3050471.64 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:02:18,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2576753.3333333335, ans=0.125
2023-11-23 23:02:19,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2576753.3333333335, ans=0.2
2023-11-23 23:02:21,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.68 vs. limit=10.0
2023-11-23 23:02:25,424 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:02:25,785 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.85 vs. limit=22.5
2023-11-23 23:02:31,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.41 vs. limit=15.0
2023-11-23 23:02:40,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2576886.6666666665, ans=0.1
2023-11-23 23:02:51,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2576953.3333333335, ans=0.125
2023-11-23 23:02:58,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386550
2023-11-23 23:03:02,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.whiten.whitening_limit, batch_count=2576953.3333333335, ans=12.0
2023-11-23 23:03:05,631 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.04 vs. limit=15.0
2023-11-23 23:03:06,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2577020.0, ans=0.1
2023-11-23 23:03:11,805 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.95 vs. limit=15.0
2023-11-23 23:03:15,787 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1800, loss[loss=0.0341, simple_loss=0.04014, pruned_loss=0.004653, audio_tagging_loss=0.009378, over 15944.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.091, pruned_loss=0.01331, audio_tagging_loss=0.009098, over 3054032.96 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:03:23,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2577086.6666666665, ans=0.125
2023-11-23 23:03:37,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2577153.3333333335, ans=0.0
2023-11-23 23:03:46,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2577220.0, ans=0.0
2023-11-23 23:03:59,270 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386600
2023-11-23 23:03:59,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2577286.6666666665, ans=0.125
2023-11-23 23:04:05,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2577353.3333333335, ans=0.04949747468305833
2023-11-23 23:04:08,237 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.512e+01 8.514e+01 9.080e+01 9.768e+01 1.273e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-23 23:04:08,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2577353.3333333335, ans=0.2
2023-11-23 23:04:18,923 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1850, loss[loss=0.07006, simple_loss=0.1018, pruned_loss=0.01161, audio_tagging_loss=0.00757, over 15228.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09163, pruned_loss=0.01347, audio_tagging_loss=0.008944, over 3053695.30 frames. ], batch size: 57, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:04:37,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2577486.6666666665, ans=0.05
2023-11-23 23:04:48,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2577553.3333333335, ans=0.125
2023-11-23 23:05:02,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386650
2023-11-23 23:05:18,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2577686.6666666665, ans=0.2
2023-11-23 23:05:20,247 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1900, loss[loss=0.06093, simple_loss=0.08554, pruned_loss=0.008879, audio_tagging_loss=0.009279, over 14683.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09068, pruned_loss=0.01327, audio_tagging_loss=0.008929, over 3044303.93 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:05:24,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2577753.3333333335, ans=0.125
2023-11-23 23:05:35,976 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.52 vs. limit=15.0
2023-11-23 23:05:50,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2577886.6666666665, ans=0.05
2023-11-23 23:06:04,481 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386700
2023-11-23 23:06:13,474 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.773e+01 8.280e+01 8.966e+01 9.619e+01 1.252e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-23 23:06:23,281 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 1950, loss[loss=0.09032, simple_loss=0.119, pruned_loss=0.02241, audio_tagging_loss=0.008392, over 16382.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09101, pruned_loss=0.01331, audio_tagging_loss=0.008881, over 3043574.28 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:06:38,219 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.48 vs. limit=10.0
2023-11-23 23:06:44,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2578153.3333333335, ans=0.0
2023-11-23 23:07:07,534 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386750
2023-11-23 23:07:21,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2578353.3333333335, ans=0.2
2023-11-23 23:07:26,430 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2000, loss[loss=0.05643, simple_loss=0.07676, pruned_loss=0.01141, audio_tagging_loss=0.006646, over 16589.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09012, pruned_loss=0.01313, audio_tagging_loss=0.008936, over 3052303.66 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:07:27,973 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:07:34,839 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.71 vs. limit=15.0
2023-11-23 23:07:41,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2578486.6666666665, ans=0.1
2023-11-23 23:08:09,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386800
2023-11-23 23:08:14,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2578620.0, ans=0.125
2023-11-23 23:08:17,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2578686.6666666665, ans=0.125
2023-11-23 23:08:19,217 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.226e+01 8.902e+01 9.752e+01 1.387e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-23 23:08:28,802 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2050, loss[loss=0.08348, simple_loss=0.1174, pruned_loss=0.01694, audio_tagging_loss=0.007817, over 14686.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09107, pruned_loss=0.01337, audio_tagging_loss=0.008795, over 3050524.51 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:08:36,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2578753.3333333335, ans=0.07
2023-11-23 23:08:52,786 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=14.02 vs. limit=15.0
2023-11-23 23:09:12,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386850
2023-11-23 23:09:24,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2579020.0, ans=0.0
2023-11-23 23:09:30,985 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2100, loss[loss=0.06639, simple_loss=0.09034, pruned_loss=0.01336, audio_tagging_loss=0.007855, over 14372.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09184, pruned_loss=0.01336, audio_tagging_loss=0.008731, over 3058515.18 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:09:33,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2579086.6666666665, ans=0.125
2023-11-23 23:10:14,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386900
2023-11-23 23:10:21,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2579353.3333333335, ans=0.1
2023-11-23 23:10:22,638 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.91 vs. limit=10.0
2023-11-23 23:10:23,223 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.184e+01 8.426e+01 9.085e+01 9.880e+01 1.258e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-23 23:10:25,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2579353.3333333335, ans=0.125
2023-11-23 23:10:26,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2579353.3333333335, ans=0.125
2023-11-23 23:10:33,281 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2150, loss[loss=0.09522, simple_loss=0.1309, pruned_loss=0.02196, audio_tagging_loss=0.007801, over 15683.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09249, pruned_loss=0.01364, audio_tagging_loss=0.00877, over 3047429.79 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:10:47,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2579486.6666666665, ans=0.0
2023-11-23 23:10:48,437 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.79 vs. limit=10.0
2023-11-23 23:10:59,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2579553.3333333335, ans=0.07
2023-11-23 23:11:07,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2579553.3333333335, ans=0.95
2023-11-23 23:11:10,842 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:11:11,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2579620.0, ans=0.125
2023-11-23 23:11:17,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 386950
2023-11-23 23:11:36,346 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2200, loss[loss=0.07832, simple_loss=0.1008, pruned_loss=0.01896, audio_tagging_loss=0.008976, over 16597.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.0936, pruned_loss=0.01397, audio_tagging_loss=0.008808, over 3050260.02 frames. ], batch size: 62, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:11:49,046 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.26 vs. limit=15.0
2023-11-23 23:11:56,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2579820.0, ans=0.125
2023-11-23 23:11:56,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2579820.0, ans=0.0
2023-11-23 23:12:13,043 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.50 vs. limit=15.0
2023-11-23 23:12:20,251 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387000
2023-11-23 23:12:25,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2580020.0, ans=0.0
2023-11-23 23:12:26,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_na.min_abs, batch_count=2580020.0, ans=0.02
2023-11-23 23:12:29,890 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.440e+01 8.536e+01 9.005e+01 9.625e+01 2.688e+02, threshold=1.801e+02, percent-clipped=1.0
2023-11-23 23:12:38,246 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2250, loss[loss=0.0597, simple_loss=0.08567, pruned_loss=0.009431, audio_tagging_loss=0.007433, over 14481.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.0929, pruned_loss=0.01391, audio_tagging_loss=0.008886, over 3045775.13 frames. ], batch size: 55, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:12:53,177 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.37 vs. limit=15.0
2023-11-23 23:12:56,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2580153.3333333335, ans=0.2
2023-11-23 23:13:02,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2580153.3333333335, ans=0.125
2023-11-23 23:13:14,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2580220.0, ans=0.125
2023-11-23 23:13:15,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2580286.6666666665, ans=0.125
2023-11-23 23:13:22,561 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387050
2023-11-23 23:13:34,989 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:13:41,186 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2300, loss[loss=0.07341, simple_loss=0.09353, pruned_loss=0.01549, audio_tagging_loss=0.01116, over 15621.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09212, pruned_loss=0.01377, audio_tagging_loss=0.008938, over 3044370.47 frames. ], batch size: 58, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:13:45,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2580420.0, ans=0.09899494936611666
2023-11-23 23:14:02,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2580486.6666666665, ans=0.1
2023-11-23 23:14:13,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2580553.3333333335, ans=0.1
2023-11-23 23:14:14,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2580553.3333333335, ans=0.125
2023-11-23 23:14:23,245 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2580620.0, ans=0.125
2023-11-23 23:14:24,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387100
2023-11-23 23:14:34,373 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.489e+01 9.011e+01 9.554e+01 1.239e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-23 23:14:36,321 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:14:43,648 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2350, loss[loss=0.04833, simple_loss=0.05809, pruned_loss=0.008041, audio_tagging_loss=0.01124, over 15713.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09124, pruned_loss=0.0137, audio_tagging_loss=0.009034, over 3046920.50 frames. ], batch size: 61, lr: 2.07e-03, grad_scale: 16.0
2023-11-23 23:14:45,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2580753.3333333335, ans=0.2
2023-11-23 23:14:54,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2580820.0, ans=0.1
2023-11-23 23:14:57,347 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.30 vs. limit=15.0
2023-11-23 23:15:08,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2580886.6666666665, ans=0.0
2023-11-23 23:15:27,115 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387150
2023-11-23 23:15:43,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2581086.6666666665, ans=0.125
2023-11-23 23:15:44,725 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2400, loss[loss=0.09966, simple_loss=0.136, pruned_loss=0.02207, audio_tagging_loss=0.00958, over 15456.00 frames. ], tot_loss[loss=0.06917, simple_loss=0.09234, pruned_loss=0.0139, audio_tagging_loss=0.009095, over 3042586.78 frames. ], batch size: 56, lr: 2.07e-03, grad_scale: 32.0
2023-11-23 23:16:14,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2581220.0, ans=0.125
2023-11-23 23:16:16,724 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:16:28,354 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387200
2023-11-23 23:16:29,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2581286.6666666665, ans=0.125
2023-11-23 23:16:39,151 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.662e+01 8.668e+01 9.147e+01 9.893e+01 1.218e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-23 23:16:44,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2581353.3333333335, ans=0.125
2023-11-23 23:16:46,728 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2450, loss[loss=0.06955, simple_loss=0.09356, pruned_loss=0.01238, audio_tagging_loss=0.01039, over 14545.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09142, pruned_loss=0.01374, audio_tagging_loss=0.009207, over 3041467.27 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:16:56,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2581420.0, ans=0.2
2023-11-23 23:16:58,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2581420.0, ans=0.125
2023-11-23 23:17:18,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2581553.3333333335, ans=0.2
2023-11-23 23:17:29,927 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387250
2023-11-23 23:17:31,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2581620.0, ans=0.125
2023-11-23 23:17:50,313 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2500, loss[loss=0.07981, simple_loss=0.11, pruned_loss=0.01805, audio_tagging_loss=0.006748, over 16440.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09085, pruned_loss=0.01353, audio_tagging_loss=0.009231, over 3049426.15 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:17:55,405 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:17:56,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2581753.3333333335, ans=0.0
2023-11-23 23:17:56,827 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.86 vs. limit=15.0
2023-11-23 23:18:14,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2581886.6666666665, ans=0.1
2023-11-23 23:18:15,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2581886.6666666665, ans=0.0
2023-11-23 23:18:26,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2581953.3333333335, ans=0.0
2023-11-23 23:18:33,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387300
2023-11-23 23:18:40,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2582020.0, ans=0.95
2023-11-23 23:18:42,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2582020.0, ans=0.1
2023-11-23 23:18:44,137 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.523e+01 8.436e+01 9.159e+01 9.960e+01 1.411e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-23 23:18:51,373 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2550, loss[loss=0.0647, simple_loss=0.09657, pruned_loss=0.01041, audio_tagging_loss=0.006003, over 15489.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09195, pruned_loss=0.01378, audio_tagging_loss=0.009048, over 3053996.27 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:18:55,523 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=10.16 vs. limit=15.0
2023-11-23 23:19:08,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2582153.3333333335, ans=0.05
2023-11-23 23:19:26,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2582220.0, ans=0.125
2023-11-23 23:19:29,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2582286.6666666665, ans=0.1
2023-11-23 23:19:35,147 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387350
2023-11-23 23:19:52,920 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2600, loss[loss=0.08357, simple_loss=0.1125, pruned_loss=0.01926, audio_tagging_loss=0.008067, over 15815.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09103, pruned_loss=0.01361, audio_tagging_loss=0.008994, over 3060256.40 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:19:53,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2582420.0, ans=0.0
2023-11-23 23:20:09,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2582486.6666666665, ans=0.0
2023-11-23 23:20:11,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2582486.6666666665, ans=0.125
2023-11-23 23:20:22,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2582553.3333333335, ans=0.0
2023-11-23 23:20:35,781 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.62 vs. limit=22.5
2023-11-23 23:20:35,868 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.96 vs. limit=15.0
2023-11-23 23:20:36,550 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387400
2023-11-23 23:20:48,949 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.750e+01 8.392e+01 8.863e+01 9.833e+01 1.286e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-23 23:20:56,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2582753.3333333335, ans=15.0
2023-11-23 23:20:56,695 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2650, loss[loss=0.07759, simple_loss=0.1022, pruned_loss=0.01893, audio_tagging_loss=0.007539, over 15017.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09152, pruned_loss=0.01368, audio_tagging_loss=0.008963, over 3053934.63 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:21:18,279 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.78 vs. limit=10.0
2023-11-23 23:21:24,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2582886.6666666665, ans=0.125
2023-11-23 23:21:26,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2582886.6666666665, ans=0.125
2023-11-23 23:21:40,183 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387450
2023-11-23 23:21:58,186 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.09 vs. limit=10.0
2023-11-23 23:21:58,880 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2700, loss[loss=0.07023, simple_loss=0.09718, pruned_loss=0.01295, audio_tagging_loss=0.008686, over 14771.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.0918, pruned_loss=0.01367, audio_tagging_loss=0.009012, over 3052752.24 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:22:22,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2583220.0, ans=0.0
2023-11-23 23:22:24,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2583220.0, ans=0.1
2023-11-23 23:22:42,496 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387500
2023-11-23 23:22:44,123 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=16.19 vs. limit=15.0
2023-11-23 23:22:54,186 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.525e+01 8.709e+01 9.550e+01 1.033e+02 1.468e+02, threshold=1.910e+02, percent-clipped=0.0
2023-11-23 23:22:55,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2583353.3333333335, ans=0.0
2023-11-23 23:23:00,145 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2750, loss[loss=0.06556, simple_loss=0.0825, pruned_loss=0.01366, audio_tagging_loss=0.01064, over 15521.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09117, pruned_loss=0.01357, audio_tagging_loss=0.009009, over 3050937.08 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:23:25,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2583553.3333333335, ans=0.125
2023-11-23 23:23:26,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2583553.3333333335, ans=0.0
2023-11-23 23:23:30,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2583553.3333333335, ans=0.125
2023-11-23 23:23:38,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2583620.0, ans=0.2
2023-11-23 23:23:44,155 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387550
2023-11-23 23:23:54,121 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:24:02,882 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2800, loss[loss=0.08296, simple_loss=0.1232, pruned_loss=0.01444, audio_tagging_loss=0.006907, over 15384.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09078, pruned_loss=0.01327, audio_tagging_loss=0.008961, over 3043464.74 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:24:17,556 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:24:46,383 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387600
2023-11-23 23:24:48,924 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.56 vs. limit=15.0
2023-11-23 23:24:56,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2584020.0, ans=0.0
2023-11-23 23:24:59,635 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.278e+01 8.804e+01 9.483e+01 1.291e+02, threshold=1.761e+02, percent-clipped=0.0
2023-11-23 23:25:03,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2584020.0, ans=0.125
2023-11-23 23:25:04,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2584020.0, ans=0.125
2023-11-23 23:25:06,135 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2850, loss[loss=0.05246, simple_loss=0.07046, pruned_loss=0.009825, audio_tagging_loss=0.007401, over 14463.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09111, pruned_loss=0.01345, audio_tagging_loss=0.008868, over 3044538.68 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:25:13,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2584086.6666666665, ans=0.025
2023-11-23 23:25:25,298 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:25:49,597 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387650
2023-11-23 23:25:57,580 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:26:07,811 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2900, loss[loss=0.06303, simple_loss=0.08208, pruned_loss=0.01129, audio_tagging_loss=0.0107, over 14286.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09114, pruned_loss=0.01335, audio_tagging_loss=0.008945, over 3037176.53 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:26:09,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_abs, batch_count=2584420.0, ans=0.5
2023-11-23 23:26:16,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2584420.0, ans=0.125
2023-11-23 23:26:51,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387700
2023-11-23 23:27:01,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2584686.6666666665, ans=0.2
2023-11-23 23:27:05,507 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.293e+01 9.102e+01 9.710e+01 1.234e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-23 23:27:08,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2584686.6666666665, ans=0.0
2023-11-23 23:27:10,865 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 2950, loss[loss=0.04283, simple_loss=0.0501, pruned_loss=0.0101, audio_tagging_loss=0.007686, over 14593.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09126, pruned_loss=0.01354, audio_tagging_loss=0.008978, over 3046882.77 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:27:15,654 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.35 vs. limit=15.0
2023-11-23 23:27:20,123 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.91 vs. limit=15.0
2023-11-23 23:27:32,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2584820.0, ans=0.0
2023-11-23 23:27:34,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2584886.6666666665, ans=0.125
2023-11-23 23:27:54,381 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387750
2023-11-23 23:27:57,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2584953.3333333335, ans=0.125
2023-11-23 23:28:08,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2585020.0, ans=0.95
2023-11-23 23:28:13,270 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3000, loss[loss=0.09536, simple_loss=0.1262, pruned_loss=0.02175, audio_tagging_loss=0.01054, over 15504.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09103, pruned_loss=0.01354, audio_tagging_loss=0.00905, over 3042161.13 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:28:13,274 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-23 23:28:53,116 INFO [train_asr.py:1253] (0/4) Epoch 33, validation: loss=0.05846, simple_loss=0.05103, pruned_loss=0.005194, audio_tagging_loss=0.02775, over 4681554.00 frames. 
2023-11-23 23:28:53,117 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-23 23:28:53,951 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.50 vs. limit=15.0
2023-11-23 23:29:19,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2585220.0, ans=0.125
2023-11-23 23:29:30,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2585286.6666666665, ans=0.1
2023-11-23 23:29:37,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387800
2023-11-23 23:29:48,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2585353.3333333335, ans=0.125
2023-11-23 23:29:51,586 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.566e+01 9.149e+01 9.920e+01 1.258e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-23 23:29:51,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2585353.3333333335, ans=0.125
2023-11-23 23:29:56,999 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3050, loss[loss=0.07624, simple_loss=0.107, pruned_loss=0.01369, audio_tagging_loss=0.00906, over 15460.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09176, pruned_loss=0.01366, audio_tagging_loss=0.009138, over 3048312.09 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:29:59,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2585420.0, ans=0.0
2023-11-23 23:30:22,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2585553.3333333335, ans=0.0
2023-11-23 23:30:28,123 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.97 vs. limit=15.0
2023-11-23 23:30:35,042 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:30:42,361 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387850
2023-11-23 23:30:50,781 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.01 vs. limit=15.0
2023-11-23 23:31:01,223 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3100, loss[loss=0.06352, simple_loss=0.07943, pruned_loss=0.01366, audio_tagging_loss=0.01015, over 15514.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09323, pruned_loss=0.014, audio_tagging_loss=0.009017, over 3054203.12 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:31:05,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2585753.3333333335, ans=0.1
2023-11-23 23:31:14,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2585820.0, ans=0.125
2023-11-23 23:31:20,870 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.09 vs. limit=15.0
2023-11-23 23:31:45,707 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387900
2023-11-23 23:31:59,122 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.740e+01 8.717e+01 9.122e+01 9.882e+01 2.384e+02, threshold=1.824e+02, percent-clipped=1.0
2023-11-23 23:32:03,853 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3150, loss[loss=0.063, simple_loss=0.08927, pruned_loss=0.01068, audio_tagging_loss=0.007679, over 14916.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09255, pruned_loss=0.01386, audio_tagging_loss=0.009111, over 3050618.99 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 8.0
2023-11-23 23:32:04,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2586086.6666666665, ans=0.2
2023-11-23 23:32:10,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2586086.6666666665, ans=0.125
2023-11-23 23:32:20,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2586153.3333333335, ans=0.125
2023-11-23 23:32:41,686 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.58 vs. limit=15.0
2023-11-23 23:32:47,549 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 387950
2023-11-23 23:32:53,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2586353.3333333335, ans=0.0
2023-11-23 23:32:59,938 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.36 vs. limit=10.0
2023-11-23 23:33:04,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2586353.3333333335, ans=0.125
2023-11-23 23:33:06,484 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3200, loss[loss=0.07012, simple_loss=0.09556, pruned_loss=0.01315, audio_tagging_loss=0.009193, over 15299.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09168, pruned_loss=0.01359, audio_tagging_loss=0.009203, over 3046502.94 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:33:06,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2586420.0, ans=0.125
2023-11-23 23:33:18,683 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.37 vs. limit=15.0
2023-11-23 23:33:50,075 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388000
2023-11-23 23:33:51,966 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-388000.pt
2023-11-23 23:34:06,877 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.117e+01 8.288e+01 8.870e+01 9.620e+01 1.279e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-23 23:34:11,754 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3250, loss[loss=0.08044, simple_loss=0.1124, pruned_loss=0.01559, audio_tagging_loss=0.008655, over 14648.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09165, pruned_loss=0.01349, audio_tagging_loss=0.009257, over 3048664.13 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:34:14,862 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.92 vs. limit=10.0
2023-11-23 23:34:19,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2586753.3333333335, ans=0.025
2023-11-23 23:34:25,248 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.24 vs. limit=10.0
2023-11-23 23:34:34,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2586820.0, ans=0.0
2023-11-23 23:34:53,467 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.45 vs. limit=15.0
2023-11-23 23:34:56,658 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388050
2023-11-23 23:35:15,189 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3300, loss[loss=0.06622, simple_loss=0.08927, pruned_loss=0.01321, audio_tagging_loss=0.008367, over 16614.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09306, pruned_loss=0.01372, audio_tagging_loss=0.009202, over 3051407.00 frames. ], batch size: 65, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:35:26,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2587153.3333333335, ans=0.125
2023-11-23 23:35:43,762 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.34 vs. limit=15.0
2023-11-23 23:35:45,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2587220.0, ans=0.125
2023-11-23 23:35:48,523 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.09 vs. limit=15.0
2023-11-23 23:35:58,560 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388100
2023-11-23 23:35:59,358 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=12.89 vs. limit=15.0
2023-11-23 23:36:12,202 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.941e+01 8.544e+01 9.048e+01 9.852e+01 1.338e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-23 23:36:14,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2587353.3333333335, ans=0.1
2023-11-23 23:36:18,182 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3350, loss[loss=0.05773, simple_loss=0.07744, pruned_loss=0.01083, audio_tagging_loss=0.008176, over 15121.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.0927, pruned_loss=0.01358, audio_tagging_loss=0.00912, over 3050972.30 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:36:37,224 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:37:00,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2587620.0, ans=0.0
2023-11-23 23:37:02,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388150
2023-11-23 23:37:03,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2587620.0, ans=0.0
2023-11-23 23:37:20,749 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3400, loss[loss=0.07528, simple_loss=0.1008, pruned_loss=0.01865, audio_tagging_loss=0.006223, over 14793.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09303, pruned_loss=0.01368, audio_tagging_loss=0.009044, over 3051968.38 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:37:21,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2587753.3333333335, ans=0.0
2023-11-23 23:37:22,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2587753.3333333335, ans=0.125
2023-11-23 23:37:23,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2587753.3333333335, ans=0.05
2023-11-23 23:37:27,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2587753.3333333335, ans=0.125
2023-11-23 23:37:34,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2587820.0, ans=0.125
2023-11-23 23:37:45,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2587886.6666666665, ans=0.1
2023-11-23 23:37:51,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2587886.6666666665, ans=0.1
2023-11-23 23:38:04,365 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388200
2023-11-23 23:38:17,824 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.027e+01 8.375e+01 8.962e+01 9.583e+01 1.424e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-23 23:38:22,589 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3450, loss[loss=0.04875, simple_loss=0.06143, pruned_loss=0.006869, audio_tagging_loss=0.01117, over 14800.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09237, pruned_loss=0.0136, audio_tagging_loss=0.008877, over 3043987.74 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:38:22,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2588086.6666666665, ans=0.125
2023-11-23 23:38:37,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2588153.3333333335, ans=0.0
2023-11-23 23:38:41,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2588153.3333333335, ans=0.1
2023-11-23 23:38:43,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2588153.3333333335, ans=0.125
2023-11-23 23:39:02,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2588286.6666666665, ans=0.125
2023-11-23 23:39:03,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2588286.6666666665, ans=0.125
2023-11-23 23:39:07,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388250
2023-11-23 23:39:12,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2588353.3333333335, ans=0.2
2023-11-23 23:39:26,130 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3500, loss[loss=0.07375, simple_loss=0.09907, pruned_loss=0.0157, audio_tagging_loss=0.008518, over 14709.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09228, pruned_loss=0.0138, audio_tagging_loss=0.008872, over 3047541.99 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:39:41,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2588486.6666666665, ans=0.0
2023-11-23 23:39:57,876 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:40:01,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2588620.0, ans=0.125
2023-11-23 23:40:09,086 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388300
2023-11-23 23:40:21,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2588686.6666666665, ans=0.125
2023-11-23 23:40:23,681 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.850e+01 8.607e+01 9.289e+01 1.010e+02 1.260e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-23 23:40:28,527 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3550, loss[loss=0.07719, simple_loss=0.1084, pruned_loss=0.01522, audio_tagging_loss=0.007762, over 15491.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09162, pruned_loss=0.01361, audio_tagging_loss=0.008928, over 3043168.95 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:40:31,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2588753.3333333335, ans=0.1
2023-11-23 23:40:36,213 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.01 vs. limit=15.0
2023-11-23 23:40:46,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2588820.0, ans=0.125
2023-11-23 23:40:53,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2588886.6666666665, ans=0.125
2023-11-23 23:40:54,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2588886.6666666665, ans=0.0
2023-11-23 23:40:58,147 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.83 vs. limit=15.0
2023-11-23 23:41:00,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2588886.6666666665, ans=0.125
2023-11-23 23:41:01,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.10 vs. limit=10.0
2023-11-23 23:41:06,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.41 vs. limit=6.0
2023-11-23 23:41:12,580 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388350
2023-11-23 23:41:28,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2589020.0, ans=0.1
2023-11-23 23:41:30,140 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3600, loss[loss=0.07559, simple_loss=0.09754, pruned_loss=0.01686, audio_tagging_loss=0.009954, over 14441.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09194, pruned_loss=0.0137, audio_tagging_loss=0.008869, over 3044918.20 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:41:39,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2589086.6666666665, ans=0.2
2023-11-23 23:41:41,558 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.74 vs. limit=15.0
2023-11-23 23:41:50,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2589153.3333333335, ans=0.125
2023-11-23 23:41:50,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2589153.3333333335, ans=0.125
2023-11-23 23:42:03,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2589220.0, ans=0.125
2023-11-23 23:42:05,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2589220.0, ans=0.0
2023-11-23 23:42:14,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388400
2023-11-23 23:42:28,667 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.733e+01 8.372e+01 8.860e+01 9.569e+01 1.357e+02, threshold=1.772e+02, percent-clipped=0.0
2023-11-23 23:42:32,295 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3650, loss[loss=0.08812, simple_loss=0.1218, pruned_loss=0.0222, audio_tagging_loss=0.005037, over 16022.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09291, pruned_loss=0.01391, audio_tagging_loss=0.008812, over 3048524.90 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:42:42,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2589420.0, ans=0.125
2023-11-23 23:42:57,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2589553.3333333335, ans=0.125
2023-11-23 23:43:16,783 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388450
2023-11-23 23:43:27,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2589686.6666666665, ans=0.0
2023-11-23 23:43:27,550 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:43:35,022 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:43:35,933 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3700, loss[loss=0.05129, simple_loss=0.06664, pruned_loss=0.009519, audio_tagging_loss=0.008457, over 14653.00 frames. ], tot_loss[loss=0.06904, simple_loss=0.09258, pruned_loss=0.01392, audio_tagging_loss=0.008831, over 3044356.80 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:43:43,135 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:43:43,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2589753.3333333335, ans=10.0
2023-11-23 23:43:59,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2589886.6666666665, ans=0.0
2023-11-23 23:44:19,879 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388500
2023-11-23 23:44:21,686 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.45 vs. limit=15.0
2023-11-23 23:44:29,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2590020.0, ans=0.0
2023-11-23 23:44:31,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2590020.0, ans=15.0
2023-11-23 23:44:31,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2590020.0, ans=0.1
2023-11-23 23:44:33,989 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.322e+01 8.486e+01 9.089e+01 9.768e+01 1.177e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-23 23:44:37,527 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3750, loss[loss=0.09405, simple_loss=0.1223, pruned_loss=0.02612, audio_tagging_loss=0.006761, over 16041.00 frames. ], tot_loss[loss=0.06899, simple_loss=0.09208, pruned_loss=0.01407, audio_tagging_loss=0.008881, over 3041491.88 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:45:18,805 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.05 vs. limit=15.0
2023-11-23 23:45:21,625 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:45:21,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388550
2023-11-23 23:45:37,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2590353.3333333335, ans=0.125
2023-11-23 23:45:39,699 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3800, loss[loss=0.06548, simple_loss=0.08176, pruned_loss=0.01372, audio_tagging_loss=0.01088, over 14851.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09264, pruned_loss=0.01417, audio_tagging_loss=0.008961, over 3044206.47 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:45:45,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2590420.0, ans=0.2
2023-11-23 23:46:15,228 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.63 vs. limit=8.0
2023-11-23 23:46:19,253 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-23 23:46:23,903 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388600
2023-11-23 23:46:24,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2590620.0, ans=0.1
2023-11-23 23:46:24,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2590620.0, ans=10.0
2023-11-23 23:46:35,127 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.18 vs. limit=15.0
2023-11-23 23:46:37,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2590686.6666666665, ans=0.125
2023-11-23 23:46:39,771 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.689e+01 8.380e+01 8.887e+01 9.611e+01 1.323e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-23 23:46:44,015 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3850, loss[loss=0.08373, simple_loss=0.118, pruned_loss=0.01635, audio_tagging_loss=0.008393, over 14814.00 frames. ], tot_loss[loss=0.07009, simple_loss=0.09388, pruned_loss=0.01416, audio_tagging_loss=0.008987, over 3041527.29 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:46:44,755 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.89 vs. limit=15.0
2023-11-23 23:46:49,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2590753.3333333335, ans=0.125
2023-11-23 23:46:59,578 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.61 vs. limit=22.5
2023-11-23 23:47:11,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2590886.6666666665, ans=0.1
2023-11-23 23:47:16,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2590886.6666666665, ans=0.0
2023-11-23 23:47:20,555 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.64 vs. limit=15.0
2023-11-23 23:47:27,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2590953.3333333335, ans=0.125
2023-11-23 23:47:27,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff3.min_abs, batch_count=2590953.3333333335, ans=0.2
2023-11-23 23:47:28,127 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388650
2023-11-23 23:47:31,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2590953.3333333335, ans=0.09899494936611666
2023-11-23 23:47:41,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2591020.0, ans=0.1
2023-11-23 23:47:45,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2591020.0, ans=0.5
2023-11-23 23:47:47,376 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3900, loss[loss=0.08284, simple_loss=0.1168, pruned_loss=0.01359, audio_tagging_loss=0.01085, over 16723.00 frames. ], tot_loss[loss=0.06985, simple_loss=0.09375, pruned_loss=0.01402, audio_tagging_loss=0.00896, over 3031572.74 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:47:52,829 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.78 vs. limit=22.5
2023-11-23 23:47:56,659 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0
2023-11-23 23:48:15,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2591220.0, ans=0.125
2023-11-23 23:48:19,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2591220.0, ans=0.2
2023-11-23 23:48:32,161 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388700
2023-11-23 23:48:34,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2591286.6666666665, ans=0.0
2023-11-23 23:48:42,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2591353.3333333335, ans=0.0
2023-11-23 23:48:46,590 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.330e+01 8.934e+01 9.660e+01 1.290e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-23 23:48:50,241 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 3950, loss[loss=0.0684, simple_loss=0.09776, pruned_loss=0.0125, audio_tagging_loss=0.007018, over 15316.00 frames. ], tot_loss[loss=0.06977, simple_loss=0.09383, pruned_loss=0.01388, audio_tagging_loss=0.008977, over 3035164.72 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:48:55,718 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.68 vs. limit=6.0
2023-11-23 23:49:01,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2591420.0, ans=0.0
2023-11-23 23:49:03,915 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.25 vs. limit=15.0
2023-11-23 23:49:08,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2591486.6666666665, ans=0.0
2023-11-23 23:49:26,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2591553.3333333335, ans=0.125
2023-11-23 23:49:26,724 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.03 vs. limit=15.0
2023-11-23 23:49:34,239 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388750
2023-11-23 23:49:38,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2591620.0, ans=0.125
2023-11-23 23:49:39,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.96 vs. limit=15.0
2023-11-23 23:49:42,107 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-23 23:49:44,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2591686.6666666665, ans=0.125
2023-11-23 23:49:53,534 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4000, loss[loss=0.06917, simple_loss=0.09263, pruned_loss=0.0122, audio_tagging_loss=0.01066, over 15496.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09349, pruned_loss=0.01392, audio_tagging_loss=0.008959, over 3035731.56 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:50:32,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2591953.3333333335, ans=10.0
2023-11-23 23:50:37,106 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388800
2023-11-23 23:50:54,120 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.286e+01 8.337e+01 9.138e+01 9.888e+01 1.432e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-23 23:50:56,675 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4050, loss[loss=0.06178, simple_loss=0.0889, pruned_loss=0.00784, audio_tagging_loss=0.009485, over 15458.00 frames. ], tot_loss[loss=0.06957, simple_loss=0.09348, pruned_loss=0.0138, audio_tagging_loss=0.009028, over 3044631.81 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:51:00,290 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:51:26,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2592220.0, ans=0.1
2023-11-23 23:51:33,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2592286.6666666665, ans=0.1
2023-11-23 23:51:40,569 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388850
2023-11-23 23:51:58,747 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4100, loss[loss=0.0724, simple_loss=0.1034, pruned_loss=0.01512, audio_tagging_loss=0.005586, over 14905.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09265, pruned_loss=0.01366, audio_tagging_loss=0.009078, over 3044309.86 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:52:05,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2592420.0, ans=10.0
2023-11-23 23:52:23,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2592553.3333333335, ans=0.125
2023-11-23 23:52:40,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2592620.0, ans=0.1
2023-11-23 23:52:43,309 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388900
2023-11-23 23:52:44,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2592620.0, ans=0.125
2023-11-23 23:52:59,652 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.379e+01 9.260e+01 9.829e+01 1.268e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-23 23:53:02,091 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4150, loss[loss=0.08062, simple_loss=0.1141, pruned_loss=0.01611, audio_tagging_loss=0.007448, over 16459.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09269, pruned_loss=0.01372, audio_tagging_loss=0.009054, over 3040520.11 frames. ], batch size: 62, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:53:02,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=2592753.3333333335, ans=0.02
2023-11-23 23:53:02,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2592753.3333333335, ans=0.1
2023-11-23 23:53:05,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2592753.3333333335, ans=0.0
2023-11-23 23:53:32,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2592886.6666666665, ans=0.0
2023-11-23 23:53:33,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2592886.6666666665, ans=0.0
2023-11-23 23:53:45,374 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 388950
2023-11-23 23:53:48,203 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-23 23:53:48,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2592953.3333333335, ans=0.125
2023-11-23 23:53:59,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2593020.0, ans=0.125
2023-11-23 23:54:04,226 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4200, loss[loss=0.05437, simple_loss=0.0714, pruned_loss=0.01079, audio_tagging_loss=0.007878, over 14879.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09227, pruned_loss=0.01363, audio_tagging_loss=0.008895, over 3037853.17 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:54:07,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2593086.6666666665, ans=0.125
2023-11-23 23:54:12,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2593086.6666666665, ans=0.125
2023-11-23 23:54:23,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2593153.3333333335, ans=0.125
2023-11-23 23:54:42,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2593286.6666666665, ans=0.07
2023-11-23 23:54:45,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2593286.6666666665, ans=0.0
2023-11-23 23:54:46,141 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.18 vs. limit=22.5
2023-11-23 23:54:47,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2593286.6666666665, ans=0.0
2023-11-23 23:54:47,990 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389000
2023-11-23 23:55:01,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2593353.3333333335, ans=0.1
2023-11-23 23:55:04,540 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.273e+01 8.440e+01 9.074e+01 9.920e+01 1.262e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-23 23:55:04,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2593353.3333333335, ans=0.125
2023-11-23 23:55:07,010 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4250, loss[loss=0.09813, simple_loss=0.135, pruned_loss=0.02453, audio_tagging_loss=0.006097, over 15211.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09225, pruned_loss=0.01354, audio_tagging_loss=0.008877, over 3040322.49 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:55:08,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2593420.0, ans=0.0
2023-11-23 23:55:15,800 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2593420.0, ans=0.125
2023-11-23 23:55:21,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2593486.6666666665, ans=0.09899494936611666
2023-11-23 23:55:37,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2593553.3333333335, ans=0.0
2023-11-23 23:55:39,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2593553.3333333335, ans=0.0
2023-11-23 23:55:43,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2593553.3333333335, ans=0.1
2023-11-23 23:55:51,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389050
2023-11-23 23:55:57,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.45 vs. limit=10.0
2023-11-23 23:56:00,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2593686.6666666665, ans=0.0
2023-11-23 23:56:10,043 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4300, loss[loss=0.08305, simple_loss=0.1191, pruned_loss=0.01626, audio_tagging_loss=0.007257, over 15926.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09278, pruned_loss=0.01361, audio_tagging_loss=0.008785, over 3043961.16 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:56:13,598 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.85 vs. limit=15.0
2023-11-23 23:56:39,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2593886.6666666665, ans=0.125
2023-11-23 23:56:39,568 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.86 vs. limit=15.0
2023-11-23 23:56:53,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389100
2023-11-23 23:57:10,081 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.164e+01 8.627e+01 9.104e+01 9.923e+01 1.149e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-23 23:57:12,442 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4350, loss[loss=0.08521, simple_loss=0.1193, pruned_loss=0.01907, audio_tagging_loss=0.00649, over 14785.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09291, pruned_loss=0.0136, audio_tagging_loss=0.008761, over 3043036.41 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-23 23:57:17,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2594086.6666666665, ans=0.125
2023-11-23 23:57:19,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2594086.6666666665, ans=0.0
2023-11-23 23:57:44,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2594220.0, ans=0.0
2023-11-23 23:57:53,288 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.26 vs. limit=6.0
2023-11-23 23:57:53,418 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.18 vs. limit=6.0
2023-11-23 23:57:56,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389150
2023-11-23 23:58:15,020 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4400, loss[loss=0.06, simple_loss=0.08304, pruned_loss=0.01091, audio_tagging_loss=0.007569, over 14934.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09313, pruned_loss=0.0137, audio_tagging_loss=0.008692, over 3043359.78 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 32.0
2023-11-23 23:58:18,063 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.41 vs. limit=15.0
2023-11-23 23:58:19,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.30 vs. limit=15.0
2023-11-23 23:58:31,655 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.85 vs. limit=15.0
2023-11-23 23:58:32,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2594486.6666666665, ans=0.125
2023-11-23 23:58:34,455 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.41 vs. limit=15.0
2023-11-23 23:58:43,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2594553.3333333335, ans=0.125
2023-11-23 23:58:58,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389200
2023-11-23 23:59:00,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2594620.0, ans=0.0
2023-11-23 23:59:09,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=2594686.6666666665, ans=0.02
2023-11-23 23:59:14,494 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.468e+01 8.522e+01 9.303e+01 9.983e+01 1.286e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-23 23:59:16,886 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4450, loss[loss=0.06137, simple_loss=0.09046, pruned_loss=0.007499, audio_tagging_loss=0.008637, over 15245.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09276, pruned_loss=0.0136, audio_tagging_loss=0.00868, over 3039690.21 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0
2023-11-23 23:59:17,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2594753.3333333335, ans=0.09899494936611666
2023-11-23 23:59:38,232 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.91 vs. limit=22.5
2023-11-23 23:59:50,355 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.15 vs. limit=12.0
2023-11-24 00:00:00,982 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389250
2023-11-24 00:00:01,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2594953.3333333335, ans=0.0
2023-11-24 00:00:12,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2595020.0, ans=0.0
2023-11-24 00:00:19,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2595086.6666666665, ans=0.2
2023-11-24 00:00:20,333 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4500, loss[loss=0.06515, simple_loss=0.08183, pruned_loss=0.01357, audio_tagging_loss=0.01067, over 15808.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09151, pruned_loss=0.01342, audio_tagging_loss=0.008797, over 3046766.26 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:00:27,034 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.32 vs. limit=10.0
2023-11-24 00:00:31,843 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.50 vs. limit=5.0
2023-11-24 00:00:59,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2595286.6666666665, ans=0.0
2023-11-24 00:01:02,034 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.59 vs. limit=15.0
2023-11-24 00:01:04,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389300
2023-11-24 00:01:04,869 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.85 vs. limit=15.0
2023-11-24 00:01:12,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2595353.3333333335, ans=0.1
2023-11-24 00:01:19,570 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.819e+01 8.069e+01 8.788e+01 9.879e+01 1.178e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-24 00:01:22,012 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4550, loss[loss=0.06503, simple_loss=0.0902, pruned_loss=0.01072, audio_tagging_loss=0.009215, over 16616.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09082, pruned_loss=0.01334, audio_tagging_loss=0.008882, over 3046920.12 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:01:52,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2595553.3333333335, ans=0.2
2023-11-24 00:02:05,559 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389350
2023-11-24 00:02:10,193 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 00:02:13,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2595686.6666666665, ans=0.0
2023-11-24 00:02:23,780 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4600, loss[loss=0.07433, simple_loss=0.1078, pruned_loss=0.009762, audio_tagging_loss=0.01067, over 15788.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09171, pruned_loss=0.01356, audio_tagging_loss=0.009004, over 3049925.82 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:02:26,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2595753.3333333335, ans=0.125
2023-11-24 00:02:31,474 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.58 vs. limit=12.0
2023-11-24 00:02:35,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2595820.0, ans=0.125
2023-11-24 00:02:53,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2595886.6666666665, ans=0.2
2023-11-24 00:03:07,381 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389400
2023-11-24 00:03:13,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2596020.0, ans=0.125
2023-11-24 00:03:24,568 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.756e+01 8.634e+01 9.201e+01 1.007e+02 1.274e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-24 00:03:26,985 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4650, loss[loss=0.08872, simple_loss=0.1173, pruned_loss=0.01972, audio_tagging_loss=0.01034, over 15403.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09144, pruned_loss=0.01346, audio_tagging_loss=0.009058, over 3044523.83 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:03:33,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2596086.6666666665, ans=0.04949747468305833
2023-11-24 00:03:42,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2596153.3333333335, ans=0.125
2023-11-24 00:03:59,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2596220.0, ans=0.125
2023-11-24 00:04:10,441 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389450
2023-11-24 00:04:23,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2596353.3333333335, ans=0.0
2023-11-24 00:04:28,288 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.69 vs. limit=15.0
2023-11-24 00:04:28,863 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4700, loss[loss=0.0846, simple_loss=0.1268, pruned_loss=0.01579, audio_tagging_loss=0.005435, over 15701.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.0919, pruned_loss=0.01358, audio_tagging_loss=0.009059, over 3044228.15 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:04:44,797 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.67 vs. limit=15.0
2023-11-24 00:04:50,774 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.44 vs. limit=22.5
2023-11-24 00:05:02,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2596553.3333333335, ans=0.125
2023-11-24 00:05:10,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2596620.0, ans=0.1
2023-11-24 00:05:12,907 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389500
2023-11-24 00:05:15,513 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:05:28,430 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.978e+01 8.415e+01 9.088e+01 1.001e+02 1.234e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-24 00:05:30,847 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4750, loss[loss=0.0635, simple_loss=0.08188, pruned_loss=0.0108, audio_tagging_loss=0.01176, over 14640.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09189, pruned_loss=0.01353, audio_tagging_loss=0.009189, over 3045679.77 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:06:08,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2596953.3333333335, ans=0.0
2023-11-24 00:06:14,946 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389550
2023-11-24 00:06:35,122 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4800, loss[loss=0.03843, simple_loss=0.04926, pruned_loss=0.005505, audio_tagging_loss=0.008297, over 15379.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09226, pruned_loss=0.01354, audio_tagging_loss=0.009202, over 3046797.53 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:06:41,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2597086.6666666665, ans=0.0
2023-11-24 00:06:45,163 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.21 vs. limit=15.0
2023-11-24 00:06:55,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2597153.3333333335, ans=0.125
2023-11-24 00:07:02,119 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.18 vs. limit=10.0
2023-11-24 00:07:16,994 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.15 vs. limit=15.0
2023-11-24 00:07:19,038 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389600
2023-11-24 00:07:32,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2597353.3333333335, ans=0.125
2023-11-24 00:07:37,071 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.10 vs. limit=22.5
2023-11-24 00:07:37,710 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.181e+01 8.337e+01 8.951e+01 9.611e+01 1.135e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-24 00:07:37,755 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4850, loss[loss=0.08047, simple_loss=0.1149, pruned_loss=0.01548, audio_tagging_loss=0.007558, over 15118.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09226, pruned_loss=0.01364, audio_tagging_loss=0.009253, over 3057112.44 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:07:38,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.94 vs. limit=6.0
2023-11-24 00:07:40,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2597420.0, ans=0.125
2023-11-24 00:07:52,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2597486.6666666665, ans=0.0
2023-11-24 00:08:10,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2597553.3333333335, ans=0.125
2023-11-24 00:08:19,546 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.16 vs. limit=22.5
2023-11-24 00:08:21,553 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389650
2023-11-24 00:08:21,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2597620.0, ans=0.125
2023-11-24 00:08:27,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2597686.6666666665, ans=0.125
2023-11-24 00:08:28,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2597686.6666666665, ans=0.125
2023-11-24 00:08:39,296 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4900, loss[loss=0.06478, simple_loss=0.08538, pruned_loss=0.01454, audio_tagging_loss=0.007553, over 14603.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09154, pruned_loss=0.01357, audio_tagging_loss=0.009243, over 3046450.90 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:08:53,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2597820.0, ans=0.0
2023-11-24 00:08:55,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2597820.0, ans=0.0
2023-11-24 00:09:08,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2597886.6666666665, ans=0.125
2023-11-24 00:09:17,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2597953.3333333335, ans=0.2
2023-11-24 00:09:23,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389700
2023-11-24 00:09:43,037 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.493e+01 8.944e+01 9.853e+01 1.226e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-24 00:09:43,084 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 4950, loss[loss=0.0679, simple_loss=0.0952, pruned_loss=0.01239, audio_tagging_loss=0.007911, over 14952.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09156, pruned_loss=0.01338, audio_tagging_loss=0.009105, over 3043148.34 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:09:44,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2598086.6666666665, ans=0.0
2023-11-24 00:09:59,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.02 vs. limit=10.0
2023-11-24 00:10:14,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.87 vs. limit=22.5
2023-11-24 00:10:17,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2598220.0, ans=0.05
2023-11-24 00:10:19,914 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.65 vs. limit=15.0
2023-11-24 00:10:21,328 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.58 vs. limit=6.0
2023-11-24 00:10:26,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389750
2023-11-24 00:10:35,715 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.73 vs. limit=15.0
2023-11-24 00:10:45,885 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5000, loss[loss=0.06402, simple_loss=0.08667, pruned_loss=0.01019, audio_tagging_loss=0.0105, over 14849.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09133, pruned_loss=0.01326, audio_tagging_loss=0.008995, over 3042436.06 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:10:49,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2598420.0, ans=0.125
2023-11-24 00:10:58,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2598486.6666666665, ans=0.125
2023-11-24 00:10:58,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2598486.6666666665, ans=0.0
2023-11-24 00:11:18,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2598553.3333333335, ans=0.1
2023-11-24 00:11:29,702 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389800
2023-11-24 00:11:36,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2598686.6666666665, ans=0.125
2023-11-24 00:11:47,899 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.099e+01 8.775e+01 9.573e+01 1.290e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-24 00:11:47,944 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5050, loss[loss=0.07353, simple_loss=0.0959, pruned_loss=0.01643, audio_tagging_loss=0.009159, over 15098.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.0915, pruned_loss=0.01339, audio_tagging_loss=0.008989, over 3040021.62 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:11:58,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.31 vs. limit=15.0
2023-11-24 00:12:02,228 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.53 vs. limit=15.0
2023-11-24 00:12:10,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2598820.0, ans=0.125
2023-11-24 00:12:29,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2598953.3333333335, ans=0.0
2023-11-24 00:12:32,490 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389850
2023-11-24 00:12:50,816 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5100, loss[loss=0.04999, simple_loss=0.06444, pruned_loss=0.009003, audio_tagging_loss=0.008761, over 14413.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09163, pruned_loss=0.01342, audio_tagging_loss=0.008897, over 3040480.03 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:12:55,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2599086.6666666665, ans=0.2
2023-11-24 00:13:13,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.84 vs. limit=15.0
2023-11-24 00:13:25,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2599220.0, ans=0.1
2023-11-24 00:13:32,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2599286.6666666665, ans=0.125
2023-11-24 00:13:33,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2599286.6666666665, ans=0.125
2023-11-24 00:13:34,792 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389900
2023-11-24 00:13:41,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2599353.3333333335, ans=0.0
2023-11-24 00:13:43,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn2.whiten.whitening_limit, batch_count=2599353.3333333335, ans=22.5
2023-11-24 00:13:51,354 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.89 vs. limit=15.0
2023-11-24 00:13:54,294 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.310e+01 9.019e+01 9.639e+01 1.666e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-24 00:13:54,348 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5150, loss[loss=0.08476, simple_loss=0.1175, pruned_loss=0.01836, audio_tagging_loss=0.007644, over 15790.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.092, pruned_loss=0.01345, audio_tagging_loss=0.008917, over 3037148.37 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:14:00,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.83 vs. limit=15.0
2023-11-24 00:14:12,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2599486.6666666665, ans=0.2
2023-11-24 00:14:20,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2599553.3333333335, ans=0.0
2023-11-24 00:14:20,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2599553.3333333335, ans=0.125
2023-11-24 00:14:34,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2599620.0, ans=0.125
2023-11-24 00:14:38,583 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 389950
2023-11-24 00:14:56,533 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5200, loss[loss=0.07069, simple_loss=0.09113, pruned_loss=0.01397, audio_tagging_loss=0.01116, over 16589.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09212, pruned_loss=0.01338, audio_tagging_loss=0.008986, over 3033756.37 frames. ], batch size: 62, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:14:57,250 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.23 vs. limit=15.0
2023-11-24 00:14:58,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2599753.3333333335, ans=0.125
2023-11-24 00:15:10,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2599820.0, ans=0.125
2023-11-24 00:15:28,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2599886.6666666665, ans=0.2
2023-11-24 00:15:32,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2599886.6666666665, ans=0.09899494936611666
2023-11-24 00:15:41,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390000
2023-11-24 00:15:49,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2600020.0, ans=0.0
2023-11-24 00:15:55,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2600020.0, ans=0.1
2023-11-24 00:15:57,992 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.36 vs. limit=15.0
2023-11-24 00:16:00,312 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.701e+01 8.431e+01 8.968e+01 9.666e+01 1.343e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-24 00:16:00,358 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5250, loss[loss=0.0508, simple_loss=0.06577, pruned_loss=0.009021, audio_tagging_loss=0.008889, over 14892.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09247, pruned_loss=0.01347, audio_tagging_loss=0.008901, over 3040680.38 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:16:02,266 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.41 vs. limit=12.0
2023-11-24 00:16:08,739 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.00 vs. limit=15.0
2023-11-24 00:16:24,373 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.72 vs. limit=10.0
2023-11-24 00:16:43,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390050
2023-11-24 00:16:58,909 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.57 vs. limit=22.5
2023-11-24 00:17:03,032 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5300, loss[loss=0.08423, simple_loss=0.1202, pruned_loss=0.0187, audio_tagging_loss=0.005448, over 15194.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09262, pruned_loss=0.0135, audio_tagging_loss=0.008817, over 3040612.37 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:17:03,533 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.67 vs. limit=15.0
2023-11-24 00:17:04,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2600420.0, ans=0.125
2023-11-24 00:17:25,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.05 vs. limit=15.0
2023-11-24 00:17:33,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2600553.3333333335, ans=0.125
2023-11-24 00:17:34,350 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.31 vs. limit=22.5
2023-11-24 00:17:37,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2600553.3333333335, ans=0.125
2023-11-24 00:17:45,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2600620.0, ans=0.125
2023-11-24 00:17:46,793 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390100
2023-11-24 00:17:50,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2600620.0, ans=0.125
2023-11-24 00:18:04,824 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5350, loss[loss=0.05434, simple_loss=0.07254, pruned_loss=0.00923, audio_tagging_loss=0.008842, over 14364.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09221, pruned_loss=0.01344, audio_tagging_loss=0.008842, over 3036684.76 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:18:05,947 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.889e+01 8.377e+01 8.739e+01 9.630e+01 1.201e+02, threshold=1.748e+02, percent-clipped=0.0
2023-11-24 00:18:23,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2600820.0, ans=0.0
2023-11-24 00:18:24,979 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.28 vs. limit=10.0
2023-11-24 00:18:25,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2600820.0, ans=0.09899494936611666
2023-11-24 00:18:37,582 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:18:48,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390150
2023-11-24 00:18:57,496 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.29 vs. limit=15.0
2023-11-24 00:19:00,929 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.07 vs. limit=15.0
2023-11-24 00:19:06,221 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5400, loss[loss=0.06972, simple_loss=0.09381, pruned_loss=0.01311, audio_tagging_loss=0.009708, over 14339.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09188, pruned_loss=0.01321, audio_tagging_loss=0.008873, over 3037689.00 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:19:30,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2601220.0, ans=0.05
2023-11-24 00:19:50,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390200
2023-11-24 00:20:09,853 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5450, loss[loss=0.07904, simple_loss=0.1087, pruned_loss=0.01442, audio_tagging_loss=0.01026, over 14370.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09289, pruned_loss=0.01355, audio_tagging_loss=0.00884, over 3037286.96 frames. ], batch size: 54, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:20:10,929 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.432e+01 8.413e+01 8.965e+01 9.765e+01 1.201e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 00:20:52,461 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390250
2023-11-24 00:21:11,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.56 vs. limit=10.0
2023-11-24 00:21:11,503 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5500, loss[loss=0.06085, simple_loss=0.06977, pruned_loss=0.01269, audio_tagging_loss=0.01327, over 15446.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09238, pruned_loss=0.0135, audio_tagging_loss=0.008904, over 3034022.74 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:21:42,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.91 vs. limit=15.0
2023-11-24 00:21:50,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2601953.3333333335, ans=0.125
2023-11-24 00:21:54,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.32 vs. limit=15.0
2023-11-24 00:21:55,011 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390300
2023-11-24 00:21:57,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2601953.3333333335, ans=0.125
2023-11-24 00:22:05,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2602020.0, ans=0.0
2023-11-24 00:22:13,318 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5550, loss[loss=0.06896, simple_loss=0.08399, pruned_loss=0.01266, audio_tagging_loss=0.0143, over 15051.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09143, pruned_loss=0.01336, audio_tagging_loss=0.009087, over 3035861.18 frames. ], batch size: 61, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:22:14,449 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.193e+01 8.280e+01 9.193e+01 9.915e+01 1.422e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 00:22:38,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2602220.0, ans=0.0
2023-11-24 00:22:38,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2602220.0, ans=0.1
2023-11-24 00:22:57,021 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390350
2023-11-24 00:23:00,104 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.75 vs. limit=15.0
2023-11-24 00:23:08,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2602353.3333333335, ans=0.0
2023-11-24 00:23:16,746 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5600, loss[loss=0.048, simple_loss=0.05333, pruned_loss=0.009505, audio_tagging_loss=0.01183, over 15099.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09097, pruned_loss=0.01331, audio_tagging_loss=0.009136, over 3037707.77 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:23:22,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2602420.0, ans=0.2
2023-11-24 00:23:28,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2602486.6666666665, ans=0.0
2023-11-24 00:23:37,130 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.53 vs. limit=5.0
2023-11-24 00:23:56,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=2602620.0, ans=15.0
2023-11-24 00:23:59,845 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390400
2023-11-24 00:24:00,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2602620.0, ans=0.125
2023-11-24 00:24:01,279 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 00:24:06,004 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.73 vs. limit=6.0
2023-11-24 00:24:11,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2602686.6666666665, ans=0.1
2023-11-24 00:24:18,482 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5650, loss[loss=0.048, simple_loss=0.05929, pruned_loss=0.007803, audio_tagging_loss=0.01055, over 16023.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.0902, pruned_loss=0.0132, audio_tagging_loss=0.00925, over 3045966.19 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:24:19,627 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.276e+01 8.461e+01 8.950e+01 9.725e+01 1.309e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-24 00:24:54,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2602886.6666666665, ans=0.125
2023-11-24 00:25:00,417 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.02 vs. limit=6.0
2023-11-24 00:25:02,295 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390450
2023-11-24 00:25:10,029 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.02 vs. limit=22.5
2023-11-24 00:25:14,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2603020.0, ans=0.0
2023-11-24 00:25:18,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2603020.0, ans=0.0
2023-11-24 00:25:20,369 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5700, loss[loss=0.07338, simple_loss=0.1063, pruned_loss=0.01218, audio_tagging_loss=0.008073, over 16308.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09132, pruned_loss=0.0133, audio_tagging_loss=0.009187, over 3041879.24 frames. ], batch size: 60, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:25:29,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2603086.6666666665, ans=0.2
2023-11-24 00:25:37,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2603153.3333333335, ans=0.125
2023-11-24 00:26:04,000 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390500
2023-11-24 00:26:09,482 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.71 vs. limit=22.5
2023-11-24 00:26:12,210 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.12 vs. limit=5.0
2023-11-24 00:26:23,066 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5750, loss[loss=0.06275, simple_loss=0.07289, pruned_loss=0.01501, audio_tagging_loss=0.0113, over 15228.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.0912, pruned_loss=0.0134, audio_tagging_loss=0.00901, over 3051143.91 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 32.0
2023-11-24 00:26:24,164 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.481e+01 9.036e+01 9.796e+01 1.280e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 00:26:27,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2603420.0, ans=0.125
2023-11-24 00:26:45,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=2603486.6666666665, ans=0.025
2023-11-24 00:26:58,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2603620.0, ans=0.2
2023-11-24 00:26:59,238 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.80 vs. limit=22.5
2023-11-24 00:27:01,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2603620.0, ans=0.125
2023-11-24 00:27:06,364 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390550
2023-11-24 00:27:07,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2603620.0, ans=0.5
2023-11-24 00:27:07,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2603620.0, ans=0.025
2023-11-24 00:27:14,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2603686.6666666665, ans=0.125
2023-11-24 00:27:25,136 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5800, loss[loss=0.07392, simple_loss=0.1073, pruned_loss=0.01281, audio_tagging_loss=0.007476, over 14350.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09149, pruned_loss=0.01345, audio_tagging_loss=0.008908, over 3047991.85 frames. ], batch size: 53, lr: 2.06e-03, grad_scale: 8.0
2023-11-24 00:27:34,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2603753.3333333335, ans=0.0
2023-11-24 00:27:43,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2603820.0, ans=0.125
2023-11-24 00:27:51,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2603886.6666666665, ans=0.2
2023-11-24 00:28:08,517 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390600
2023-11-24 00:28:19,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2604020.0, ans=0.125
2023-11-24 00:28:20,054 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0
2023-11-24 00:28:26,603 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5850, loss[loss=0.07065, simple_loss=0.09925, pruned_loss=0.0113, audio_tagging_loss=0.009724, over 13841.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09214, pruned_loss=0.0135, audio_tagging_loss=0.00881, over 3045066.64 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 8.0
2023-11-24 00:28:30,556 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.606e+01 8.379e+01 8.883e+01 9.652e+01 2.888e+02, threshold=1.777e+02, percent-clipped=1.0
2023-11-24 00:28:54,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2604220.0, ans=0.125
2023-11-24 00:29:06,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2604286.6666666665, ans=0.1
2023-11-24 00:29:10,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390650
2023-11-24 00:29:11,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2604286.6666666665, ans=0.0
2023-11-24 00:29:15,743 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.85 vs. limit=10.0
2023-11-24 00:29:18,622 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.55 vs. limit=10.0
2023-11-24 00:29:23,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2604353.3333333335, ans=0.125
2023-11-24 00:29:29,333 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5900, loss[loss=0.04249, simple_loss=0.04903, pruned_loss=0.00679, audio_tagging_loss=0.01119, over 14391.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09132, pruned_loss=0.0133, audio_tagging_loss=0.008861, over 3043466.29 frames. ], batch size: 55, lr: 2.06e-03, grad_scale: 8.0
2023-11-24 00:29:38,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2604420.0, ans=0.1
2023-11-24 00:29:53,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2604553.3333333335, ans=0.0
2023-11-24 00:29:56,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2604553.3333333335, ans=0.125
2023-11-24 00:30:13,465 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390700
2023-11-24 00:30:13,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2604620.0, ans=0.125
2023-11-24 00:30:29,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2604686.6666666665, ans=0.125
2023-11-24 00:30:32,177 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 5950, loss[loss=0.04234, simple_loss=0.05613, pruned_loss=0.006408, audio_tagging_loss=0.007866, over 16010.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09204, pruned_loss=0.0135, audio_tagging_loss=0.008796, over 3046828.11 frames. ], batch size: 62, lr: 2.06e-03, grad_scale: 8.0
2023-11-24 00:30:35,901 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.451e+01 9.116e+01 9.981e+01 1.115e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 00:30:54,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2604820.0, ans=0.125
2023-11-24 00:31:07,149 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:31:15,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390750
2023-11-24 00:31:23,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2605020.0, ans=0.1
2023-11-24 00:31:33,667 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6000, loss[loss=0.06425, simple_loss=0.08966, pruned_loss=0.0126, audio_tagging_loss=0.006825, over 16049.00 frames. ], tot_loss[loss=0.06869, simple_loss=0.09246, pruned_loss=0.01374, audio_tagging_loss=0.008727, over 3050658.19 frames. ], batch size: 58, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:31:33,671 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 00:32:09,902 INFO [train_asr.py:1253] (0/4) Epoch 33, validation: loss=0.05769, simple_loss=0.05098, pruned_loss=0.005124, audio_tagging_loss=0.02707, over 4681554.00 frames. 
2023-11-24 00:32:09,903 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 00:32:10,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2605086.6666666665, ans=0.0
2023-11-24 00:32:29,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2605153.3333333335, ans=0.125
2023-11-24 00:32:33,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2605220.0, ans=0.2
2023-11-24 00:32:49,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2605286.6666666665, ans=0.0
2023-11-24 00:32:52,908 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390800
2023-11-24 00:32:55,893 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 00:33:01,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2605353.3333333335, ans=0.0
2023-11-24 00:33:11,949 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6050, loss[loss=0.07552, simple_loss=0.09692, pruned_loss=0.01651, audio_tagging_loss=0.01055, over 15437.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09245, pruned_loss=0.01367, audio_tagging_loss=0.0087, over 3052471.09 frames. ], batch size: 57, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:33:12,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2605420.0, ans=0.125
2023-11-24 00:33:14,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2605420.0, ans=0.125
2023-11-24 00:33:15,470 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.390e+01 8.432e+01 8.885e+01 9.962e+01 1.302e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-24 00:33:23,585 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.41 vs. limit=5.0
2023-11-24 00:33:30,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.89 vs. limit=15.0
2023-11-24 00:33:31,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2605486.6666666665, ans=0.125
2023-11-24 00:33:50,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2605620.0, ans=0.125
2023-11-24 00:33:55,007 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390850
2023-11-24 00:33:56,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2605620.0, ans=0.125
2023-11-24 00:33:58,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2605620.0, ans=0.0
2023-11-24 00:34:10,547 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2605686.6666666665, ans=0.0
2023-11-24 00:34:12,719 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6100, loss[loss=0.04736, simple_loss=0.05565, pruned_loss=0.007468, audio_tagging_loss=0.01206, over 15317.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09204, pruned_loss=0.01358, audio_tagging_loss=0.0088, over 3047313.62 frames. ], batch size: 59, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:34:23,995 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.41 vs. limit=22.5
2023-11-24 00:34:30,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2605820.0, ans=0.0
2023-11-24 00:34:31,310 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:34:56,488 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390900
2023-11-24 00:35:00,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2605953.3333333335, ans=0.125
2023-11-24 00:35:14,634 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6150, loss[loss=0.0685, simple_loss=0.09603, pruned_loss=0.01163, audio_tagging_loss=0.008856, over 15298.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09271, pruned_loss=0.01368, audio_tagging_loss=0.008868, over 3054418.57 frames. ], batch size: 56, lr: 2.06e-03, grad_scale: 16.0
2023-11-24 00:35:19,341 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.208e+01 8.886e+01 9.597e+01 1.503e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-24 00:35:31,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2606153.3333333335, ans=0.125
2023-11-24 00:35:57,647 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 390950
2023-11-24 00:36:03,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2606353.3333333335, ans=0.0
2023-11-24 00:36:14,927 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.74 vs. limit=15.0
2023-11-24 00:36:17,591 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6200, loss[loss=0.05782, simple_loss=0.08168, pruned_loss=0.008852, audio_tagging_loss=0.008129, over 15939.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.0918, pruned_loss=0.01344, audio_tagging_loss=0.008953, over 3053313.09 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:36:32,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2606486.6666666665, ans=0.1
2023-11-24 00:36:38,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2606486.6666666665, ans=0.125
2023-11-24 00:36:39,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2606486.6666666665, ans=0.1
2023-11-24 00:36:44,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2606553.3333333335, ans=0.125
2023-11-24 00:37:01,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2606620.0, ans=0.0
2023-11-24 00:37:02,078 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391000
2023-11-24 00:37:20,042 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6250, loss[loss=0.06941, simple_loss=0.09744, pruned_loss=0.01167, audio_tagging_loss=0.009023, over 15199.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09191, pruned_loss=0.01356, audio_tagging_loss=0.009081, over 3048980.54 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:37:23,528 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.852e+01 8.315e+01 8.981e+01 9.556e+01 1.931e+02, threshold=1.796e+02, percent-clipped=1.0
2023-11-24 00:37:26,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2606753.3333333335, ans=0.2
2023-11-24 00:37:27,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2606753.3333333335, ans=0.0
2023-11-24 00:37:40,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2606820.0, ans=0.1
2023-11-24 00:38:04,302 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391050
2023-11-24 00:38:14,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2607020.0, ans=0.125
2023-11-24 00:38:15,721 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.58 vs. limit=22.5
2023-11-24 00:38:21,065 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:38:22,594 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6300, loss[loss=0.08592, simple_loss=0.1103, pruned_loss=0.02128, audio_tagging_loss=0.009463, over 16209.00 frames. ], tot_loss[loss=0.06837, simple_loss=0.09129, pruned_loss=0.01359, audio_tagging_loss=0.009137, over 3051991.72 frames. ], batch size: 62, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:38:22,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2607086.6666666665, ans=0.125
2023-11-24 00:38:43,994 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.18 vs. limit=6.0
2023-11-24 00:38:51,057 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.91 vs. limit=22.5
2023-11-24 00:39:00,065 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:39:02,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2607286.6666666665, ans=0.1
2023-11-24 00:39:05,783 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391100
2023-11-24 00:39:12,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2607353.3333333335, ans=0.125
2023-11-24 00:39:12,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2607353.3333333335, ans=0.125
2023-11-24 00:39:19,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2607353.3333333335, ans=0.1
2023-11-24 00:39:25,696 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6350, loss[loss=0.08485, simple_loss=0.1204, pruned_loss=0.01775, audio_tagging_loss=0.006901, over 16268.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.0907, pruned_loss=0.01328, audio_tagging_loss=0.009263, over 3050196.39 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:39:29,228 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.017e+01 8.323e+01 9.041e+01 9.620e+01 1.215e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 00:39:39,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2607486.6666666665, ans=0.1
2023-11-24 00:39:45,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2607486.6666666665, ans=0.125
2023-11-24 00:40:01,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2607620.0, ans=0.125
2023-11-24 00:40:05,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2607620.0, ans=0.125
2023-11-24 00:40:09,159 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391150
2023-11-24 00:40:27,283 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6400, loss[loss=0.05941, simple_loss=0.07557, pruned_loss=0.008512, audio_tagging_loss=0.01311, over 14903.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09009, pruned_loss=0.01311, audio_tagging_loss=0.009348, over 3044763.19 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 00:40:39,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=2607820.0, ans=10.0
2023-11-24 00:40:41,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2607820.0, ans=0.1
2023-11-24 00:40:47,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2607820.0, ans=0.2
2023-11-24 00:41:11,020 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391200
2023-11-24 00:41:16,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2608020.0, ans=0.125
2023-11-24 00:41:23,982 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.68 vs. limit=15.0
2023-11-24 00:41:28,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2608086.6666666665, ans=0.125
2023-11-24 00:41:29,313 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6450, loss[loss=0.08201, simple_loss=0.1017, pruned_loss=0.01988, audio_tagging_loss=0.01126, over 15053.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09123, pruned_loss=0.01324, audio_tagging_loss=0.009304, over 3046958.31 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:41:34,712 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.651e+01 8.349e+01 8.985e+01 9.691e+01 1.713e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-24 00:42:12,466 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=15.0
2023-11-24 00:42:14,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391250
2023-11-24 00:42:28,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2608353.3333333335, ans=0.125
2023-11-24 00:42:33,542 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6500, loss[loss=0.07639, simple_loss=0.09927, pruned_loss=0.0165, audio_tagging_loss=0.01025, over 14273.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09052, pruned_loss=0.01316, audio_tagging_loss=0.00941, over 3050146.16 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:42:43,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2608420.0, ans=0.0
2023-11-24 00:42:49,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2608486.6666666665, ans=0.04949747468305833
2023-11-24 00:42:56,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2608553.3333333335, ans=0.1
2023-11-24 00:43:17,563 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391300
2023-11-24 00:43:19,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2608620.0, ans=0.1
2023-11-24 00:43:35,734 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6550, loss[loss=0.0672, simple_loss=0.08832, pruned_loss=0.01359, audio_tagging_loss=0.009445, over 15029.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09014, pruned_loss=0.01316, audio_tagging_loss=0.009365, over 3053375.40 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:43:40,540 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.079e+01 8.397e+01 9.072e+01 9.736e+01 1.207e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 00:43:42,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2608753.3333333335, ans=0.0
2023-11-24 00:43:50,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2608820.0, ans=0.125
2023-11-24 00:44:11,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2608886.6666666665, ans=0.125
2023-11-24 00:44:19,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391350
2023-11-24 00:44:22,305 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:44:35,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2609020.0, ans=0.0
2023-11-24 00:44:37,632 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6600, loss[loss=0.05947, simple_loss=0.08651, pruned_loss=0.008886, audio_tagging_loss=0.007331, over 15091.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09057, pruned_loss=0.01335, audio_tagging_loss=0.009217, over 3047118.95 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:44:43,889 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.21 vs. limit=10.0
2023-11-24 00:44:53,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2609153.3333333335, ans=0.2
2023-11-24 00:45:21,721 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391400
2023-11-24 00:45:32,222 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:45:41,432 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6650, loss[loss=0.07795, simple_loss=0.09963, pruned_loss=0.01895, audio_tagging_loss=0.009186, over 14668.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09054, pruned_loss=0.0134, audio_tagging_loss=0.009118, over 3048531.49 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:45:46,131 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.357e+01 8.351e+01 8.932e+01 9.624e+01 1.119e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 00:46:16,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2609620.0, ans=0.2
2023-11-24 00:46:16,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2609620.0, ans=0.2
2023-11-24 00:46:19,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2609620.0, ans=0.125
2023-11-24 00:46:25,050 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391450
2023-11-24 00:46:25,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2609620.0, ans=15.0
2023-11-24 00:46:42,712 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6700, loss[loss=0.08202, simple_loss=0.1009, pruned_loss=0.02227, audio_tagging_loss=0.009295, over 14254.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09098, pruned_loss=0.01353, audio_tagging_loss=0.008959, over 3048910.01 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:46:52,515 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.87 vs. limit=6.0
2023-11-24 00:47:23,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2609953.3333333335, ans=0.07
2023-11-24 00:47:26,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391500
2023-11-24 00:47:34,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2610020.0, ans=0.125
2023-11-24 00:47:39,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2610020.0, ans=0.0
2023-11-24 00:47:45,023 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6750, loss[loss=0.0836, simple_loss=0.1249, pruned_loss=0.01496, audio_tagging_loss=0.006212, over 15280.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09044, pruned_loss=0.0133, audio_tagging_loss=0.008888, over 3044949.76 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:47:47,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2610086.6666666665, ans=0.0
2023-11-24 00:47:49,705 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.433e+01 8.920e+01 9.710e+01 1.340e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-24 00:47:54,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2610086.6666666665, ans=0.1
2023-11-24 00:47:55,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2610086.6666666665, ans=0.0
2023-11-24 00:48:13,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2610220.0, ans=0.125
2023-11-24 00:48:17,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2610220.0, ans=0.125
2023-11-24 00:48:28,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391550
2023-11-24 00:48:31,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2610286.6666666665, ans=0.125
2023-11-24 00:48:32,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2610286.6666666665, ans=0.125
2023-11-24 00:48:42,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2610353.3333333335, ans=0.0
2023-11-24 00:48:44,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2610353.3333333335, ans=0.125
2023-11-24 00:48:48,397 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6800, loss[loss=0.06897, simple_loss=0.09569, pruned_loss=0.01358, audio_tagging_loss=0.00754, over 16843.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09041, pruned_loss=0.01334, audio_tagging_loss=0.008913, over 3046342.73 frames. ], batch size: 62, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 00:48:49,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2610420.0, ans=0.0
2023-11-24 00:49:20,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2610553.3333333335, ans=0.125
2023-11-24 00:49:22,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2610553.3333333335, ans=0.035
2023-11-24 00:49:30,705 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.39 vs. limit=15.0
2023-11-24 00:49:31,711 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391600
2023-11-24 00:49:31,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2610620.0, ans=0.0
2023-11-24 00:49:50,169 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6850, loss[loss=0.09549, simple_loss=0.1208, pruned_loss=0.02514, audio_tagging_loss=0.009934, over 14658.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09035, pruned_loss=0.0134, audio_tagging_loss=0.008939, over 3042897.49 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:49:56,145 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.614e+01 8.434e+01 9.033e+01 9.858e+01 1.368e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 00:50:23,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2610886.6666666665, ans=0.1
2023-11-24 00:50:25,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2610886.6666666665, ans=0.1
2023-11-24 00:50:28,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2610953.3333333335, ans=0.125
2023-11-24 00:50:34,017 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391650
2023-11-24 00:50:41,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2611020.0, ans=0.0
2023-11-24 00:50:52,595 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6900, loss[loss=0.07474, simple_loss=0.09722, pruned_loss=0.01853, audio_tagging_loss=0.007592, over 15212.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09221, pruned_loss=0.0137, audio_tagging_loss=0.008838, over 3046323.52 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:51:07,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2611153.3333333335, ans=0.07
2023-11-24 00:51:26,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2611220.0, ans=0.125
2023-11-24 00:51:36,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391700
2023-11-24 00:51:40,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.96 vs. limit=15.0
2023-11-24 00:51:41,371 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 00:51:45,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2611353.3333333335, ans=0.125
2023-11-24 00:51:55,499 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 6950, loss[loss=0.07194, simple_loss=0.1057, pruned_loss=0.01139, audio_tagging_loss=0.007719, over 14972.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09195, pruned_loss=0.01364, audio_tagging_loss=0.008819, over 3051970.89 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:52:03,202 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.470e+01 9.036e+01 9.958e+01 1.197e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 00:52:29,120 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.08 vs. limit=15.0
2023-11-24 00:52:33,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2611620.0, ans=0.2
2023-11-24 00:52:35,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2611620.0, ans=0.125
2023-11-24 00:52:38,493 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391750
2023-11-24 00:52:49,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2611686.6666666665, ans=0.0
2023-11-24 00:52:57,555 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7000, loss[loss=0.0943, simple_loss=0.1452, pruned_loss=0.01589, audio_tagging_loss=0.005809, over 15816.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09275, pruned_loss=0.01367, audio_tagging_loss=0.008881, over 3050603.63 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:52:59,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2611753.3333333335, ans=0.125
2023-11-24 00:53:03,003 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.46 vs. limit=6.0
2023-11-24 00:53:03,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2611753.3333333335, ans=0.125
2023-11-24 00:53:16,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2611820.0, ans=0.0
2023-11-24 00:53:24,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2611886.6666666665, ans=0.2
2023-11-24 00:53:32,276 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.83 vs. limit=15.0
2023-11-24 00:53:41,595 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391800
2023-11-24 00:53:59,714 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7050, loss[loss=0.05732, simple_loss=0.08218, pruned_loss=0.006541, audio_tagging_loss=0.009688, over 14556.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09169, pruned_loss=0.01354, audio_tagging_loss=0.009027, over 3054171.09 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:54:07,537 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.402e+01 9.074e+01 1.006e+02 1.351e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 00:54:24,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2612153.3333333335, ans=0.125
2023-11-24 00:54:40,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2612286.6666666665, ans=0.125
2023-11-24 00:54:43,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391850
2023-11-24 00:55:02,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2612420.0, ans=0.125
2023-11-24 00:55:02,811 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7100, loss[loss=0.07537, simple_loss=0.1036, pruned_loss=0.01536, audio_tagging_loss=0.008215, over 16020.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09135, pruned_loss=0.01346, audio_tagging_loss=0.00913, over 3050429.29 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:55:14,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2612486.6666666665, ans=0.07
2023-11-24 00:55:27,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2612553.3333333335, ans=0.125
2023-11-24 00:55:37,388 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.87 vs. limit=22.5
2023-11-24 00:55:40,697 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:55:42,495 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:55:45,725 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391900
2023-11-24 00:55:55,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2612686.6666666665, ans=0.125
2023-11-24 00:55:55,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2612686.6666666665, ans=0.0
2023-11-24 00:56:01,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2612686.6666666665, ans=0.125
2023-11-24 00:56:04,789 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.21 vs. limit=15.0
2023-11-24 00:56:05,204 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7150, loss[loss=0.06022, simple_loss=0.07654, pruned_loss=0.01295, audio_tagging_loss=0.008994, over 14231.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09146, pruned_loss=0.01348, audio_tagging_loss=0.009148, over 3046191.98 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 00:56:10,571 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0
2023-11-24 00:56:11,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2612753.3333333335, ans=0.125
2023-11-24 00:56:12,275 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.300e+01 8.358e+01 9.047e+01 9.923e+01 1.303e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 00:56:14,258 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.63 vs. limit=15.0
2023-11-24 00:56:43,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2612953.3333333335, ans=0.125
2023-11-24 00:56:49,181 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 391950
2023-11-24 00:56:52,081 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.11 vs. limit=15.0
2023-11-24 00:57:02,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2613020.0, ans=0.125
2023-11-24 00:57:05,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2613086.6666666665, ans=0.125
2023-11-24 00:57:06,669 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7200, loss[loss=0.05233, simple_loss=0.06573, pruned_loss=0.01021, audio_tagging_loss=0.009257, over 14855.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09151, pruned_loss=0.01344, audio_tagging_loss=0.009253, over 3051460.02 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:57:50,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392000
2023-11-24 00:57:52,072 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-392000.pt
2023-11-24 00:58:05,208 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.71 vs. limit=10.0
2023-11-24 00:58:12,148 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7250, loss[loss=0.0673, simple_loss=0.09029, pruned_loss=0.01337, audio_tagging_loss=0.008786, over 15677.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09168, pruned_loss=0.01353, audio_tagging_loss=0.009264, over 3053812.22 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:58:20,980 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.420e+01 8.844e+01 9.320e+01 1.006e+02 1.273e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 00:58:29,800 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.74 vs. limit=10.0
2023-11-24 00:58:49,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2613620.0, ans=0.1
2023-11-24 00:58:55,271 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392050
2023-11-24 00:59:00,832 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 00:59:12,418 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.46 vs. limit=22.5
2023-11-24 00:59:15,266 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7300, loss[loss=0.08805, simple_loss=0.1209, pruned_loss=0.01886, audio_tagging_loss=0.008736, over 15239.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.092, pruned_loss=0.01352, audio_tagging_loss=0.009153, over 3046110.86 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 00:59:20,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2613753.3333333335, ans=0.125
2023-11-24 00:59:35,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2613820.0, ans=0.0
2023-11-24 00:59:41,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2613886.6666666665, ans=0.125
2023-11-24 00:59:46,595 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.11 vs. limit=15.0
2023-11-24 00:59:58,589 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392100
2023-11-24 01:00:03,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2614020.0, ans=0.2
2023-11-24 01:00:07,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2614020.0, ans=0.0
2023-11-24 01:00:09,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2614020.0, ans=0.125
2023-11-24 01:00:16,240 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7350, loss[loss=0.0553, simple_loss=0.07452, pruned_loss=0.009089, audio_tagging_loss=0.008951, over 14596.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.092, pruned_loss=0.01345, audio_tagging_loss=0.008942, over 3047844.26 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:00:17,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2614086.6666666665, ans=0.1
2023-11-24 01:00:23,328 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.380e+01 8.518e+01 9.030e+01 9.706e+01 1.263e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 01:00:46,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2614220.0, ans=0.125
2023-11-24 01:00:55,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2614286.6666666665, ans=0.125
2023-11-24 01:01:00,297 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392150
2023-11-24 01:01:02,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2614286.6666666665, ans=0.0
2023-11-24 01:01:04,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2614286.6666666665, ans=0.125
2023-11-24 01:01:09,398 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.07 vs. limit=15.0
2023-11-24 01:01:18,052 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7400, loss[loss=0.05056, simple_loss=0.06616, pruned_loss=0.008832, audio_tagging_loss=0.008653, over 14486.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09167, pruned_loss=0.01357, audio_tagging_loss=0.008789, over 3043529.84 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:01:23,666 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:01:50,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2614553.3333333335, ans=0.1
2023-11-24 01:02:02,050 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392200
2023-11-24 01:02:02,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2614620.0, ans=0.0
2023-11-24 01:02:06,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2614620.0, ans=0.125
2023-11-24 01:02:11,596 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.96 vs. limit=10.0
2023-11-24 01:02:21,666 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7450, loss[loss=0.07264, simple_loss=0.09778, pruned_loss=0.01626, audio_tagging_loss=0.007486, over 15241.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09163, pruned_loss=0.01358, audio_tagging_loss=0.008722, over 3042028.99 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:02:26,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2614753.3333333335, ans=0.0
2023-11-24 01:02:26,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2614753.3333333335, ans=0.125
2023-11-24 01:02:26,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2614753.3333333335, ans=0.125
2023-11-24 01:02:26,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2614753.3333333335, ans=0.0
2023-11-24 01:02:28,727 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.098e+01 8.459e+01 9.049e+01 9.753e+01 1.412e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 01:02:32,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2614820.0, ans=0.125
2023-11-24 01:02:39,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2614820.0, ans=0.0
2023-11-24 01:02:51,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2614886.6666666665, ans=0.125
2023-11-24 01:03:05,977 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392250
2023-11-24 01:03:23,819 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7500, loss[loss=0.05599, simple_loss=0.07139, pruned_loss=0.01263, audio_tagging_loss=0.007659, over 14631.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09084, pruned_loss=0.01341, audio_tagging_loss=0.008825, over 3039787.97 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:03:34,055 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.00 vs. limit=22.5
2023-11-24 01:03:56,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.46 vs. limit=15.0
2023-11-24 01:04:06,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2615286.6666666665, ans=0.2
2023-11-24 01:04:07,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392300
2023-11-24 01:04:09,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2615286.6666666665, ans=0.125
2023-11-24 01:04:23,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2615353.3333333335, ans=0.0
2023-11-24 01:04:25,649 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7550, loss[loss=0.0635, simple_loss=0.08022, pruned_loss=0.01484, audio_tagging_loss=0.008551, over 14620.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.0902, pruned_loss=0.01326, audio_tagging_loss=0.008817, over 3045731.91 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:04:29,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2615420.0, ans=0.125
2023-11-24 01:04:33,432 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.868e+01 8.602e+01 9.110e+01 9.644e+01 1.333e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 01:04:35,342 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.86 vs. limit=15.0
2023-11-24 01:04:49,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1.whitening_limit, batch_count=2615486.6666666665, ans=10.0
2023-11-24 01:04:50,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2615553.3333333335, ans=0.125
2023-11-24 01:04:50,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2615553.3333333335, ans=0.125
2023-11-24 01:05:09,883 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392350
2023-11-24 01:05:25,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2615686.6666666665, ans=0.2
2023-11-24 01:05:29,046 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7600, loss[loss=0.05501, simple_loss=0.06367, pruned_loss=0.01243, audio_tagging_loss=0.01074, over 15321.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09031, pruned_loss=0.01314, audio_tagging_loss=0.008855, over 3045081.97 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 01:05:42,125 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.48 vs. limit=15.0
2023-11-24 01:05:51,816 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.08 vs. limit=15.0
2023-11-24 01:06:00,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2615886.6666666665, ans=0.0
2023-11-24 01:06:03,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2615886.6666666665, ans=0.0
2023-11-24 01:06:12,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392400
2023-11-24 01:06:30,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2616086.6666666665, ans=0.125
2023-11-24 01:06:31,608 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7650, loss[loss=0.06035, simple_loss=0.07583, pruned_loss=0.01372, audio_tagging_loss=0.008716, over 15752.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.08973, pruned_loss=0.01315, audio_tagging_loss=0.008894, over 3034959.12 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:06:39,748 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.599e+01 8.392e+01 8.875e+01 9.417e+01 1.557e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-24 01:06:47,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2616153.3333333335, ans=0.5
2023-11-24 01:07:09,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2616286.6666666665, ans=0.125
2023-11-24 01:07:15,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392450
2023-11-24 01:07:20,526 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:07:22,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2616353.3333333335, ans=0.1
2023-11-24 01:07:33,373 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7700, loss[loss=0.07234, simple_loss=0.08695, pruned_loss=0.0179, audio_tagging_loss=0.01097, over 14410.00 frames. ], tot_loss[loss=0.06638, simple_loss=0.089, pruned_loss=0.01296, audio_tagging_loss=0.008916, over 3036793.44 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:07:39,551 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2616420.0, ans=0.1
2023-11-24 01:07:41,284 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.06 vs. limit=15.0
2023-11-24 01:07:51,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2616486.6666666665, ans=0.1
2023-11-24 01:08:10,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2616620.0, ans=0.125
2023-11-24 01:08:11,850 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.55 vs. limit=12.0
2023-11-24 01:08:17,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392500
2023-11-24 01:08:25,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2616686.6666666665, ans=0.125
2023-11-24 01:08:25,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2616686.6666666665, ans=0.05
2023-11-24 01:08:36,717 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7750, loss[loss=0.06184, simple_loss=0.08774, pruned_loss=0.009718, audio_tagging_loss=0.008248, over 14708.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.09008, pruned_loss=0.01315, audio_tagging_loss=0.00897, over 3037720.47 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:08:36,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2616753.3333333335, ans=0.1
2023-11-24 01:08:45,018 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.197e+01 8.416e+01 8.954e+01 9.884e+01 1.252e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 01:08:51,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2616820.0, ans=0.04949747468305833
2023-11-24 01:09:04,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.09 vs. limit=15.0
2023-11-24 01:09:19,625 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392550
2023-11-24 01:09:36,223 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:09:38,496 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7800, loss[loss=0.04275, simple_loss=0.04988, pruned_loss=0.006989, audio_tagging_loss=0.01082, over 13982.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.08967, pruned_loss=0.01323, audio_tagging_loss=0.009054, over 3031657.74 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:09:44,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.27 vs. limit=22.5
2023-11-24 01:09:47,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2617086.6666666665, ans=0.2
2023-11-24 01:10:10,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.55 vs. limit=15.0
2023-11-24 01:10:22,330 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392600
2023-11-24 01:10:27,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2617353.3333333335, ans=0.0
2023-11-24 01:10:41,568 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7850, loss[loss=0.06361, simple_loss=0.08687, pruned_loss=0.009512, audio_tagging_loss=0.01066, over 16333.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09016, pruned_loss=0.01331, audio_tagging_loss=0.009143, over 3030705.44 frames. ], batch size: 62, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:10:49,997 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.592e+01 8.631e+01 9.206e+01 9.904e+01 1.275e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 01:10:55,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2617486.6666666665, ans=0.125
2023-11-24 01:11:00,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.00 vs. limit=12.0
2023-11-24 01:11:03,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2617486.6666666665, ans=0.125
2023-11-24 01:11:25,094 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392650
2023-11-24 01:11:39,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2617686.6666666665, ans=0.125
2023-11-24 01:11:43,263 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7900, loss[loss=0.08042, simple_loss=0.1145, pruned_loss=0.01735, audio_tagging_loss=0.005811, over 14984.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09093, pruned_loss=0.01366, audio_tagging_loss=0.009145, over 3030772.62 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:11:46,794 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2617753.3333333335, ans=0.125
2023-11-24 01:11:50,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2617753.3333333335, ans=0.0
2023-11-24 01:12:15,604 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=13.29 vs. limit=15.0
2023-11-24 01:12:26,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392700
2023-11-24 01:12:46,555 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 7950, loss[loss=0.04409, simple_loss=0.05023, pruned_loss=0.006429, audio_tagging_loss=0.01255, over 17191.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.08984, pruned_loss=0.01348, audio_tagging_loss=0.009345, over 3033802.03 frames. ], batch size: 67, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:12:47,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2618086.6666666665, ans=0.125
2023-11-24 01:12:52,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2618086.6666666665, ans=0.125
2023-11-24 01:12:54,801 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.531e+01 8.932e+01 9.571e+01 1.197e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 01:12:59,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2618153.3333333335, ans=0.0
2023-11-24 01:13:03,665 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:13:05,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2618153.3333333335, ans=0.125
2023-11-24 01:13:30,107 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392750
2023-11-24 01:13:36,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2618353.3333333335, ans=0.125
2023-11-24 01:13:48,833 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8000, loss[loss=0.0798, simple_loss=0.1085, pruned_loss=0.01581, audio_tagging_loss=0.009744, over 15700.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.08884, pruned_loss=0.01336, audio_tagging_loss=0.009466, over 3032820.20 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:14:03,559 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.38 vs. limit=10.0
2023-11-24 01:14:19,814 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.08 vs. limit=12.0
2023-11-24 01:14:20,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2618553.3333333335, ans=0.1
2023-11-24 01:14:21,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2618553.3333333335, ans=0.125
2023-11-24 01:14:24,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2618553.3333333335, ans=0.125
2023-11-24 01:14:26,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2618620.0, ans=0.2
2023-11-24 01:14:27,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2618620.0, ans=0.0
2023-11-24 01:14:31,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2618620.0, ans=0.0
2023-11-24 01:14:32,143 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392800
2023-11-24 01:14:50,745 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8050, loss[loss=0.08625, simple_loss=0.1207, pruned_loss=0.01585, audio_tagging_loss=0.01007, over 15254.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09001, pruned_loss=0.01338, audio_tagging_loss=0.009373, over 3041001.67 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:15:01,891 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.420e+01 8.885e+01 9.707e+01 1.213e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-24 01:15:06,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2618820.0, ans=0.125
2023-11-24 01:15:08,577 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.99 vs. limit=15.0
2023-11-24 01:15:31,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2618953.3333333335, ans=0.0
2023-11-24 01:15:34,920 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392850
2023-11-24 01:15:38,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2618953.3333333335, ans=0.95
2023-11-24 01:15:53,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2619086.6666666665, ans=0.125
2023-11-24 01:15:54,262 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8100, loss[loss=0.05417, simple_loss=0.07163, pruned_loss=0.009066, audio_tagging_loss=0.009284, over 15030.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09019, pruned_loss=0.01339, audio_tagging_loss=0.009268, over 3035202.54 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:16:01,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2619086.6666666665, ans=0.2
2023-11-24 01:16:14,481 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.43 vs. limit=10.0
2023-11-24 01:16:22,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2619220.0, ans=0.95
2023-11-24 01:16:25,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2619220.0, ans=0.125
2023-11-24 01:16:38,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392900
2023-11-24 01:16:52,047 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.34 vs. limit=15.0
2023-11-24 01:16:56,159 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8150, loss[loss=0.07053, simple_loss=0.09241, pruned_loss=0.01535, audio_tagging_loss=0.008979, over 16130.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09164, pruned_loss=0.01351, audio_tagging_loss=0.009024, over 3043418.77 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:17:00,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2619420.0, ans=0.0
2023-11-24 01:17:04,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2619420.0, ans=0.1
2023-11-24 01:17:06,598 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.428e+01 9.309e+01 1.006e+02 1.505e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-24 01:17:37,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2619620.0, ans=0.125
2023-11-24 01:17:39,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2619620.0, ans=0.125
2023-11-24 01:17:40,906 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 392950
2023-11-24 01:17:47,274 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.21 vs. limit=22.5
2023-11-24 01:17:48,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2619686.6666666665, ans=0.125
2023-11-24 01:17:59,138 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8200, loss[loss=0.0768, simple_loss=0.1028, pruned_loss=0.01513, audio_tagging_loss=0.01028, over 15054.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09186, pruned_loss=0.01344, audio_tagging_loss=0.008901, over 3046005.38 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:18:01,540 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:18:06,081 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.71 vs. limit=8.0
2023-11-24 01:18:12,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2619820.0, ans=0.125
2023-11-24 01:18:19,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2619820.0, ans=0.0
2023-11-24 01:18:31,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2619886.6666666665, ans=0.125
2023-11-24 01:18:38,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2619953.3333333335, ans=0.125
2023-11-24 01:18:43,383 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393000
2023-11-24 01:18:47,663 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.65 vs. limit=15.0
2023-11-24 01:18:57,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2620020.0, ans=0.0
2023-11-24 01:19:00,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2620020.0, ans=0.015
2023-11-24 01:19:02,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2620086.6666666665, ans=0.0
2023-11-24 01:19:03,403 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8250, loss[loss=0.06733, simple_loss=0.07831, pruned_loss=0.01637, audio_tagging_loss=0.01181, over 14947.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09253, pruned_loss=0.01352, audio_tagging_loss=0.00883, over 3047056.23 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:19:06,273 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.96 vs. limit=15.0
2023-11-24 01:19:13,054 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.535e+01 8.227e+01 9.050e+01 1.002e+02 1.257e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 01:19:33,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2620220.0, ans=0.125
2023-11-24 01:19:35,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2620220.0, ans=0.125
2023-11-24 01:19:42,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.69 vs. limit=10.0
2023-11-24 01:19:47,629 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393050
2023-11-24 01:19:48,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2620286.6666666665, ans=0.0
2023-11-24 01:19:56,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2620353.3333333335, ans=0.1
2023-11-24 01:20:05,500 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8300, loss[loss=0.07284, simple_loss=0.0997, pruned_loss=0.01696, audio_tagging_loss=0.006033, over 15032.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09255, pruned_loss=0.01363, audio_tagging_loss=0.008866, over 3047612.46 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:20:09,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2620420.0, ans=0.0
2023-11-24 01:20:12,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff3.min_abs, batch_count=2620420.0, ans=0.2
2023-11-24 01:20:30,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2620553.3333333335, ans=0.125
2023-11-24 01:20:38,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2620553.3333333335, ans=0.125
2023-11-24 01:20:49,958 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393100
2023-11-24 01:20:50,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=2620620.0, ans=10.0
2023-11-24 01:20:57,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2620686.6666666665, ans=0.0
2023-11-24 01:20:58,488 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:21:07,466 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8350, loss[loss=0.06637, simple_loss=0.091, pruned_loss=0.01381, audio_tagging_loss=0.007058, over 16615.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09282, pruned_loss=0.01357, audio_tagging_loss=0.008858, over 3051204.09 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:21:07,974 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.79 vs. limit=6.0
2023-11-24 01:21:19,990 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.856e+01 8.568e+01 9.324e+01 1.020e+02 1.487e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 01:21:45,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2620953.3333333335, ans=0.0
2023-11-24 01:21:50,344 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0
2023-11-24 01:21:51,033 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393150
2023-11-24 01:22:08,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2621020.0, ans=0.125
2023-11-24 01:22:11,137 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8400, loss[loss=0.07598, simple_loss=0.1057, pruned_loss=0.0152, audio_tagging_loss=0.007928, over 14537.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09318, pruned_loss=0.01361, audio_tagging_loss=0.008754, over 3045341.85 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:22:18,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2621086.6666666665, ans=0.1
2023-11-24 01:22:21,094 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.29 vs. limit=15.0
2023-11-24 01:22:54,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393200
2023-11-24 01:23:12,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2621420.0, ans=0.125
2023-11-24 01:23:12,778 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8450, loss[loss=0.05707, simple_loss=0.07985, pruned_loss=0.008453, audio_tagging_loss=0.008692, over 16391.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09229, pruned_loss=0.01354, audio_tagging_loss=0.008845, over 3050051.42 frames. ], batch size: 60, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:23:23,344 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.428e+01 8.989e+01 9.532e+01 1.176e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 01:23:56,087 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393250
2023-11-24 01:24:03,509 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:24:03,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2621686.6666666665, ans=0.0
2023-11-24 01:24:09,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2621686.6666666665, ans=0.125
2023-11-24 01:24:13,902 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8500, loss[loss=0.07249, simple_loss=0.1009, pruned_loss=0.01557, audio_tagging_loss=0.00646, over 15039.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.0922, pruned_loss=0.01365, audio_tagging_loss=0.008893, over 3048630.87 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:24:23,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2621753.3333333335, ans=0.125
2023-11-24 01:24:26,922 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.26 vs. limit=15.0
2023-11-24 01:24:34,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2621820.0, ans=0.0
2023-11-24 01:24:39,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2621886.6666666665, ans=0.2
2023-11-24 01:24:49,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2621886.6666666665, ans=0.2
2023-11-24 01:24:57,410 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393300
2023-11-24 01:25:08,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2622020.0, ans=0.2
2023-11-24 01:25:13,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2622020.0, ans=0.0
2023-11-24 01:25:17,341 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8550, loss[loss=0.06676, simple_loss=0.08987, pruned_loss=0.01196, audio_tagging_loss=0.009866, over 15580.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.0923, pruned_loss=0.0136, audio_tagging_loss=0.009035, over 3048669.44 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:25:23,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2622086.6666666665, ans=0.0
2023-11-24 01:25:26,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2622086.6666666665, ans=0.125
2023-11-24 01:25:27,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2622086.6666666665, ans=0.125
2023-11-24 01:25:29,174 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.943e+01 8.811e+01 9.309e+01 9.853e+01 1.247e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-24 01:25:41,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2622220.0, ans=0.1
2023-11-24 01:25:45,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2622220.0, ans=0.1
2023-11-24 01:25:57,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2622286.6666666665, ans=0.0
2023-11-24 01:25:59,934 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393350
2023-11-24 01:26:10,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2622353.3333333335, ans=0.125
2023-11-24 01:26:17,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2622420.0, ans=0.2
2023-11-24 01:26:18,354 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8600, loss[loss=0.08253, simple_loss=0.1126, pruned_loss=0.01805, audio_tagging_loss=0.008179, over 15692.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09231, pruned_loss=0.01368, audio_tagging_loss=0.009148, over 3044577.62 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:26:35,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2622486.6666666665, ans=0.1
2023-11-24 01:26:35,358 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.37 vs. limit=15.0
2023-11-24 01:26:37,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2622486.6666666665, ans=0.05
2023-11-24 01:26:46,170 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.71 vs. limit=15.0
2023-11-24 01:26:57,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2622620.0, ans=0.0
2023-11-24 01:26:57,473 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.95 vs. limit=15.0
2023-11-24 01:26:58,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.42 vs. limit=15.0
2023-11-24 01:27:01,481 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393400
2023-11-24 01:27:06,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2622686.6666666665, ans=0.125
2023-11-24 01:27:15,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2622686.6666666665, ans=0.0
2023-11-24 01:27:17,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2622686.6666666665, ans=0.125
2023-11-24 01:27:19,489 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8650, loss[loss=0.04786, simple_loss=0.06812, pruned_loss=0.00733, audio_tagging_loss=0.006474, over 15296.00 frames. ], tot_loss[loss=0.06886, simple_loss=0.0922, pruned_loss=0.01365, audio_tagging_loss=0.009108, over 3044370.73 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:27:24,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2622753.3333333335, ans=0.0
2023-11-24 01:27:27,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2622753.3333333335, ans=0.0
2023-11-24 01:27:32,817 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.293e+01 8.465e+01 9.032e+01 9.917e+01 1.291e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 01:27:53,894 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.61 vs. limit=10.0
2023-11-24 01:27:54,814 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.82 vs. limit=15.0
2023-11-24 01:28:03,772 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393450
2023-11-24 01:28:11,631 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.75 vs. limit=15.0
2023-11-24 01:28:22,594 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8700, loss[loss=0.0807, simple_loss=0.1102, pruned_loss=0.01794, audio_tagging_loss=0.007653, over 15636.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.09319, pruned_loss=0.01382, audio_tagging_loss=0.009129, over 3043400.25 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:28:36,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2623153.3333333335, ans=0.125
2023-11-24 01:28:44,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2623153.3333333335, ans=0.1
2023-11-24 01:28:58,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2623286.6666666665, ans=0.2
2023-11-24 01:29:05,288 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393500
2023-11-24 01:29:13,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2623353.3333333335, ans=0.0
2023-11-24 01:29:21,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2623353.3333333335, ans=0.125
2023-11-24 01:29:24,782 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8750, loss[loss=0.06213, simple_loss=0.08068, pruned_loss=0.01033, audio_tagging_loss=0.01146, over 14437.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.093, pruned_loss=0.01371, audio_tagging_loss=0.009157, over 3048958.66 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:29:30,315 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.31 vs. limit=22.5
2023-11-24 01:29:33,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2623420.0, ans=0.125
2023-11-24 01:29:36,660 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.231e+01 8.417e+01 9.143e+01 1.010e+02 1.352e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 01:29:40,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2623486.6666666665, ans=0.1
2023-11-24 01:29:46,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2623486.6666666665, ans=0.125
2023-11-24 01:29:46,362 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:29:55,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2623553.3333333335, ans=0.0
2023-11-24 01:29:57,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2623553.3333333335, ans=0.0
2023-11-24 01:30:01,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2623620.0, ans=0.125
2023-11-24 01:30:06,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2623620.0, ans=0.0
2023-11-24 01:30:07,473 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393550
2023-11-24 01:30:11,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2623620.0, ans=0.0
2023-11-24 01:30:21,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2623686.6666666665, ans=0.125
2023-11-24 01:30:25,625 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8800, loss[loss=0.06002, simple_loss=0.0708, pruned_loss=0.01454, audio_tagging_loss=0.01008, over 14755.00 frames. ], tot_loss[loss=0.06975, simple_loss=0.09352, pruned_loss=0.0138, audio_tagging_loss=0.009194, over 3045008.36 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:30:49,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2623886.6666666665, ans=0.125
2023-11-24 01:30:59,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2623886.6666666665, ans=0.0
2023-11-24 01:31:04,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2623953.3333333335, ans=0.2
2023-11-24 01:31:08,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393600
2023-11-24 01:31:16,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2624020.0, ans=0.0
2023-11-24 01:31:22,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2624020.0, ans=0.1
2023-11-24 01:31:27,898 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8850, loss[loss=0.07886, simple_loss=0.09869, pruned_loss=0.02039, audio_tagging_loss=0.009118, over 15385.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09233, pruned_loss=0.01358, audio_tagging_loss=0.009327, over 3052972.66 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:31:39,630 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.345e+01 8.328e+01 9.022e+01 9.725e+01 1.238e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-24 01:31:40,854 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:31:46,910 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.75 vs. limit=15.0
2023-11-24 01:31:57,178 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.93 vs. limit=15.0
2023-11-24 01:32:02,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2624286.6666666665, ans=0.0
2023-11-24 01:32:09,898 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393650
2023-11-24 01:32:19,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2624353.3333333335, ans=0.0
2023-11-24 01:32:28,680 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8900, loss[loss=0.08218, simple_loss=0.1139, pruned_loss=0.01619, audio_tagging_loss=0.009057, over 15743.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09173, pruned_loss=0.01352, audio_tagging_loss=0.009117, over 3049228.60 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:32:35,322 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:32:45,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2624486.6666666665, ans=0.125
2023-11-24 01:33:12,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393700
2023-11-24 01:33:30,363 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 8950, loss[loss=0.06322, simple_loss=0.08837, pruned_loss=0.01286, audio_tagging_loss=0.006177, over 15847.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09274, pruned_loss=0.01366, audio_tagging_loss=0.00892, over 3050277.49 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:33:37,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2624753.3333333335, ans=0.0
2023-11-24 01:33:42,259 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.467e+01 9.115e+01 9.992e+01 1.363e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 01:33:42,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2624820.0, ans=0.125
2023-11-24 01:33:49,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2624820.0, ans=0.125
2023-11-24 01:33:59,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2624886.6666666665, ans=0.0
2023-11-24 01:34:13,844 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393750
2023-11-24 01:34:27,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2625020.0, ans=0.0
2023-11-24 01:34:32,143 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9000, loss[loss=0.0534, simple_loss=0.07066, pruned_loss=0.009735, audio_tagging_loss=0.008335, over 15598.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09198, pruned_loss=0.01358, audio_tagging_loss=0.008849, over 3052320.26 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:34:32,146 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 01:35:10,897 INFO [train_asr.py:1253] (0/4) Epoch 33, validation: loss=0.05892, simple_loss=0.05094, pruned_loss=0.005119, audio_tagging_loss=0.02833, over 4681554.00 frames. 
2023-11-24 01:35:10,898 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 01:35:16,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2625086.6666666665, ans=0.0
2023-11-24 01:35:18,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2625086.6666666665, ans=0.125
2023-11-24 01:35:33,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2625153.3333333335, ans=0.125
2023-11-24 01:35:34,528 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.00 vs. limit=10.0
2023-11-24 01:35:50,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2625286.6666666665, ans=0.2
2023-11-24 01:35:53,955 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393800
2023-11-24 01:36:12,541 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9050, loss[loss=0.06823, simple_loss=0.1027, pruned_loss=0.01048, audio_tagging_loss=0.00639, over 15495.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09166, pruned_loss=0.01362, audio_tagging_loss=0.008855, over 3051933.16 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:36:21,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2625420.0, ans=0.125
2023-11-24 01:36:25,595 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.819e+01 9.377e+01 1.005e+02 1.265e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 01:36:33,291 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.61 vs. limit=15.0
2023-11-24 01:36:55,937 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393850
2023-11-24 01:36:58,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2625620.0, ans=0.2
2023-11-24 01:37:04,282 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:37:14,627 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9100, loss[loss=0.06298, simple_loss=0.08636, pruned_loss=0.01314, audio_tagging_loss=0.006659, over 15895.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09214, pruned_loss=0.01363, audio_tagging_loss=0.00883, over 3060646.69 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:37:24,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2625753.3333333335, ans=0.1
2023-11-24 01:37:27,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2625820.0, ans=0.125
2023-11-24 01:37:57,114 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393900
2023-11-24 01:38:15,241 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9150, loss[loss=0.07144, simple_loss=0.1086, pruned_loss=0.01045, audio_tagging_loss=0.006674, over 15201.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09211, pruned_loss=0.01361, audio_tagging_loss=0.008788, over 3060525.03 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:38:27,505 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 7.991e+01 8.782e+01 9.725e+01 1.593e+02, threshold=1.756e+02, percent-clipped=0.0
2023-11-24 01:38:34,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.53 vs. limit=12.0
2023-11-24 01:38:58,281 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 393950
2023-11-24 01:39:12,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2626353.3333333335, ans=0.125
2023-11-24 01:39:13,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2626353.3333333335, ans=0.0
2023-11-24 01:39:13,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2626353.3333333335, ans=0.0
2023-11-24 01:39:15,962 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0
2023-11-24 01:39:16,555 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9200, loss[loss=0.05187, simple_loss=0.07003, pruned_loss=0.008527, audio_tagging_loss=0.00833, over 15793.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09176, pruned_loss=0.01351, audio_tagging_loss=0.008747, over 3059985.79 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 01:39:58,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394000
2023-11-24 01:40:03,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.41 vs. limit=10.0
2023-11-24 01:40:03,831 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.28 vs. limit=10.0
2023-11-24 01:40:05,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2626686.6666666665, ans=0.125
2023-11-24 01:40:18,499 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9250, loss[loss=0.05535, simple_loss=0.07956, pruned_loss=0.006502, audio_tagging_loss=0.009069, over 14764.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09216, pruned_loss=0.01366, audio_tagging_loss=0.008778, over 3062505.86 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:40:27,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2626753.3333333335, ans=0.125
2023-11-24 01:40:31,441 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.040e+01 8.331e+01 9.082e+01 9.914e+01 1.295e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 01:40:43,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2626886.6666666665, ans=0.125
2023-11-24 01:41:00,554 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394050
2023-11-24 01:41:00,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2626953.3333333335, ans=0.125
2023-11-24 01:41:09,296 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.48 vs. limit=22.5
2023-11-24 01:41:19,487 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9300, loss[loss=0.05671, simple_loss=0.07502, pruned_loss=0.009188, audio_tagging_loss=0.01001, over 14845.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09188, pruned_loss=0.0137, audio_tagging_loss=0.008835, over 3056914.88 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:41:21,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2627086.6666666665, ans=0.125
2023-11-24 01:41:37,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2627153.3333333335, ans=0.125
2023-11-24 01:41:44,964 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.31 vs. limit=15.0
2023-11-24 01:42:03,203 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394100
2023-11-24 01:42:21,235 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9350, loss[loss=0.06635, simple_loss=0.08722, pruned_loss=0.01433, audio_tagging_loss=0.008412, over 14535.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09159, pruned_loss=0.01378, audio_tagging_loss=0.008859, over 3053990.59 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:42:35,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2627486.6666666665, ans=0.0
2023-11-24 01:42:36,053 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.369e+01 8.431e+01 9.003e+01 9.569e+01 1.110e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 01:42:38,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2627486.6666666665, ans=0.125
2023-11-24 01:43:04,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394150
2023-11-24 01:43:17,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2627686.6666666665, ans=0.2
2023-11-24 01:43:22,668 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9400, loss[loss=0.07925, simple_loss=0.1117, pruned_loss=0.01453, audio_tagging_loss=0.008893, over 14967.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09133, pruned_loss=0.01366, audio_tagging_loss=0.009016, over 3050776.89 frames. ], batch size: 54, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:43:32,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2627753.3333333335, ans=0.2
2023-11-24 01:43:44,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2627820.0, ans=0.125
2023-11-24 01:43:49,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2627886.6666666665, ans=0.0
2023-11-24 01:43:49,518 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.93 vs. limit=15.0
2023-11-24 01:44:00,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.04 vs. limit=15.0
2023-11-24 01:44:06,273 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394200
2023-11-24 01:44:08,321 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.52 vs. limit=22.5
2023-11-24 01:44:25,390 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9450, loss[loss=0.0817, simple_loss=0.1106, pruned_loss=0.01868, audio_tagging_loss=0.007703, over 15544.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09188, pruned_loss=0.01376, audio_tagging_loss=0.009111, over 3048512.41 frames. ], batch size: 59, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:44:25,411 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:44:25,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2628086.6666666665, ans=0.125
2023-11-24 01:44:39,837 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.278e+01 8.363e+01 8.998e+01 9.818e+01 1.304e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 01:44:42,746 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.99 vs. limit=22.5
2023-11-24 01:45:09,166 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394250
2023-11-24 01:45:18,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2628353.3333333335, ans=0.0
2023-11-24 01:45:26,608 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9500, loss[loss=0.06718, simple_loss=0.08445, pruned_loss=0.0136, audio_tagging_loss=0.01136, over 16641.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09154, pruned_loss=0.01366, audio_tagging_loss=0.009219, over 3046018.20 frames. ], batch size: 62, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:45:32,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2628420.0, ans=0.07
2023-11-24 01:45:43,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2628486.6666666665, ans=0.0
2023-11-24 01:45:45,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2628486.6666666665, ans=0.1
2023-11-24 01:45:52,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2628553.3333333335, ans=0.2
2023-11-24 01:45:57,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2628553.3333333335, ans=0.125
2023-11-24 01:46:06,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2628620.0, ans=0.125
2023-11-24 01:46:09,893 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394300
2023-11-24 01:46:13,037 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.55 vs. limit=15.0
2023-11-24 01:46:13,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2628620.0, ans=0.1
2023-11-24 01:46:27,672 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9550, loss[loss=0.07491, simple_loss=0.09909, pruned_loss=0.0151, audio_tagging_loss=0.01026, over 16324.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09201, pruned_loss=0.01376, audio_tagging_loss=0.009231, over 3046388.16 frames. ], batch size: 61, lr: 2.05e-03, grad_scale: 8.0
2023-11-24 01:46:37,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2628753.3333333335, ans=0.0
2023-11-24 01:46:38,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2628753.3333333335, ans=0.1
2023-11-24 01:46:42,914 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.969e+01 8.519e+01 9.047e+01 9.704e+01 1.211e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 01:46:53,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.49 vs. limit=12.0
2023-11-24 01:47:00,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2628886.6666666665, ans=0.125
2023-11-24 01:47:10,300 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394350
2023-11-24 01:47:29,291 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9600, loss[loss=0.07855, simple_loss=0.1087, pruned_loss=0.01562, audio_tagging_loss=0.008584, over 15269.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09229, pruned_loss=0.01375, audio_tagging_loss=0.009322, over 3049716.56 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:47:29,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2629086.6666666665, ans=0.2
2023-11-24 01:47:37,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.43 vs. limit=22.5
2023-11-24 01:47:38,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2629086.6666666665, ans=0.125
2023-11-24 01:47:39,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2629086.6666666665, ans=0.125
2023-11-24 01:47:42,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2629153.3333333335, ans=0.125
2023-11-24 01:47:54,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2629220.0, ans=0.125
2023-11-24 01:47:55,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2629220.0, ans=0.0
2023-11-24 01:48:01,713 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.49 vs. limit=22.5
2023-11-24 01:48:13,548 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394400
2023-11-24 01:48:21,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2629353.3333333335, ans=0.1
2023-11-24 01:48:26,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2629353.3333333335, ans=0.125
2023-11-24 01:48:31,242 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9650, loss[loss=0.06272, simple_loss=0.08332, pruned_loss=0.01388, audio_tagging_loss=0.00719, over 15392.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.0922, pruned_loss=0.01373, audio_tagging_loss=0.009258, over 3048338.17 frames. ], batch size: 58, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:48:45,258 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.168e+01 8.217e+01 8.903e+01 9.505e+01 1.138e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-24 01:48:49,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2629486.6666666665, ans=0.2
2023-11-24 01:49:11,509 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.00 vs. limit=15.0
2023-11-24 01:49:14,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394450
2023-11-24 01:49:15,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2629620.0, ans=0.125
2023-11-24 01:49:17,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2629620.0, ans=0.125
2023-11-24 01:49:18,522 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.55 vs. limit=15.0
2023-11-24 01:49:24,793 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.74 vs. limit=15.0
2023-11-24 01:49:32,175 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9700, loss[loss=0.06556, simple_loss=0.08386, pruned_loss=0.01344, audio_tagging_loss=0.01019, over 15128.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09174, pruned_loss=0.01368, audio_tagging_loss=0.009106, over 3044237.25 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:50:04,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2629886.6666666665, ans=0.125
2023-11-24 01:50:15,344 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394500
2023-11-24 01:50:34,843 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9750, loss[loss=0.06266, simple_loss=0.08083, pruned_loss=0.009591, audio_tagging_loss=0.01265, over 14746.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09139, pruned_loss=0.01357, audio_tagging_loss=0.009096, over 3034144.46 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:50:49,139 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.192e+01 8.620e+01 9.194e+01 9.972e+01 1.344e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 01:50:50,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2630153.3333333335, ans=0.125
2023-11-24 01:51:03,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2630220.0, ans=0.125
2023-11-24 01:51:05,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.23 vs. limit=10.0
2023-11-24 01:51:17,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394550
2023-11-24 01:51:36,070 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9800, loss[loss=0.07901, simple_loss=0.1116, pruned_loss=0.0143, audio_tagging_loss=0.008897, over 13468.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09191, pruned_loss=0.01367, audio_tagging_loss=0.008971, over 3035617.65 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:51:51,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2630486.6666666665, ans=0.125
2023-11-24 01:51:55,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2630486.6666666665, ans=0.125
2023-11-24 01:52:17,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2630620.0, ans=0.125
2023-11-24 01:52:19,683 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394600
2023-11-24 01:52:33,014 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:52:37,801 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9850, loss[loss=0.03588, simple_loss=0.05179, pruned_loss=0.002718, audio_tagging_loss=0.007261, over 14370.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09237, pruned_loss=0.01379, audio_tagging_loss=0.008825, over 3036107.42 frames. ], batch size: 55, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:52:50,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.99 vs. limit=22.5
2023-11-24 01:52:53,142 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.063e+01 8.642e+01 9.395e+01 1.002e+02 1.404e+02, threshold=1.879e+02, percent-clipped=0.0
2023-11-24 01:52:53,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2630820.0, ans=0.0
2023-11-24 01:53:13,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2630886.6666666665, ans=0.1
2023-11-24 01:53:21,240 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394650
2023-11-24 01:53:24,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2630953.3333333335, ans=0.0
2023-11-24 01:53:40,119 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9900, loss[loss=0.09296, simple_loss=0.133, pruned_loss=0.0206, audio_tagging_loss=0.00586, over 14953.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.0924, pruned_loss=0.01374, audio_tagging_loss=0.008808, over 3037695.90 frames. ], batch size: 53, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:53:40,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2631086.6666666665, ans=0.1
2023-11-24 01:54:23,709 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394700
2023-11-24 01:54:38,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2631353.3333333335, ans=0.125
2023-11-24 01:54:42,052 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 9950, loss[loss=0.06073, simple_loss=0.0791, pruned_loss=0.01087, audio_tagging_loss=0.01031, over 15364.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09277, pruned_loss=0.01387, audio_tagging_loss=0.008738, over 3042274.59 frames. ], batch size: 57, lr: 2.05e-03, grad_scale: 16.0
2023-11-24 01:54:45,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2631420.0, ans=0.125
2023-11-24 01:54:54,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2631486.6666666665, ans=0.0
2023-11-24 01:54:56,124 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.637e+01 9.131e+01 9.854e+01 1.265e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 01:55:02,686 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.87 vs. limit=12.0
2023-11-24 01:55:12,134 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.42 vs. limit=10.0
2023-11-24 01:55:15,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2631553.3333333335, ans=0.0
2023-11-24 01:55:16,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2631553.3333333335, ans=0.125
2023-11-24 01:55:18,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=2631620.0, ans=22.5
2023-11-24 01:55:24,965 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394750
2023-11-24 01:55:26,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2631620.0, ans=0.125
2023-11-24 01:55:42,558 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10000, loss[loss=0.06788, simple_loss=0.09462, pruned_loss=0.011, audio_tagging_loss=0.009577, over 15267.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09246, pruned_loss=0.01366, audio_tagging_loss=0.008649, over 3042014.92 frames. ], batch size: 56, lr: 2.05e-03, grad_scale: 32.0
2023-11-24 01:55:42,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2631753.3333333335, ans=0.125
2023-11-24 01:56:05,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2631820.0, ans=0.2
2023-11-24 01:56:25,163 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.60 vs. limit=15.0
2023-11-24 01:56:25,648 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394800
2023-11-24 01:56:45,105 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10050, loss[loss=0.07548, simple_loss=0.1063, pruned_loss=0.01352, audio_tagging_loss=0.008831, over 15569.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09193, pruned_loss=0.01367, audio_tagging_loss=0.008692, over 3043102.50 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 01:56:52,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2632086.6666666665, ans=0.5
2023-11-24 01:56:58,942 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.73 vs. limit=12.0
2023-11-24 01:56:59,416 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.081e+01 8.389e+01 9.133e+01 9.609e+01 1.226e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 01:57:03,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2632153.3333333335, ans=10.0
2023-11-24 01:57:10,244 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.20 vs. limit=6.0
2023-11-24 01:57:26,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2632286.6666666665, ans=0.125
2023-11-24 01:57:27,590 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394850
2023-11-24 01:57:46,295 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10100, loss[loss=0.06045, simple_loss=0.07414, pruned_loss=0.01054, audio_tagging_loss=0.01284, over 14713.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09226, pruned_loss=0.01375, audio_tagging_loss=0.008751, over 3048175.14 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 01:57:59,286 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.54 vs. limit=12.0
2023-11-24 01:58:03,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2632486.6666666665, ans=0.0
2023-11-24 01:58:29,549 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394900
2023-11-24 01:58:36,510 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:58:47,637 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10150, loss[loss=0.06212, simple_loss=0.08216, pruned_loss=0.01086, audio_tagging_loss=0.01018, over 15851.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09214, pruned_loss=0.01376, audio_tagging_loss=0.008927, over 3045438.95 frames. ], batch size: 61, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 01:59:02,461 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.176e+01 8.517e+01 9.184e+01 1.003e+02 1.404e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-24 01:59:17,280 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 01:59:18,663 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 01:59:19,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2632886.6666666665, ans=0.1
2023-11-24 01:59:25,127 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0
2023-11-24 01:59:30,951 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 394950
2023-11-24 01:59:49,435 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10200, loss[loss=0.07499, simple_loss=0.09427, pruned_loss=0.01816, audio_tagging_loss=0.009686, over 14710.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09244, pruned_loss=0.01381, audio_tagging_loss=0.008956, over 3048011.58 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:00:02,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2633153.3333333335, ans=0.125
2023-11-24 02:00:08,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2633153.3333333335, ans=0.1
2023-11-24 02:00:10,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2633153.3333333335, ans=0.0
2023-11-24 02:00:14,009 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 02:00:32,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395000
2023-11-24 02:00:49,369 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.24 vs. limit=6.0
2023-11-24 02:00:52,215 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10250, loss[loss=0.05816, simple_loss=0.07835, pruned_loss=0.009633, audio_tagging_loss=0.009356, over 15017.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09328, pruned_loss=0.01395, audio_tagging_loss=0.008961, over 3050046.28 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:01:07,146 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.621e+01 8.620e+01 9.148e+01 9.659e+01 1.254e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 02:01:19,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2633553.3333333335, ans=0.0
2023-11-24 02:01:36,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395050
2023-11-24 02:01:46,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2633686.6666666665, ans=0.125
2023-11-24 02:01:54,959 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10300, loss[loss=0.07515, simple_loss=0.08891, pruned_loss=0.01643, audio_tagging_loss=0.01427, over 14110.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09321, pruned_loss=0.01378, audio_tagging_loss=0.009006, over 3053322.93 frames. ], batch size: 54, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:01:55,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2633753.3333333335, ans=0.0
2023-11-24 02:02:27,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2633886.6666666665, ans=0.125
2023-11-24 02:02:31,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2633953.3333333335, ans=0.0
2023-11-24 02:02:32,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2633953.3333333335, ans=0.125
2023-11-24 02:02:37,634 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.12 vs. limit=10.0
2023-11-24 02:02:38,297 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395100
2023-11-24 02:02:56,677 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10350, loss[loss=0.07318, simple_loss=0.08277, pruned_loss=0.0208, audio_tagging_loss=0.01101, over 15301.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.0927, pruned_loss=0.01374, audio_tagging_loss=0.009111, over 3052197.02 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:02:59,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2634086.6666666665, ans=0.125
2023-11-24 02:03:12,160 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.449e+01 8.937e+01 9.719e+01 1.290e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-24 02:03:39,993 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395150
2023-11-24 02:03:58,868 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10400, loss[loss=0.06691, simple_loss=0.09365, pruned_loss=0.009807, audio_tagging_loss=0.01028, over 15407.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09233, pruned_loss=0.01353, audio_tagging_loss=0.009233, over 3053954.15 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:04:15,639 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:04:31,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2634553.3333333335, ans=0.125
2023-11-24 02:04:41,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395200
2023-11-24 02:04:57,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2634686.6666666665, ans=0.1
2023-11-24 02:04:59,979 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10450, loss[loss=0.07221, simple_loss=0.1014, pruned_loss=0.01389, audio_tagging_loss=0.007599, over 15741.00 frames. ], tot_loss[loss=0.06906, simple_loss=0.09259, pruned_loss=0.01363, audio_tagging_loss=0.009141, over 3051715.14 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:05:14,724 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.359e+01 8.851e+01 9.617e+01 1.233e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-24 02:05:25,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2634886.6666666665, ans=0.0
2023-11-24 02:05:43,216 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395250
2023-11-24 02:05:49,301 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:05:49,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2635020.0, ans=0.0
2023-11-24 02:05:50,799 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.84 vs. limit=15.0
2023-11-24 02:06:01,542 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10500, loss[loss=0.07265, simple_loss=0.1007, pruned_loss=0.01354, audio_tagging_loss=0.008761, over 15777.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09189, pruned_loss=0.01357, audio_tagging_loss=0.00906, over 3046467.05 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:06:03,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2635086.6666666665, ans=0.0
2023-11-24 02:06:03,387 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.09 vs. limit=12.0
2023-11-24 02:06:09,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2635086.6666666665, ans=0.125
2023-11-24 02:06:11,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2635086.6666666665, ans=0.04949747468305833
2023-11-24 02:06:11,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2635086.6666666665, ans=0.04949747468305833
2023-11-24 02:06:18,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2635153.3333333335, ans=0.2
2023-11-24 02:06:22,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2635153.3333333335, ans=0.1
2023-11-24 02:06:25,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2635220.0, ans=0.0
2023-11-24 02:06:29,734 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.17 vs. limit=15.0
2023-11-24 02:06:31,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2635220.0, ans=0.0
2023-11-24 02:06:41,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2635286.6666666665, ans=0.125
2023-11-24 02:06:44,669 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395300
2023-11-24 02:07:04,408 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10550, loss[loss=0.07409, simple_loss=0.1067, pruned_loss=0.01384, audio_tagging_loss=0.00693, over 15783.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09261, pruned_loss=0.01359, audio_tagging_loss=0.008878, over 3040776.59 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:07:05,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2635420.0, ans=0.0
2023-11-24 02:07:14,995 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.75 vs. limit=15.0
2023-11-24 02:07:19,135 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.993e+01 8.732e+01 9.283e+01 1.036e+02 1.632e+02, threshold=1.857e+02, percent-clipped=0.0
2023-11-24 02:07:21,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2635486.6666666665, ans=0.125
2023-11-24 02:07:30,552 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.35 vs. limit=12.0
2023-11-24 02:07:32,536 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2635553.3333333335, ans=0.125
2023-11-24 02:07:48,238 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395350
2023-11-24 02:08:05,716 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10600, loss[loss=0.07729, simple_loss=0.09301, pruned_loss=0.02104, audio_tagging_loss=0.009747, over 16281.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09237, pruned_loss=0.01353, audio_tagging_loss=0.008874, over 3046287.81 frames. ], batch size: 61, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:08:13,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2635753.3333333335, ans=0.125
2023-11-24 02:08:48,918 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395400
2023-11-24 02:08:50,570 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:09:07,231 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10650, loss[loss=0.05044, simple_loss=0.05981, pruned_loss=0.007201, audio_tagging_loss=0.01334, over 14449.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09291, pruned_loss=0.01358, audio_tagging_loss=0.008907, over 3044039.57 frames. ], batch size: 54, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:09:19,870 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.51 vs. limit=22.5
2023-11-24 02:09:23,868 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.571e+01 8.426e+01 9.214e+01 9.689e+01 1.168e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 02:09:27,112 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.58 vs. limit=15.0
2023-11-24 02:09:30,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2636153.3333333335, ans=0.1
2023-11-24 02:09:51,017 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395450
2023-11-24 02:09:54,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2636286.6666666665, ans=0.2
2023-11-24 02:09:57,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2636353.3333333335, ans=0.1
2023-11-24 02:10:10,661 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10700, loss[loss=0.06775, simple_loss=0.08622, pruned_loss=0.01486, audio_tagging_loss=0.009782, over 14467.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.09265, pruned_loss=0.01354, audio_tagging_loss=0.008927, over 3037715.51 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:10:12,035 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:10:30,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2636486.6666666665, ans=0.1
2023-11-24 02:10:44,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2636553.3333333335, ans=0.125
2023-11-24 02:10:44,654 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.28 vs. limit=12.0
2023-11-24 02:10:53,376 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395500
2023-11-24 02:10:58,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2636686.6666666665, ans=0.0
2023-11-24 02:11:00,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2636686.6666666665, ans=0.125
2023-11-24 02:11:04,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2636686.6666666665, ans=0.125
2023-11-24 02:11:11,241 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10750, loss[loss=0.06728, simple_loss=0.09605, pruned_loss=0.01132, audio_tagging_loss=0.007935, over 14122.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09321, pruned_loss=0.01373, audio_tagging_loss=0.008829, over 3038306.36 frames. ], batch size: 53, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:11:12,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2636753.3333333335, ans=0.125
2023-11-24 02:11:26,615 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.296e+01 8.958e+01 9.992e+01 1.267e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-24 02:11:54,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2636953.3333333335, ans=0.0
2023-11-24 02:11:55,180 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395550
2023-11-24 02:11:58,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2636953.3333333335, ans=0.125
2023-11-24 02:12:11,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2637086.6666666665, ans=0.0
2023-11-24 02:12:12,806 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10800, loss[loss=0.08451, simple_loss=0.1248, pruned_loss=0.01643, audio_tagging_loss=0.005704, over 16092.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.0929, pruned_loss=0.01371, audio_tagging_loss=0.008817, over 3036993.55 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:12:31,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2637153.3333333335, ans=0.125
2023-11-24 02:12:47,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2637220.0, ans=0.09899494936611666
2023-11-24 02:12:56,742 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395600
2023-11-24 02:13:16,941 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10850, loss[loss=0.05509, simple_loss=0.07987, pruned_loss=0.008655, audio_tagging_loss=0.006502, over 14030.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09178, pruned_loss=0.01349, audio_tagging_loss=0.00884, over 3033946.03 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:13:33,529 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.044e+01 8.225e+01 8.902e+01 9.397e+01 1.304e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-24 02:13:44,698 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.55 vs. limit=22.5
2023-11-24 02:14:00,312 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395650
2023-11-24 02:14:04,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2637620.0, ans=0.025
2023-11-24 02:14:10,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2637686.6666666665, ans=0.125
2023-11-24 02:14:16,171 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 02:14:18,579 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10900, loss[loss=0.05301, simple_loss=0.06636, pruned_loss=0.01153, audio_tagging_loss=0.008296, over 15129.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09148, pruned_loss=0.01323, audio_tagging_loss=0.008897, over 3041108.17 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:14:35,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2637820.0, ans=0.0
2023-11-24 02:14:51,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2637886.6666666665, ans=0.0
2023-11-24 02:15:01,907 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395700
2023-11-24 02:15:14,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2638020.0, ans=0.0
2023-11-24 02:15:19,295 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 10950, loss[loss=0.05966, simple_loss=0.08273, pruned_loss=0.01044, audio_tagging_loss=0.007854, over 15609.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.0921, pruned_loss=0.01334, audio_tagging_loss=0.00892, over 3047220.38 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:15:36,930 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.137e+01 8.249e+01 9.167e+01 9.838e+01 1.256e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 02:15:59,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2638286.6666666665, ans=0.0
2023-11-24 02:16:02,778 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395750
2023-11-24 02:16:04,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2638286.6666666665, ans=0.125
2023-11-24 02:16:07,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2638353.3333333335, ans=0.1
2023-11-24 02:16:22,046 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11000, loss[loss=0.0699, simple_loss=0.08811, pruned_loss=0.0166, audio_tagging_loss=0.009239, over 15069.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09187, pruned_loss=0.01326, audio_tagging_loss=0.009019, over 3045679.31 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:16:25,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2638420.0, ans=0.1
2023-11-24 02:16:30,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2638420.0, ans=0.1
2023-11-24 02:16:33,490 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 02:16:43,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2638486.6666666665, ans=0.0
2023-11-24 02:16:46,018 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.80 vs. limit=15.0
2023-11-24 02:16:52,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2638553.3333333335, ans=0.125
2023-11-24 02:17:04,770 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395800
2023-11-24 02:17:20,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2638686.6666666665, ans=0.2
2023-11-24 02:17:24,183 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11050, loss[loss=0.05923, simple_loss=0.08415, pruned_loss=0.01053, audio_tagging_loss=0.006628, over 16326.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09111, pruned_loss=0.01309, audio_tagging_loss=0.009209, over 3050503.09 frames. ], batch size: 60, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:17:25,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2638753.3333333335, ans=0.0
2023-11-24 02:17:31,663 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.65 vs. limit=15.0
2023-11-24 02:17:33,032 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.90 vs. limit=12.0
2023-11-24 02:17:40,568 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.311e+01 8.882e+01 9.606e+01 1.257e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-24 02:17:59,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2638886.6666666665, ans=0.125
2023-11-24 02:18:05,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2638953.3333333335, ans=0.2
2023-11-24 02:18:06,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2638953.3333333335, ans=0.2
2023-11-24 02:18:07,193 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395850
2023-11-24 02:18:13,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2639020.0, ans=0.0
2023-11-24 02:18:22,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2639020.0, ans=0.0
2023-11-24 02:18:25,474 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11100, loss[loss=0.05599, simple_loss=0.07309, pruned_loss=0.008837, audio_tagging_loss=0.01061, over 14692.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09051, pruned_loss=0.01308, audio_tagging_loss=0.009351, over 3057399.23 frames. ], batch size: 54, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:18:25,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2639086.6666666665, ans=0.125
2023-11-24 02:18:30,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2639086.6666666665, ans=0.125
2023-11-24 02:18:40,749 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.37 vs. limit=6.0
2023-11-24 02:18:47,977 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.86 vs. limit=15.0
2023-11-24 02:18:54,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2639220.0, ans=0.2
2023-11-24 02:19:04,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.95 vs. limit=12.0
2023-11-24 02:19:08,925 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395900
2023-11-24 02:19:18,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2639353.3333333335, ans=0.125
2023-11-24 02:19:27,081 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11150, loss[loss=0.04813, simple_loss=0.05967, pruned_loss=0.007169, audio_tagging_loss=0.01112, over 13744.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09053, pruned_loss=0.0132, audio_tagging_loss=0.009491, over 3055483.96 frames. ], batch size: 54, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:19:45,344 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.565e+01 8.458e+01 9.119e+01 9.750e+01 1.333e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 02:19:46,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2639486.6666666665, ans=0.1
2023-11-24 02:19:55,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2639553.3333333335, ans=0.07
2023-11-24 02:19:59,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2639553.3333333335, ans=0.1
2023-11-24 02:20:07,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2639620.0, ans=0.2
2023-11-24 02:20:10,029 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 395950
2023-11-24 02:20:11,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2639620.0, ans=0.09899494936611666
2023-11-24 02:20:29,248 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11200, loss[loss=0.06389, simple_loss=0.08572, pruned_loss=0.01246, audio_tagging_loss=0.008572, over 14512.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09083, pruned_loss=0.01332, audio_tagging_loss=0.009456, over 3055822.48 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:20:33,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2639753.3333333335, ans=0.125
2023-11-24 02:20:41,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2639820.0, ans=0.125
2023-11-24 02:20:49,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2639820.0, ans=0.1
2023-11-24 02:20:54,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2639886.6666666665, ans=0.0
2023-11-24 02:21:11,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2639953.3333333335, ans=0.0
2023-11-24 02:21:12,775 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396000
2023-11-24 02:21:14,227 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-396000.pt
2023-11-24 02:21:34,637 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11250, loss[loss=0.08013, simple_loss=0.1119, pruned_loss=0.01572, audio_tagging_loss=0.008476, over 14780.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09042, pruned_loss=0.01345, audio_tagging_loss=0.009453, over 3048676.18 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:21:53,435 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.556e+01 8.444e+01 8.848e+01 9.459e+01 1.220e+02, threshold=1.770e+02, percent-clipped=0.0
2023-11-24 02:22:18,385 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396050
2023-11-24 02:22:24,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2640353.3333333335, ans=0.0
2023-11-24 02:22:31,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2640353.3333333335, ans=0.1
2023-11-24 02:22:36,430 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11300, loss[loss=0.06748, simple_loss=0.09279, pruned_loss=0.01288, audio_tagging_loss=0.008209, over 13811.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09071, pruned_loss=0.01334, audio_tagging_loss=0.009241, over 3046446.20 frames. ], batch size: 52, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:22:39,537 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.95 vs. limit=15.0
2023-11-24 02:23:19,471 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396100
2023-11-24 02:23:30,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2640686.6666666665, ans=0.0
2023-11-24 02:23:35,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2640686.6666666665, ans=0.125
2023-11-24 02:23:38,693 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11350, loss[loss=0.09597, simple_loss=0.1226, pruned_loss=0.0273, audio_tagging_loss=0.007366, over 14632.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09147, pruned_loss=0.01343, audio_tagging_loss=0.009114, over 3052720.03 frames. ], batch size: 53, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:23:51,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2640820.0, ans=0.125
2023-11-24 02:23:57,202 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.374e+01 8.369e+01 8.906e+01 9.576e+01 1.384e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-24 02:23:57,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2640820.0, ans=0.1
2023-11-24 02:24:19,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2640953.3333333335, ans=0.2
2023-11-24 02:24:22,847 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396150
2023-11-24 02:24:25,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2640953.3333333335, ans=0.04949747468305833
2023-11-24 02:24:29,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2641020.0, ans=0.125
2023-11-24 02:24:41,100 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11400, loss[loss=0.07081, simple_loss=0.09422, pruned_loss=0.0129, audio_tagging_loss=0.0108, over 15691.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09161, pruned_loss=0.01341, audio_tagging_loss=0.008992, over 3051653.15 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:24:47,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2641086.6666666665, ans=0.125
2023-11-24 02:24:54,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2641153.3333333335, ans=0.125
2023-11-24 02:25:23,951 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396200
2023-11-24 02:25:31,396 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.99 vs. limit=15.0
2023-11-24 02:25:42,739 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11450, loss[loss=0.06867, simple_loss=0.09011, pruned_loss=0.01498, audio_tagging_loss=0.008637, over 15556.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09194, pruned_loss=0.01355, audio_tagging_loss=0.008961, over 3046770.40 frames. ], batch size: 58, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:25:43,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2641420.0, ans=0.0
2023-11-24 02:25:49,999 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.17 vs. limit=15.0
2023-11-24 02:25:59,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2641486.6666666665, ans=0.1
2023-11-24 02:26:02,135 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.524e+01 9.029e+01 9.681e+01 1.161e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 02:26:05,258 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.55 vs. limit=15.0
2023-11-24 02:26:07,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2641553.3333333335, ans=0.0
2023-11-24 02:26:12,178 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.78 vs. limit=15.0
2023-11-24 02:26:26,338 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396250
2023-11-24 02:26:34,471 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.60 vs. limit=10.0
2023-11-24 02:26:42,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2641686.6666666665, ans=0.0
2023-11-24 02:26:45,798 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11500, loss[loss=0.0599, simple_loss=0.07876, pruned_loss=0.009835, audio_tagging_loss=0.01069, over 14515.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09155, pruned_loss=0.01354, audio_tagging_loss=0.008974, over 3051036.17 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:27:24,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2641953.3333333335, ans=0.2
2023-11-24 02:27:30,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396300
2023-11-24 02:27:47,701 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11550, loss[loss=0.07105, simple_loss=0.09116, pruned_loss=0.01751, audio_tagging_loss=0.007961, over 14676.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09165, pruned_loss=0.01354, audio_tagging_loss=0.008901, over 3049790.57 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:27:49,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2642086.6666666665, ans=0.125
2023-11-24 02:27:49,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2642086.6666666665, ans=0.125
2023-11-24 02:27:51,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2642086.6666666665, ans=0.0
2023-11-24 02:28:06,103 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.401e+01 9.028e+01 9.629e+01 1.407e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 02:28:10,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2642153.3333333335, ans=0.0
2023-11-24 02:28:24,974 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.42 vs. limit=15.0
2023-11-24 02:28:26,658 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 02:28:31,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396350
2023-11-24 02:28:49,987 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11600, loss[loss=0.05894, simple_loss=0.07186, pruned_loss=0.01006, audio_tagging_loss=0.01296, over 15222.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09142, pruned_loss=0.01332, audio_tagging_loss=0.008987, over 3046760.05 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:28:52,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2642420.0, ans=0.125
2023-11-24 02:28:59,968 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.86 vs. limit=15.0
2023-11-24 02:29:30,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.30 vs. limit=15.0
2023-11-24 02:29:34,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396400
2023-11-24 02:29:43,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2642686.6666666665, ans=0.0
2023-11-24 02:29:48,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2642686.6666666665, ans=0.125
2023-11-24 02:29:53,916 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11650, loss[loss=0.06406, simple_loss=0.09377, pruned_loss=0.006538, audio_tagging_loss=0.01064, over 14972.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09272, pruned_loss=0.01338, audio_tagging_loss=0.008929, over 3046853.21 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:30:12,883 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.278e+01 8.940e+01 9.530e+01 1.250e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 02:30:17,200 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.74 vs. limit=15.0
2023-11-24 02:30:21,281 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.76 vs. limit=15.0
2023-11-24 02:30:25,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2642886.6666666665, ans=0.125
2023-11-24 02:30:37,977 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396450
2023-11-24 02:30:41,128 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=14.26 vs. limit=15.0
2023-11-24 02:30:55,487 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11700, loss[loss=0.05868, simple_loss=0.06489, pruned_loss=0.0118, audio_tagging_loss=0.01444, over 15680.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09226, pruned_loss=0.01343, audio_tagging_loss=0.008937, over 3047438.61 frames. ], batch size: 59, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:31:13,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.89 vs. limit=22.5
2023-11-24 02:31:28,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2643220.0, ans=0.125
2023-11-24 02:31:39,052 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396500
2023-11-24 02:31:56,465 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11750, loss[loss=0.05842, simple_loss=0.07444, pruned_loss=0.01249, audio_tagging_loss=0.008707, over 14662.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.0919, pruned_loss=0.01348, audio_tagging_loss=0.009019, over 3045687.53 frames. ], batch size: 53, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:31:57,710 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.80 vs. limit=15.0
2023-11-24 02:32:12,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2643486.6666666665, ans=0.0
2023-11-24 02:32:16,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2643486.6666666665, ans=0.125
2023-11-24 02:32:17,388 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.547e+01 9.165e+01 9.829e+01 1.276e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 02:32:40,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396550
2023-11-24 02:32:54,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2643686.6666666665, ans=0.125
2023-11-24 02:32:56,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2643686.6666666665, ans=15.0
2023-11-24 02:33:00,493 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11800, loss[loss=0.08784, simple_loss=0.1144, pruned_loss=0.02274, audio_tagging_loss=0.007906, over 15692.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09148, pruned_loss=0.01366, audio_tagging_loss=0.0091, over 3047226.37 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:33:06,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2643753.3333333335, ans=0.0
2023-11-24 02:33:43,348 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396600
2023-11-24 02:33:51,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2644020.0, ans=0.2
2023-11-24 02:33:54,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2644020.0, ans=0.125
2023-11-24 02:33:58,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2644020.0, ans=0.125
2023-11-24 02:34:01,635 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11850, loss[loss=0.0755, simple_loss=0.1062, pruned_loss=0.01195, audio_tagging_loss=0.01045, over 14791.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09124, pruned_loss=0.01365, audio_tagging_loss=0.009141, over 3044164.53 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:34:04,467 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.54 vs. limit=15.0
2023-11-24 02:34:20,345 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.605e+01 8.557e+01 9.128e+01 9.993e+01 1.182e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 02:34:45,225 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396650
2023-11-24 02:34:52,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2644353.3333333335, ans=0.125
2023-11-24 02:34:55,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2644353.3333333335, ans=0.125
2023-11-24 02:34:57,136 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2644353.3333333335, ans=0.125
2023-11-24 02:34:59,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2644353.3333333335, ans=0.125
2023-11-24 02:35:02,955 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11900, loss[loss=0.06755, simple_loss=0.09386, pruned_loss=0.01312, audio_tagging_loss=0.007505, over 14656.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.0916, pruned_loss=0.01358, audio_tagging_loss=0.009223, over 3042683.98 frames. ], batch size: 56, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:35:20,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2644486.6666666665, ans=0.0
2023-11-24 02:35:40,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2644620.0, ans=0.125
2023-11-24 02:35:43,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2644620.0, ans=0.125
2023-11-24 02:35:46,497 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396700
2023-11-24 02:35:56,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2644686.6666666665, ans=0.125
2023-11-24 02:36:05,993 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 11950, loss[loss=0.08854, simple_loss=0.125, pruned_loss=0.0204, audio_tagging_loss=0.005638, over 16082.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09129, pruned_loss=0.01355, audio_tagging_loss=0.009302, over 3042504.45 frames. ], batch size: 55, lr: 2.04e-03, grad_scale: 16.0
2023-11-24 02:36:22,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2644820.0, ans=0.125
2023-11-24 02:36:23,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2644820.0, ans=0.0
2023-11-24 02:36:25,408 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.463e+01 8.152e+01 8.882e+01 9.494e+01 1.160e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-24 02:36:29,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2644886.6666666665, ans=10.0
2023-11-24 02:36:41,211 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.25 vs. limit=15.0
2023-11-24 02:36:46,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2644953.3333333335, ans=0.0
2023-11-24 02:36:47,472 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396750
2023-11-24 02:36:57,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2645020.0, ans=0.125
2023-11-24 02:37:06,016 INFO [train_asr.py:1221] (0/4) Epoch 33, batch 12000, loss[loss=0.06708, simple_loss=0.08187, pruned_loss=0.01367, audio_tagging_loss=0.01247, over 14907.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09116, pruned_loss=0.01354, audio_tagging_loss=0.009365, over 3046048.93 frames. ], batch size: 57, lr: 2.04e-03, grad_scale: 32.0
2023-11-24 02:37:06,018 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 02:37:25,384 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([6.5074, 6.3637, 6.1581, 6.2131], device='cuda:0')
2023-11-24 02:37:41,204 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.0014, 3.8966, 3.7232, 3.1299], device='cuda:0')
2023-11-24 02:37:46,451 INFO [train_asr.py:1253] (0/4) Epoch 33, validation: loss=0.05829, simple_loss=0.05098, pruned_loss=0.005164, audio_tagging_loss=0.02763, over 4681554.00 frames. 
2023-11-24 02:37:46,452 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 02:37:46,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2645086.6666666665, ans=0.2
2023-11-24 02:37:46,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2645086.6666666665, ans=0.07
2023-11-24 02:37:52,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.71 vs. limit=12.0
2023-11-24 02:38:10,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2645220.0, ans=0.125
2023-11-24 02:38:15,469 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-33.pt
2023-11-24 02:38:48,876 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 0, loss[loss=0.06996, simple_loss=0.09159, pruned_loss=0.008742, audio_tagging_loss=0.01542, over 15369.00 frames. ], tot_loss[loss=0.06996, simple_loss=0.09159, pruned_loss=0.008742, audio_tagging_loss=0.01542, over 15369.00 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:38:48,888 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 02:39:24,522 INFO [train_asr.py:1253] (0/4) Epoch 34, validation: loss=0.058, simple_loss=0.05102, pruned_loss=0.005202, audio_tagging_loss=0.02729, over 4681554.00 frames. 
2023-11-24 02:39:24,522 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 02:39:29,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2645253.3333333335, ans=0.125
2023-11-24 02:39:32,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2645253.3333333335, ans=0.0
2023-11-24 02:39:34,084 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.84 vs. limit=15.0
2023-11-24 02:39:34,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2645253.3333333335, ans=0.2
2023-11-24 02:39:37,139 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396800
2023-11-24 02:39:56,875 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2645386.6666666665, ans=0.125
2023-11-24 02:40:14,285 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.74 vs. limit=15.0
2023-11-24 02:40:16,360 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.450e+01 9.137e+01 9.827e+01 1.054e+02 1.431e+02, threshold=1.965e+02, percent-clipped=0.0
2023-11-24 02:40:27,159 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 50, loss[loss=0.09423, simple_loss=0.1283, pruned_loss=0.01884, audio_tagging_loss=0.01125, over 15531.00 frames. ], tot_loss[loss=0.07604, simple_loss=0.09124, pruned_loss=0.01323, audio_tagging_loss=0.01719, over 687518.44 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:40:28,953 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.71 vs. limit=15.0
2023-11-24 02:40:29,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2645586.6666666665, ans=0.125
2023-11-24 02:40:37,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2645586.6666666665, ans=0.125
2023-11-24 02:40:39,318 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396850
2023-11-24 02:40:52,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2645720.0, ans=0.1
2023-11-24 02:41:22,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2645853.3333333335, ans=0.0
2023-11-24 02:41:29,491 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 100, loss[loss=0.0813, simple_loss=0.1092, pruned_loss=0.01294, audio_tagging_loss=0.01374, over 16493.00 frames. ], tot_loss[loss=0.07524, simple_loss=0.09134, pruned_loss=0.01317, audio_tagging_loss=0.0164, over 1216737.95 frames. ], batch size: 60, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:41:30,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2645920.0, ans=0.0
2023-11-24 02:41:39,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2645920.0, ans=0.2
2023-11-24 02:41:41,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396900
2023-11-24 02:42:09,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2646120.0, ans=0.1
2023-11-24 02:42:20,399 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.325e+01 8.964e+01 9.568e+01 1.027e+02 1.310e+02, threshold=1.914e+02, percent-clipped=0.0
2023-11-24 02:42:29,238 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.85 vs. limit=10.0
2023-11-24 02:42:31,557 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 150, loss[loss=0.06418, simple_loss=0.08469, pruned_loss=0.01145, audio_tagging_loss=0.01038, over 14811.00 frames. ], tot_loss[loss=0.07372, simple_loss=0.09144, pruned_loss=0.0132, audio_tagging_loss=0.0148, over 1629884.31 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:42:33,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2646253.3333333335, ans=0.1
2023-11-24 02:42:44,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 396950
2023-11-24 02:43:08,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2646453.3333333335, ans=0.125
2023-11-24 02:43:20,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2646520.0, ans=0.125
2023-11-24 02:43:24,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2646520.0, ans=0.025
2023-11-24 02:43:28,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2646520.0, ans=0.1
2023-11-24 02:43:34,092 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 200, loss[loss=0.05925, simple_loss=0.07826, pruned_loss=0.009232, audio_tagging_loss=0.01088, over 15534.00 frames. ], tot_loss[loss=0.07254, simple_loss=0.09231, pruned_loss=0.01327, audio_tagging_loss=0.01311, over 1945172.82 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:43:43,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2646586.6666666665, ans=0.07
2023-11-24 02:43:46,031 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397000
2023-11-24 02:43:51,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2646653.3333333335, ans=0.0
2023-11-24 02:43:55,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2646653.3333333335, ans=0.125
2023-11-24 02:44:20,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2646786.6666666665, ans=0.125
2023-11-24 02:44:23,335 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.49 vs. limit=10.0
2023-11-24 02:44:24,878 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.338e+01 8.517e+01 9.117e+01 1.003e+02 1.656e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 02:44:31,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2646853.3333333335, ans=0.125
2023-11-24 02:44:35,767 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 250, loss[loss=0.06572, simple_loss=0.09321, pruned_loss=0.01178, audio_tagging_loss=0.007337, over 15664.00 frames. ], tot_loss[loss=0.07143, simple_loss=0.09231, pruned_loss=0.01333, audio_tagging_loss=0.01195, over 2194770.92 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:44:37,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2646920.0, ans=0.125
2023-11-24 02:44:37,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2646920.0, ans=0.125
2023-11-24 02:44:40,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2646920.0, ans=0.2
2023-11-24 02:44:47,719 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397050
2023-11-24 02:45:05,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2647053.3333333335, ans=0.125
2023-11-24 02:45:15,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2647120.0, ans=0.2
2023-11-24 02:45:33,159 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.57 vs. limit=22.5
2023-11-24 02:45:37,012 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 300, loss[loss=0.07034, simple_loss=0.097, pruned_loss=0.0113, audio_tagging_loss=0.01054, over 14671.00 frames. ], tot_loss[loss=0.07012, simple_loss=0.09163, pruned_loss=0.01314, audio_tagging_loss=0.01116, over 2383758.25 frames. ], batch size: 53, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:45:50,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397100
2023-11-24 02:45:51,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2647320.0, ans=0.0
2023-11-24 02:46:04,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2647386.6666666665, ans=0.125
2023-11-24 02:46:11,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2647386.6666666665, ans=0.125
2023-11-24 02:46:26,288 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.05 vs. limit=15.0
2023-11-24 02:46:30,122 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.278e+01 8.816e+01 9.876e+01 1.220e+02, threshold=1.763e+02, percent-clipped=0.0
2023-11-24 02:46:40,769 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 350, loss[loss=0.08131, simple_loss=0.114, pruned_loss=0.01594, audio_tagging_loss=0.008348, over 16228.00 frames. ], tot_loss[loss=0.0699, simple_loss=0.09241, pruned_loss=0.01326, audio_tagging_loss=0.01044, over 2530713.02 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:46:52,736 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397150
2023-11-24 02:46:52,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2647653.3333333335, ans=0.1
2023-11-24 02:47:02,156 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:47:14,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2647720.0, ans=0.05
2023-11-24 02:47:30,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2647853.3333333335, ans=0.0
2023-11-24 02:47:30,912 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.27 vs. limit=15.0
2023-11-24 02:47:33,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2647853.3333333335, ans=0.125
2023-11-24 02:47:35,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2647853.3333333335, ans=0.125
2023-11-24 02:47:42,081 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 400, loss[loss=0.08048, simple_loss=0.1084, pruned_loss=0.019, audio_tagging_loss=0.007278, over 15594.00 frames. ], tot_loss[loss=0.06932, simple_loss=0.09215, pruned_loss=0.01321, audio_tagging_loss=0.01003, over 2646948.31 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:47:50,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2647920.0, ans=0.125
2023-11-24 02:47:54,092 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397200
2023-11-24 02:48:10,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2648053.3333333335, ans=0.125
2023-11-24 02:48:12,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2648053.3333333335, ans=0.05
2023-11-24 02:48:23,483 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.75 vs. limit=22.5
2023-11-24 02:48:35,278 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.924e+01 8.562e+01 9.066e+01 9.786e+01 1.375e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-24 02:48:44,725 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 450, loss[loss=0.06129, simple_loss=0.08245, pruned_loss=0.01247, audio_tagging_loss=0.007592, over 15083.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09243, pruned_loss=0.01335, audio_tagging_loss=0.00982, over 2731871.25 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:48:48,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2648253.3333333335, ans=0.125
2023-11-24 02:48:57,574 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397250
2023-11-24 02:49:15,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.59 vs. limit=15.0
2023-11-24 02:49:22,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2648453.3333333335, ans=0.1
2023-11-24 02:49:38,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2648520.0, ans=0.1
2023-11-24 02:49:41,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2648520.0, ans=0.0
2023-11-24 02:49:46,804 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 500, loss[loss=0.08806, simple_loss=0.1227, pruned_loss=0.01802, audio_tagging_loss=0.008701, over 14850.00 frames. ], tot_loss[loss=0.0697, simple_loss=0.09319, pruned_loss=0.01358, audio_tagging_loss=0.009525, over 2790893.58 frames. ], batch size: 55, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:49:58,578 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.63 vs. limit=15.0
2023-11-24 02:49:59,247 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397300
2023-11-24 02:50:04,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2648653.3333333335, ans=0.125
2023-11-24 02:50:19,908 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.65 vs. limit=12.0
2023-11-24 02:50:23,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2648786.6666666665, ans=0.125
2023-11-24 02:50:38,753 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.347e+01 9.082e+01 9.817e+01 1.239e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 02:50:48,953 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 550, loss[loss=0.05788, simple_loss=0.0801, pruned_loss=0.009645, audio_tagging_loss=0.008187, over 14740.00 frames. ], tot_loss[loss=0.06946, simple_loss=0.09292, pruned_loss=0.01357, audio_tagging_loss=0.00943, over 2846258.19 frames. ], batch size: 54, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:51:00,989 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397350
2023-11-24 02:51:08,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2648986.6666666665, ans=0.1
2023-11-24 02:51:09,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2648986.6666666665, ans=0.0
2023-11-24 02:51:12,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2649053.3333333335, ans=0.125
2023-11-24 02:51:21,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2649053.3333333335, ans=0.0
2023-11-24 02:51:49,955 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 600, loss[loss=0.0629, simple_loss=0.08431, pruned_loss=0.0101, audio_tagging_loss=0.01065, over 15401.00 frames. ], tot_loss[loss=0.06983, simple_loss=0.09351, pruned_loss=0.01377, audio_tagging_loss=0.009302, over 2895941.92 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:51:50,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2649253.3333333335, ans=0.0
2023-11-24 02:51:50,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2649253.3333333335, ans=0.0
2023-11-24 02:52:03,034 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397400
2023-11-24 02:52:17,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2649386.6666666665, ans=0.125
2023-11-24 02:52:33,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2649453.3333333335, ans=0.125
2023-11-24 02:52:42,872 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.848e+01 8.375e+01 9.177e+01 9.802e+01 2.402e+02, threshold=1.835e+02, percent-clipped=1.0
2023-11-24 02:52:43,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2649520.0, ans=0.125
2023-11-24 02:52:52,943 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 650, loss[loss=0.06746, simple_loss=0.08781, pruned_loss=0.01569, audio_tagging_loss=0.007873, over 14349.00 frames. ], tot_loss[loss=0.06916, simple_loss=0.09248, pruned_loss=0.01363, audio_tagging_loss=0.009293, over 2925265.94 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:53:04,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397450
2023-11-24 02:53:10,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2649653.3333333335, ans=0.05
2023-11-24 02:53:22,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2649720.0, ans=0.95
2023-11-24 02:53:24,720 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.04 vs. limit=22.5
2023-11-24 02:53:54,372 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 700, loss[loss=0.07187, simple_loss=0.0896, pruned_loss=0.01685, audio_tagging_loss=0.01022, over 16361.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09281, pruned_loss=0.01366, audio_tagging_loss=0.009243, over 2962645.40 frames. ], batch size: 61, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:54:02,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.24 vs. limit=22.5
2023-11-24 02:54:05,092 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.61 vs. limit=15.0
2023-11-24 02:54:06,945 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397500
2023-11-24 02:54:08,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2649986.6666666665, ans=0.2
2023-11-24 02:54:32,095 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2650120.0, ans=0.125
2023-11-24 02:54:41,563 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 02:54:44,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.whiten.whitening_limit, batch_count=2650186.6666666665, ans=15.0
2023-11-24 02:54:46,886 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.98 vs. limit=15.0
2023-11-24 02:54:47,546 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.267e+01 8.698e+01 9.188e+01 9.937e+01 1.605e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-24 02:54:55,824 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 750, loss[loss=0.07913, simple_loss=0.1178, pruned_loss=0.01382, audio_tagging_loss=0.006394, over 15382.00 frames. ], tot_loss[loss=0.06979, simple_loss=0.09383, pruned_loss=0.0137, audio_tagging_loss=0.009176, over 2980621.73 frames. ], batch size: 54, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:55:09,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397550
2023-11-24 02:55:11,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2650320.0, ans=0.125
2023-11-24 02:55:25,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2650386.6666666665, ans=0.0
2023-11-24 02:55:25,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2650386.6666666665, ans=0.5
2023-11-24 02:55:27,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2650386.6666666665, ans=0.1
2023-11-24 02:55:53,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2650520.0, ans=0.2
2023-11-24 02:55:53,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2650520.0, ans=0.025
2023-11-24 02:55:58,171 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 800, loss[loss=0.05581, simple_loss=0.07113, pruned_loss=0.008551, audio_tagging_loss=0.01169, over 15088.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09294, pruned_loss=0.01346, audio_tagging_loss=0.009173, over 2992566.37 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:56:10,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397600
2023-11-24 02:56:23,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.38 vs. limit=6.0
2023-11-24 02:56:46,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2650853.3333333335, ans=0.125
2023-11-24 02:56:51,290 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.215e+01 8.448e+01 9.134e+01 9.872e+01 1.389e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 02:56:59,554 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 850, loss[loss=0.04566, simple_loss=0.05358, pruned_loss=0.006334, audio_tagging_loss=0.01253, over 15461.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09342, pruned_loss=0.01352, audio_tagging_loss=0.009178, over 3006679.64 frames. ], batch size: 60, lr: 2.01e-03, grad_scale: 32.0
2023-11-24 02:57:12,017 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397650
2023-11-24 02:57:25,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2651053.3333333335, ans=0.2
2023-11-24 02:58:02,064 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 900, loss[loss=0.08384, simple_loss=0.1167, pruned_loss=0.01584, audio_tagging_loss=0.009646, over 14792.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09269, pruned_loss=0.01351, audio_tagging_loss=0.009264, over 3014698.63 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:58:14,484 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397700
2023-11-24 02:58:38,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2651453.3333333335, ans=0.2
2023-11-24 02:58:41,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2651453.3333333335, ans=0.0
2023-11-24 02:58:56,289 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.194e+01 8.559e+01 9.218e+01 1.010e+02 1.214e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 02:58:57,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2651520.0, ans=0.1
2023-11-24 02:59:04,568 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 950, loss[loss=0.06318, simple_loss=0.08238, pruned_loss=0.01166, audio_tagging_loss=0.01034, over 15711.00 frames. ], tot_loss[loss=0.0695, simple_loss=0.09346, pruned_loss=0.01361, audio_tagging_loss=0.009158, over 3025849.27 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 02:59:14,190 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.72 vs. limit=15.0
2023-11-24 02:59:17,024 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397750
2023-11-24 02:59:28,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2651720.0, ans=0.125
2023-11-24 02:59:46,384 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.85 vs. limit=15.0
2023-11-24 02:59:54,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2651853.3333333335, ans=0.125
2023-11-24 02:59:57,849 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.76 vs. limit=8.0
2023-11-24 03:00:04,527 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.49 vs. limit=15.0
2023-11-24 03:00:06,024 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.16 vs. limit=15.0
2023-11-24 03:00:06,450 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1000, loss[loss=0.07454, simple_loss=0.104, pruned_loss=0.01363, audio_tagging_loss=0.008881, over 15414.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09227, pruned_loss=0.01359, audio_tagging_loss=0.009192, over 3023673.25 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:00:18,397 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397800
2023-11-24 03:00:27,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2651986.6666666665, ans=0.1
2023-11-24 03:00:32,261 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:00:57,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2652186.6666666665, ans=0.1
2023-11-24 03:01:01,761 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.956e+01 8.415e+01 8.930e+01 9.689e+01 1.161e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 03:01:09,022 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1050, loss[loss=0.06023, simple_loss=0.0876, pruned_loss=0.009317, audio_tagging_loss=0.007112, over 16054.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09198, pruned_loss=0.01342, audio_tagging_loss=0.008995, over 3030561.13 frames. ], batch size: 60, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:01:11,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2652253.3333333335, ans=0.1
2023-11-24 03:01:16,031 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.94 vs. limit=22.5
2023-11-24 03:01:21,497 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397850
2023-11-24 03:01:41,712 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.37 vs. limit=15.0
2023-11-24 03:01:50,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2652453.3333333335, ans=0.07
2023-11-24 03:02:11,741 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1100, loss[loss=0.07347, simple_loss=0.1018, pruned_loss=0.0155, audio_tagging_loss=0.007085, over 14632.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09119, pruned_loss=0.01329, audio_tagging_loss=0.008962, over 3035508.87 frames. ], batch size: 58, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:02:15,971 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:02:19,254 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.09 vs. limit=15.0
2023-11-24 03:02:21,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2652586.6666666665, ans=0.0
2023-11-24 03:02:24,332 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397900
2023-11-24 03:02:25,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2652653.3333333335, ans=0.125
2023-11-24 03:02:26,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2652653.3333333335, ans=0.125
2023-11-24 03:02:38,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2652720.0, ans=0.125
2023-11-24 03:03:06,345 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.355e+01 8.539e+01 9.041e+01 9.654e+01 1.220e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 03:03:12,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2652920.0, ans=0.2
2023-11-24 03:03:13,518 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1150, loss[loss=0.07116, simple_loss=0.1016, pruned_loss=0.01305, audio_tagging_loss=0.007294, over 15584.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.0916, pruned_loss=0.01328, audio_tagging_loss=0.008933, over 3043625.63 frames. ], batch size: 57, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:03:15,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2652920.0, ans=0.09899494936611666
2023-11-24 03:03:23,786 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.12 vs. limit=12.0
2023-11-24 03:03:25,502 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 397950
2023-11-24 03:03:51,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2653120.0, ans=0.0
2023-11-24 03:04:11,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2653186.6666666665, ans=0.0
2023-11-24 03:04:13,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2653253.3333333335, ans=0.125
2023-11-24 03:04:14,295 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1200, loss[loss=0.06779, simple_loss=0.1008, pruned_loss=0.009789, audio_tagging_loss=0.00758, over 17533.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09202, pruned_loss=0.01332, audio_tagging_loss=0.00884, over 3048018.42 frames. ], batch size: 66, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:04:18,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2653253.3333333335, ans=0.125
2023-11-24 03:04:26,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398000
2023-11-24 03:04:52,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2653453.3333333335, ans=0.0
2023-11-24 03:04:55,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2653453.3333333335, ans=0.1
2023-11-24 03:05:07,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=2653520.0, ans=15.0
2023-11-24 03:05:09,788 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.593e+01 9.162e+01 9.902e+01 1.327e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 03:05:13,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2653520.0, ans=0.0
2023-11-24 03:05:16,282 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1250, loss[loss=0.07314, simple_loss=0.09726, pruned_loss=0.01415, audio_tagging_loss=0.01036, over 15237.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09183, pruned_loss=0.01332, audio_tagging_loss=0.008828, over 3044072.87 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:05:16,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2653586.6666666665, ans=0.125
2023-11-24 03:05:29,472 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398050
2023-11-24 03:05:40,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2653720.0, ans=0.125
2023-11-24 03:05:44,186 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.95 vs. limit=15.0
2023-11-24 03:05:50,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2653720.0, ans=0.1
2023-11-24 03:06:04,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2653853.3333333335, ans=0.125
2023-11-24 03:06:07,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2653853.3333333335, ans=0.0
2023-11-24 03:06:12,008 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.02 vs. limit=15.0
2023-11-24 03:06:18,468 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1300, loss[loss=0.07097, simple_loss=0.08851, pruned_loss=0.01791, audio_tagging_loss=0.008808, over 15526.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09202, pruned_loss=0.0135, audio_tagging_loss=0.008868, over 3045192.86 frames. ], batch size: 59, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:06:21,019 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2653920.0, ans=0.5
2023-11-24 03:06:29,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2653986.6666666665, ans=0.0
2023-11-24 03:06:30,353 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398100
2023-11-24 03:06:37,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2653986.6666666665, ans=0.125
2023-11-24 03:07:07,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2654186.6666666665, ans=0.1
2023-11-24 03:07:13,595 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.353e+01 8.205e+01 8.973e+01 9.705e+01 1.641e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-24 03:07:19,491 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1350, loss[loss=0.08509, simple_loss=0.1134, pruned_loss=0.02129, audio_tagging_loss=0.007097, over 15311.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09233, pruned_loss=0.01369, audio_tagging_loss=0.008908, over 3045822.98 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:07:25,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2654253.3333333335, ans=0.125
2023-11-24 03:07:31,301 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398150
2023-11-24 03:07:32,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2654320.0, ans=0.125
2023-11-24 03:07:34,471 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.86 vs. limit=15.0
2023-11-24 03:07:49,036 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.52 vs. limit=22.5
2023-11-24 03:08:02,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2654453.3333333335, ans=0.0
2023-11-24 03:08:03,987 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:08:15,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2654520.0, ans=0.0
2023-11-24 03:08:16,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2654520.0, ans=0.1
2023-11-24 03:08:17,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2654520.0, ans=0.125
2023-11-24 03:08:20,668 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1400, loss[loss=0.06799, simple_loss=0.1002, pruned_loss=0.01175, audio_tagging_loss=0.006149, over 15233.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09251, pruned_loss=0.01364, audio_tagging_loss=0.008953, over 3050909.35 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:08:34,338 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398200
2023-11-24 03:08:55,403 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.21 vs. limit=15.0
2023-11-24 03:09:17,338 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.778e+01 8.476e+01 9.016e+01 9.845e+01 1.286e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 03:09:24,003 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1450, loss[loss=0.07553, simple_loss=0.09053, pruned_loss=0.0193, audio_tagging_loss=0.01097, over 14763.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09274, pruned_loss=0.01377, audio_tagging_loss=0.009005, over 3054655.41 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:09:27,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2654920.0, ans=0.125
2023-11-24 03:09:35,973 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398250
2023-11-24 03:09:38,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2654986.6666666665, ans=0.125
2023-11-24 03:09:51,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2655053.3333333335, ans=0.95
2023-11-24 03:09:51,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2655053.3333333335, ans=0.125
2023-11-24 03:10:07,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2655120.0, ans=0.5
2023-11-24 03:10:10,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2655120.0, ans=0.125
2023-11-24 03:10:18,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2655186.6666666665, ans=0.125
2023-11-24 03:10:19,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2655186.6666666665, ans=0.125
2023-11-24 03:10:19,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2655186.6666666665, ans=0.125
2023-11-24 03:10:25,535 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1500, loss[loss=0.0621, simple_loss=0.08627, pruned_loss=0.012, audio_tagging_loss=0.006961, over 14902.00 frames. ], tot_loss[loss=0.06968, simple_loss=0.09351, pruned_loss=0.01391, audio_tagging_loss=0.009015, over 3050240.35 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:10:26,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2655253.3333333335, ans=0.2
2023-11-24 03:10:37,550 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398300
2023-11-24 03:10:51,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2655386.6666666665, ans=0.0
2023-11-24 03:10:54,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2655386.6666666665, ans=0.0
2023-11-24 03:11:06,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2655453.3333333335, ans=0.0
2023-11-24 03:11:21,388 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.180e+01 8.576e+01 9.222e+01 1.014e+02 1.240e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 03:11:26,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2655586.6666666665, ans=0.125
2023-11-24 03:11:27,372 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1550, loss[loss=0.08523, simple_loss=0.1214, pruned_loss=0.01508, audio_tagging_loss=0.009466, over 15382.00 frames. ], tot_loss[loss=0.06948, simple_loss=0.09304, pruned_loss=0.01379, audio_tagging_loss=0.009165, over 3046555.28 frames. ], batch size: 56, lr: 2.01e-03, grad_scale: 16.0
2023-11-24 03:11:41,111 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398350
2023-11-24 03:11:41,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.65 vs. limit=12.0
2023-11-24 03:12:00,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2655720.0, ans=15.0
2023-11-24 03:12:06,596 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.67 vs. limit=15.0
2023-11-24 03:12:06,674 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.20 vs. limit=15.0
2023-11-24 03:12:11,638 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.38 vs. limit=15.0
2023-11-24 03:12:31,492 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1600, loss[loss=0.06426, simple_loss=0.07969, pruned_loss=0.01423, audio_tagging_loss=0.01018, over 15877.00 frames. ], tot_loss[loss=0.06955, simple_loss=0.09321, pruned_loss=0.01384, audio_tagging_loss=0.009112, over 3050798.68 frames. ], batch size: 61, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:12:43,968 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398400
2023-11-24 03:12:44,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2655986.6666666665, ans=0.025
2023-11-24 03:12:52,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2655986.6666666665, ans=0.0
2023-11-24 03:12:55,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2656053.3333333335, ans=0.1
2023-11-24 03:13:15,588 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=7.04 vs. limit=15.0
2023-11-24 03:13:19,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0
2023-11-24 03:13:27,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2656186.6666666665, ans=0.0
2023-11-24 03:13:28,034 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.243e+01 8.406e+01 9.091e+01 9.636e+01 1.326e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-24 03:13:29,899 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.65 vs. limit=15.0
2023-11-24 03:13:32,112 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.85 vs. limit=15.0
2023-11-24 03:13:32,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2656253.3333333335, ans=0.125
2023-11-24 03:13:33,941 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1650, loss[loss=0.08419, simple_loss=0.1094, pruned_loss=0.01926, audio_tagging_loss=0.01021, over 14227.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09292, pruned_loss=0.01376, audio_tagging_loss=0.009134, over 3047680.61 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:13:41,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2656253.3333333335, ans=0.1
2023-11-24 03:13:45,979 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398450
2023-11-24 03:14:09,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2656386.6666666665, ans=0.125
2023-11-24 03:14:29,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2656520.0, ans=0.125
2023-11-24 03:14:33,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2656520.0, ans=0.125
2023-11-24 03:14:36,431 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1700, loss[loss=0.0731, simple_loss=0.09447, pruned_loss=0.01399, audio_tagging_loss=0.01187, over 15119.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09153, pruned_loss=0.01353, audio_tagging_loss=0.009222, over 3045815.95 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:14:36,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2656586.6666666665, ans=0.1
2023-11-24 03:14:49,055 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398500
2023-11-24 03:14:52,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2656653.3333333335, ans=0.0
2023-11-24 03:15:31,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2656853.3333333335, ans=0.125
2023-11-24 03:15:32,713 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.306e+01 8.348e+01 8.790e+01 9.631e+01 1.141e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-24 03:15:39,523 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1750, loss[loss=0.05809, simple_loss=0.0819, pruned_loss=0.009325, audio_tagging_loss=0.00781, over 15190.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09229, pruned_loss=0.01367, audio_tagging_loss=0.009119, over 3043542.39 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:15:51,354 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398550
2023-11-24 03:16:05,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2657053.3333333335, ans=0.1
2023-11-24 03:16:11,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2657053.3333333335, ans=0.1
2023-11-24 03:16:34,580 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.16 vs. limit=15.0
2023-11-24 03:16:41,190 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1800, loss[loss=0.06381, simple_loss=0.09094, pruned_loss=0.009909, audio_tagging_loss=0.008427, over 15569.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09194, pruned_loss=0.01346, audio_tagging_loss=0.009059, over 3044160.61 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:16:53,757 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398600
2023-11-24 03:17:01,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2657320.0, ans=0.0
2023-11-24 03:17:07,889 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:17:09,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2657386.6666666665, ans=0.1
2023-11-24 03:17:10,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2657386.6666666665, ans=0.0
2023-11-24 03:17:18,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2657453.3333333335, ans=0.125
2023-11-24 03:17:20,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2657453.3333333335, ans=0.0
2023-11-24 03:17:37,904 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.560e+01 9.161e+01 9.912e+01 1.351e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 03:17:40,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2657520.0, ans=0.125
2023-11-24 03:17:43,928 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1850, loss[loss=0.07591, simple_loss=0.1101, pruned_loss=0.01468, audio_tagging_loss=0.006172, over 15459.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09254, pruned_loss=0.0137, audio_tagging_loss=0.008937, over 3046962.29 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:17:45,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2657586.6666666665, ans=0.0
2023-11-24 03:17:45,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2657586.6666666665, ans=0.125
2023-11-24 03:17:56,705 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398650
2023-11-24 03:18:05,464 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.46 vs. limit=12.0
2023-11-24 03:18:08,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2657720.0, ans=0.0
2023-11-24 03:18:09,277 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.18 vs. limit=12.0
2023-11-24 03:18:19,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2657720.0, ans=0.95
2023-11-24 03:18:21,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2657786.6666666665, ans=0.125
2023-11-24 03:18:47,059 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1900, loss[loss=0.05105, simple_loss=0.06322, pruned_loss=0.007122, audio_tagging_loss=0.01232, over 15500.00 frames. ], tot_loss[loss=0.06879, simple_loss=0.0926, pruned_loss=0.01353, audio_tagging_loss=0.008952, over 3054661.41 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:18:56,205 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.87 vs. limit=15.0
2023-11-24 03:18:59,043 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398700
2023-11-24 03:19:34,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2658120.0, ans=10.0
2023-11-24 03:19:36,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2658186.6666666665, ans=0.0
2023-11-24 03:19:43,004 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.792e+01 8.306e+01 8.939e+01 9.602e+01 1.298e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 03:19:47,773 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 1950, loss[loss=0.06726, simple_loss=0.09632, pruned_loss=0.01048, audio_tagging_loss=0.008622, over 15859.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.0917, pruned_loss=0.01346, audio_tagging_loss=0.008949, over 3048967.90 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:20:00,311 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398750
2023-11-24 03:20:00,955 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.24 vs. limit=22.5
2023-11-24 03:20:14,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2658386.6666666665, ans=0.025
2023-11-24 03:20:27,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2658453.3333333335, ans=0.125
2023-11-24 03:20:49,784 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2000, loss[loss=0.06268, simple_loss=0.0819, pruned_loss=0.01272, audio_tagging_loss=0.009009, over 15656.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09213, pruned_loss=0.01357, audio_tagging_loss=0.008927, over 3051458.14 frames. ], batch size: 61, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:20:50,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2658586.6666666665, ans=0.1
2023-11-24 03:20:58,581 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.09 vs. limit=15.0
2023-11-24 03:21:02,302 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398800
2023-11-24 03:21:04,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.34 vs. limit=22.5
2023-11-24 03:21:08,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2658653.3333333335, ans=0.125
2023-11-24 03:21:11,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2658653.3333333335, ans=0.125
2023-11-24 03:21:16,134 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2658720.0, ans=0.125
2023-11-24 03:21:22,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2658720.0, ans=0.0
2023-11-24 03:21:39,689 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.04 vs. limit=12.0
2023-11-24 03:21:45,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2658853.3333333335, ans=0.0
2023-11-24 03:21:46,687 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.321e+01 9.002e+01 9.770e+01 1.545e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 03:21:49,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2658853.3333333335, ans=0.0
2023-11-24 03:21:51,436 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2050, loss[loss=0.07428, simple_loss=0.09586, pruned_loss=0.01755, audio_tagging_loss=0.008791, over 15256.00 frames. ], tot_loss[loss=0.06887, simple_loss=0.09259, pruned_loss=0.01368, audio_tagging_loss=0.008899, over 3051682.98 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:21:58,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2658920.0, ans=0.1
2023-11-24 03:21:59,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2658920.0, ans=0.025
2023-11-24 03:22:02,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2658920.0, ans=0.2
2023-11-24 03:22:04,421 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398850
2023-11-24 03:22:15,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2659053.3333333335, ans=0.5
2023-11-24 03:22:39,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2659120.0, ans=0.125
2023-11-24 03:22:53,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2659253.3333333335, ans=0.125
2023-11-24 03:22:54,386 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2100, loss[loss=0.07852, simple_loss=0.111, pruned_loss=0.0147, audio_tagging_loss=0.008343, over 15259.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09189, pruned_loss=0.01346, audio_tagging_loss=0.008866, over 3046669.83 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:23:01,512 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.05 vs. limit=22.5
2023-11-24 03:23:06,745 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398900
2023-11-24 03:23:06,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2659320.0, ans=0.0
2023-11-24 03:23:24,131 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=7.87 vs. limit=15.0
2023-11-24 03:23:29,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2659386.6666666665, ans=0.125
2023-11-24 03:23:35,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2659453.3333333335, ans=0.0
2023-11-24 03:23:52,488 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.107e+01 8.464e+01 8.942e+01 9.567e+01 1.124e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 03:23:54,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.65 vs. limit=15.0
2023-11-24 03:23:56,163 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2150, loss[loss=0.06975, simple_loss=0.09332, pruned_loss=0.01277, audio_tagging_loss=0.01032, over 15476.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.0908, pruned_loss=0.01323, audio_tagging_loss=0.00894, over 3042599.94 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:24:09,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 398950
2023-11-24 03:24:12,095 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.93 vs. limit=15.0
2023-11-24 03:24:19,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2659653.3333333335, ans=0.1
2023-11-24 03:24:33,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2659786.6666666665, ans=0.125
2023-11-24 03:24:34,095 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:24:36,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2659786.6666666665, ans=0.1
2023-11-24 03:24:59,230 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2200, loss[loss=0.07267, simple_loss=0.09674, pruned_loss=0.01553, audio_tagging_loss=0.008769, over 14941.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09242, pruned_loss=0.01349, audio_tagging_loss=0.008893, over 3044892.33 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:25:12,636 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399000
2023-11-24 03:25:49,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2660186.6666666665, ans=0.1
2023-11-24 03:25:58,368 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.388e+01 8.513e+01 9.169e+01 1.008e+02 1.249e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 03:26:01,968 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2250, loss[loss=0.0561, simple_loss=0.07395, pruned_loss=0.009888, audio_tagging_loss=0.009237, over 15156.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09164, pruned_loss=0.01331, audio_tagging_loss=0.00898, over 3041928.39 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:26:13,841 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399050
2023-11-24 03:26:18,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2660320.0, ans=0.2
2023-11-24 03:26:24,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2660386.6666666665, ans=0.0
2023-11-24 03:26:35,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2660386.6666666665, ans=0.2
2023-11-24 03:26:42,373 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.66 vs. limit=15.0
2023-11-24 03:27:02,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2660586.6666666665, ans=0.0
2023-11-24 03:27:02,883 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2300, loss[loss=0.0862, simple_loss=0.1114, pruned_loss=0.01848, audio_tagging_loss=0.01203, over 14503.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.0916, pruned_loss=0.0135, audio_tagging_loss=0.009032, over 3047545.76 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:27:03,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2660586.6666666665, ans=0.125
2023-11-24 03:27:04,317 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:27:14,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399100
2023-11-24 03:27:15,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2660653.3333333335, ans=0.125
2023-11-24 03:27:33,044 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.09 vs. limit=12.0
2023-11-24 03:27:40,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_na.min_abs, batch_count=2660786.6666666665, ans=0.02
2023-11-24 03:27:47,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2660786.6666666665, ans=0.125
2023-11-24 03:27:57,453 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:28:00,933 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.716e+01 8.488e+01 8.965e+01 9.835e+01 1.215e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 03:28:05,082 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2350, loss[loss=0.07378, simple_loss=0.1066, pruned_loss=0.01119, audio_tagging_loss=0.009307, over 15564.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09097, pruned_loss=0.01333, audio_tagging_loss=0.009145, over 3045601.94 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:28:09,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2660920.0, ans=0.125
2023-11-24 03:28:18,723 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399150
2023-11-24 03:28:20,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.64 vs. limit=15.0
2023-11-24 03:28:54,599 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.28 vs. limit=12.0
2023-11-24 03:29:05,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2661186.6666666665, ans=0.125
2023-11-24 03:29:09,007 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2400, loss[loss=0.0822, simple_loss=0.1066, pruned_loss=0.01872, audio_tagging_loss=0.01018, over 14450.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09121, pruned_loss=0.01354, audio_tagging_loss=0.00927, over 3044596.69 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:29:21,164 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399200
2023-11-24 03:29:22,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2661320.0, ans=0.125
2023-11-24 03:29:23,494 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.25 vs. limit=15.0
2023-11-24 03:29:37,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2661386.6666666665, ans=0.0
2023-11-24 03:30:08,349 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.386e+01 9.132e+01 1.001e+02 1.473e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 03:30:10,824 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2450, loss[loss=0.07434, simple_loss=0.095, pruned_loss=0.01727, audio_tagging_loss=0.00957, over 15247.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09037, pruned_loss=0.01341, audio_tagging_loss=0.009404, over 3041608.34 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:30:15,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2661586.6666666665, ans=0.0
2023-11-24 03:30:22,916 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399250
2023-11-24 03:30:41,583 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.98 vs. limit=12.0
2023-11-24 03:31:00,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2661853.3333333335, ans=0.125
2023-11-24 03:31:11,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2661920.0, ans=0.0
2023-11-24 03:31:11,915 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2500, loss[loss=0.0738, simple_loss=0.1047, pruned_loss=0.01398, audio_tagging_loss=0.007465, over 15628.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09078, pruned_loss=0.01331, audio_tagging_loss=0.009357, over 3045276.41 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:31:16,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2661920.0, ans=0.0
2023-11-24 03:31:23,328 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.72 vs. limit=15.0
2023-11-24 03:31:25,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2661986.6666666665, ans=0.125
2023-11-24 03:31:26,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399300
2023-11-24 03:31:34,133 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.56 vs. limit=10.0
2023-11-24 03:32:05,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2662186.6666666665, ans=0.2
2023-11-24 03:32:05,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2662186.6666666665, ans=0.125
2023-11-24 03:32:10,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2662186.6666666665, ans=10.0
2023-11-24 03:32:12,862 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0
2023-11-24 03:32:13,412 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.419e+01 8.936e+01 9.796e+01 1.161e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-24 03:32:13,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2662186.6666666665, ans=0.07
2023-11-24 03:32:16,387 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2550, loss[loss=0.06226, simple_loss=0.08828, pruned_loss=0.01097, audio_tagging_loss=0.007141, over 15609.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09083, pruned_loss=0.01339, audio_tagging_loss=0.009245, over 3047808.56 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:32:19,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2662253.3333333335, ans=0.125
2023-11-24 03:32:28,508 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399350
2023-11-24 03:32:32,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2662320.0, ans=0.0
2023-11-24 03:32:37,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2.whitening_limit, batch_count=2662320.0, ans=15.0
2023-11-24 03:32:39,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2662386.6666666665, ans=0.1
2023-11-24 03:32:49,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2662386.6666666665, ans=0.0
2023-11-24 03:33:18,425 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2600, loss[loss=0.06669, simple_loss=0.08764, pruned_loss=0.01235, audio_tagging_loss=0.01052, over 14396.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.08971, pruned_loss=0.01309, audio_tagging_loss=0.009101, over 3048063.89 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:33:30,214 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399400
2023-11-24 03:33:36,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2662653.3333333335, ans=0.1
2023-11-24 03:34:07,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2662853.3333333335, ans=0.0
2023-11-24 03:34:17,524 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.388e+01 8.667e+01 9.247e+01 9.854e+01 1.236e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 03:34:19,852 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2650, loss[loss=0.06028, simple_loss=0.08101, pruned_loss=0.01132, audio_tagging_loss=0.008451, over 16549.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.08897, pruned_loss=0.01303, audio_tagging_loss=0.00908, over 3042539.97 frames. ], batch size: 62, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:34:31,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2662986.6666666665, ans=0.0
2023-11-24 03:34:32,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399450
2023-11-24 03:34:38,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2662986.6666666665, ans=0.125
2023-11-24 03:34:49,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2663053.3333333335, ans=0.0
2023-11-24 03:34:55,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2663053.3333333335, ans=0.1
2023-11-24 03:35:23,193 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2700, loss[loss=0.05774, simple_loss=0.07458, pruned_loss=0.009059, audio_tagging_loss=0.01139, over 15195.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.08928, pruned_loss=0.01312, audio_tagging_loss=0.009012, over 3046768.51 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:35:35,661 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399500
2023-11-24 03:35:36,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2663320.0, ans=0.035
2023-11-24 03:36:07,323 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.42 vs. limit=15.0
2023-11-24 03:36:12,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2663520.0, ans=0.125
2023-11-24 03:36:16,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2663520.0, ans=0.1
2023-11-24 03:36:17,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2663520.0, ans=0.125
2023-11-24 03:36:23,396 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.382e+01 8.295e+01 8.758e+01 9.490e+01 1.330e+02, threshold=1.752e+02, percent-clipped=0.0
2023-11-24 03:36:23,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2663520.0, ans=0.0
2023-11-24 03:36:24,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2663586.6666666665, ans=0.0
2023-11-24 03:36:25,807 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2750, loss[loss=0.09164, simple_loss=0.13, pruned_loss=0.01648, audio_tagging_loss=0.01015, over 15909.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.08998, pruned_loss=0.01314, audio_tagging_loss=0.009041, over 3049493.19 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:36:34,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2663586.6666666665, ans=0.025
2023-11-24 03:36:38,046 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399550
2023-11-24 03:36:51,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2663720.0, ans=0.5
2023-11-24 03:36:53,095 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.66 vs. limit=15.0
2023-11-24 03:36:58,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2663720.0, ans=0.125
2023-11-24 03:37:14,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2663853.3333333335, ans=0.0
2023-11-24 03:37:18,096 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:37:27,424 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2800, loss[loss=0.06938, simple_loss=0.0974, pruned_loss=0.01146, audio_tagging_loss=0.009213, over 14657.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09055, pruned_loss=0.01323, audio_tagging_loss=0.008992, over 3048132.90 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:37:40,126 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399600
2023-11-24 03:37:46,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2663986.6666666665, ans=0.0
2023-11-24 03:37:55,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten.whitening_limit, batch_count=2664053.3333333335, ans=15.0
2023-11-24 03:37:56,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2664053.3333333335, ans=0.125
2023-11-24 03:38:02,922 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.13 vs. limit=15.0
2023-11-24 03:38:07,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2664120.0, ans=0.125
2023-11-24 03:38:16,013 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.67 vs. limit=6.0
2023-11-24 03:38:18,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2664186.6666666665, ans=0.07
2023-11-24 03:38:25,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2664186.6666666665, ans=0.125
2023-11-24 03:38:27,683 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.351e+01 8.911e+01 9.699e+01 1.155e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-24 03:38:30,609 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2850, loss[loss=0.08758, simple_loss=0.1213, pruned_loss=0.02175, audio_tagging_loss=0.005191, over 14865.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09041, pruned_loss=0.01312, audio_tagging_loss=0.008987, over 3054178.48 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:38:42,921 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399650
2023-11-24 03:38:54,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2664386.6666666665, ans=0.1
2023-11-24 03:39:11,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2664453.3333333335, ans=0.0
2023-11-24 03:39:14,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2664453.3333333335, ans=0.125
2023-11-24 03:39:20,364 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.85 vs. limit=15.0
2023-11-24 03:39:25,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2664520.0, ans=0.0
2023-11-24 03:39:28,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2664520.0, ans=0.125
2023-11-24 03:39:30,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2664520.0, ans=0.1
2023-11-24 03:39:32,511 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2900, loss[loss=0.05351, simple_loss=0.07, pruned_loss=0.01041, audio_tagging_loss=0.0081, over 15805.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.0906, pruned_loss=0.01325, audio_tagging_loss=0.008985, over 3049319.93 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:39:34,453 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:39:35,887 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:39:44,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2664653.3333333335, ans=0.0
2023-11-24 03:39:45,038 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399700
2023-11-24 03:39:58,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2664720.0, ans=0.95
2023-11-24 03:40:07,028 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2664720.0, ans=0.0
2023-11-24 03:40:07,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2664720.0, ans=0.125
2023-11-24 03:40:13,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.28 vs. limit=15.0
2023-11-24 03:40:18,098 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.33 vs. limit=15.0
2023-11-24 03:40:18,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2664786.6666666665, ans=0.125
2023-11-24 03:40:27,086 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.90 vs. limit=10.0
2023-11-24 03:40:31,990 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.745e+01 8.311e+01 9.104e+01 9.838e+01 1.191e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 03:40:34,374 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 2950, loss[loss=0.09209, simple_loss=0.1321, pruned_loss=0.02021, audio_tagging_loss=0.005838, over 16193.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09177, pruned_loss=0.01349, audio_tagging_loss=0.008928, over 3051265.42 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:40:44,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2664920.0, ans=0.0
2023-11-24 03:40:47,062 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399750
2023-11-24 03:41:03,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2665053.3333333335, ans=0.2
2023-11-24 03:41:12,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2665120.0, ans=0.125
2023-11-24 03:41:23,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2665186.6666666665, ans=0.0
2023-11-24 03:41:30,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2665186.6666666665, ans=0.125
2023-11-24 03:41:36,938 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3000, loss[loss=0.06732, simple_loss=0.08573, pruned_loss=0.01442, audio_tagging_loss=0.01003, over 14877.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09216, pruned_loss=0.01349, audio_tagging_loss=0.008977, over 3055907.97 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:41:36,940 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 03:42:18,350 INFO [train_asr.py:1253] (0/4) Epoch 34, validation: loss=0.05766, simple_loss=0.05087, pruned_loss=0.005081, audio_tagging_loss=0.02714, over 4681554.00 frames. 
2023-11-24 03:42:18,351 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 03:42:25,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2665253.3333333335, ans=0.125
2023-11-24 03:42:31,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399800
2023-11-24 03:43:05,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2665453.3333333335, ans=0.0
2023-11-24 03:43:18,389 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.35 vs. limit=15.0
2023-11-24 03:43:18,712 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.546e+01 8.558e+01 9.257e+01 9.837e+01 1.292e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 03:43:21,065 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3050, loss[loss=0.08882, simple_loss=0.1238, pruned_loss=0.01943, audio_tagging_loss=0.007475, over 14769.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09301, pruned_loss=0.01369, audio_tagging_loss=0.008965, over 3048737.00 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:43:34,252 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399850
2023-11-24 03:43:54,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2665720.0, ans=0.025
2023-11-24 03:43:54,505 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.80 vs. limit=15.0
2023-11-24 03:43:57,324 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:43:58,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2665786.6666666665, ans=0.035
2023-11-24 03:44:09,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2665853.3333333335, ans=0.125
2023-11-24 03:44:23,998 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3100, loss[loss=0.05984, simple_loss=0.07508, pruned_loss=0.01142, audio_tagging_loss=0.01088, over 15413.00 frames. ], tot_loss[loss=0.06963, simple_loss=0.09363, pruned_loss=0.01377, audio_tagging_loss=0.00905, over 3049470.26 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:44:35,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2665986.6666666665, ans=0.07
2023-11-24 03:44:36,347 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399900
2023-11-24 03:44:52,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2666053.3333333335, ans=0.0
2023-11-24 03:44:59,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2666120.0, ans=0.2
2023-11-24 03:45:00,716 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:45:01,388 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.51 vs. limit=22.5
2023-11-24 03:45:08,968 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:45:15,868 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:45:23,796 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.765e+01 9.339e+01 1.018e+02 1.384e+02, threshold=1.868e+02, percent-clipped=0.0
2023-11-24 03:45:26,234 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3150, loss[loss=0.06719, simple_loss=0.08638, pruned_loss=0.013, audio_tagging_loss=0.011, over 16149.00 frames. ], tot_loss[loss=0.06997, simple_loss=0.09409, pruned_loss=0.01387, audio_tagging_loss=0.009052, over 3050468.35 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:45:36,505 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.77 vs. limit=15.0
2023-11-24 03:45:38,940 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 399950
2023-11-24 03:45:47,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2666320.0, ans=0.0
2023-11-24 03:46:05,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2666453.3333333335, ans=0.2
2023-11-24 03:46:06,921 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:46:08,439 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.53 vs. limit=15.0
2023-11-24 03:46:20,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2666520.0, ans=0.125
2023-11-24 03:46:23,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2666520.0, ans=0.09899494936611666
2023-11-24 03:46:24,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2666520.0, ans=0.1
2023-11-24 03:46:28,857 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3200, loss[loss=0.05631, simple_loss=0.07633, pruned_loss=0.008729, audio_tagging_loss=0.009409, over 14023.00 frames. ], tot_loss[loss=0.06921, simple_loss=0.09293, pruned_loss=0.01359, audio_tagging_loss=0.009151, over 3053771.72 frames. ], batch size: 51, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:46:41,331 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400000
2023-11-24 03:46:41,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2666653.3333333335, ans=0.0
2023-11-24 03:46:42,827 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-400000.pt
2023-11-24 03:46:50,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2666653.3333333335, ans=0.0
2023-11-24 03:47:01,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2666720.0, ans=0.025
2023-11-24 03:47:04,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2666720.0, ans=0.125
2023-11-24 03:47:31,748 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.890e+01 8.504e+01 9.216e+01 9.938e+01 1.136e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 03:47:34,781 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3250, loss[loss=0.05937, simple_loss=0.07887, pruned_loss=0.01253, audio_tagging_loss=0.007408, over 15701.00 frames. ], tot_loss[loss=0.06939, simple_loss=0.09297, pruned_loss=0.01375, audio_tagging_loss=0.009153, over 3055221.87 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:47:41,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2666920.0, ans=0.1
2023-11-24 03:47:44,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2666920.0, ans=0.125
2023-11-24 03:47:44,344 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.99 vs. limit=10.0
2023-11-24 03:47:45,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2666920.0, ans=0.2
2023-11-24 03:47:47,483 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400050
2023-11-24 03:47:47,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2666986.6666666665, ans=0.0
2023-11-24 03:48:17,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2667120.0, ans=0.1
2023-11-24 03:48:19,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2667120.0, ans=0.2
2023-11-24 03:48:22,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2667120.0, ans=0.125
2023-11-24 03:48:37,216 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3300, loss[loss=0.04182, simple_loss=0.05078, pruned_loss=0.003614, audio_tagging_loss=0.01282, over 15926.00 frames. ], tot_loss[loss=0.06896, simple_loss=0.09217, pruned_loss=0.01361, audio_tagging_loss=0.009263, over 3052049.24 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:48:49,246 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400100
2023-11-24 03:48:55,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2667320.0, ans=0.125
2023-11-24 03:49:02,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.00 vs. limit=15.0
2023-11-24 03:49:06,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2667386.6666666665, ans=0.125
2023-11-24 03:49:25,221 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.55 vs. limit=15.0
2023-11-24 03:49:35,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2667520.0, ans=0.0
2023-11-24 03:49:37,336 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.848e+01 8.541e+01 9.179e+01 9.840e+01 1.429e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 03:49:38,557 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3350, loss[loss=0.06391, simple_loss=0.07554, pruned_loss=0.01544, audio_tagging_loss=0.01069, over 14552.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09191, pruned_loss=0.01349, audio_tagging_loss=0.009208, over 3058505.40 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:49:42,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2667586.6666666665, ans=0.125
2023-11-24 03:49:49,560 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.09 vs. limit=22.5
2023-11-24 03:49:51,824 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400150
2023-11-24 03:49:57,715 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.19 vs. limit=10.0
2023-11-24 03:49:58,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2667653.3333333335, ans=0.125
2023-11-24 03:49:59,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2667653.3333333335, ans=0.125
2023-11-24 03:50:05,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2667720.0, ans=0.125
2023-11-24 03:50:19,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2667786.6666666665, ans=0.1
2023-11-24 03:50:20,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2667786.6666666665, ans=0.125
2023-11-24 03:50:27,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2667853.3333333335, ans=0.125
2023-11-24 03:50:36,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2667853.3333333335, ans=0.125
2023-11-24 03:50:41,770 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3400, loss[loss=0.05826, simple_loss=0.07703, pruned_loss=0.0114, audio_tagging_loss=0.008345, over 15049.00 frames. ], tot_loss[loss=0.06881, simple_loss=0.09243, pruned_loss=0.01355, audio_tagging_loss=0.009043, over 3062726.23 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:50:49,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2667920.0, ans=0.2
2023-11-24 03:50:51,738 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.88 vs. limit=15.0
2023-11-24 03:50:54,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400200
2023-11-24 03:51:03,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2667986.6666666665, ans=0.125
2023-11-24 03:51:04,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2667986.6666666665, ans=0.125
2023-11-24 03:51:09,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2668053.3333333335, ans=0.0
2023-11-24 03:51:33,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2668186.6666666665, ans=0.125
2023-11-24 03:51:43,867 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.001e+01 8.468e+01 9.078e+01 9.580e+01 1.134e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 03:51:45,109 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3450, loss[loss=0.08627, simple_loss=0.121, pruned_loss=0.01882, audio_tagging_loss=0.006939, over 15383.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09179, pruned_loss=0.01341, audio_tagging_loss=0.008995, over 3056172.77 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:51:51,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2668253.3333333335, ans=0.0
2023-11-24 03:51:57,040 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400250
2023-11-24 03:52:03,786 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=6.34 vs. limit=12.0
2023-11-24 03:52:14,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2668386.6666666665, ans=0.125
2023-11-24 03:52:25,709 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.81 vs. limit=15.0
2023-11-24 03:52:30,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2668453.3333333335, ans=0.09899494936611666
2023-11-24 03:52:39,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2668520.0, ans=0.07
2023-11-24 03:52:47,002 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3500, loss[loss=0.04834, simple_loss=0.06295, pruned_loss=0.008582, audio_tagging_loss=0.008287, over 14669.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.0915, pruned_loss=0.01333, audio_tagging_loss=0.008956, over 3050026.09 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:52:59,639 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400300
2023-11-24 03:53:12,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2668720.0, ans=0.1
2023-11-24 03:53:16,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=10.20 vs. limit=15.0
2023-11-24 03:53:18,510 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:53:19,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2668720.0, ans=0.0
2023-11-24 03:53:47,629 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.784e+01 9.365e+01 1.022e+02 1.307e+02, threshold=1.873e+02, percent-clipped=0.0
2023-11-24 03:53:48,880 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3550, loss[loss=0.0662, simple_loss=0.0896, pruned_loss=0.01512, audio_tagging_loss=0.006282, over 14941.00 frames. ], tot_loss[loss=0.0683, simple_loss=0.09183, pruned_loss=0.0134, audio_tagging_loss=0.008986, over 3048208.33 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:53:51,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.29 vs. limit=12.0
2023-11-24 03:53:53,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2668920.0, ans=0.125
2023-11-24 03:54:02,690 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400350
2023-11-24 03:54:14,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2669053.3333333335, ans=0.0
2023-11-24 03:54:24,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2669053.3333333335, ans=0.125
2023-11-24 03:54:30,974 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.35 vs. limit=15.0
2023-11-24 03:54:43,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2669186.6666666665, ans=0.125
2023-11-24 03:54:48,791 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.65 vs. limit=15.0
2023-11-24 03:54:52,801 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3600, loss[loss=0.06404, simple_loss=0.08942, pruned_loss=0.01167, audio_tagging_loss=0.007657, over 16319.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09134, pruned_loss=0.01329, audio_tagging_loss=0.008955, over 3043935.95 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 03:55:02,600 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:55:04,851 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400400
2023-11-24 03:55:06,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2669320.0, ans=0.125
2023-11-24 03:55:16,378 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.72 vs. limit=15.0
2023-11-24 03:55:26,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=2669386.6666666665, ans=15.0
2023-11-24 03:55:33,223 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.65 vs. limit=15.0
2023-11-24 03:55:37,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2669453.3333333335, ans=0.125
2023-11-24 03:55:47,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2669520.0, ans=0.125
2023-11-24 03:55:52,274 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2669520.0, ans=0.125
2023-11-24 03:55:54,321 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.024e+01 8.332e+01 8.867e+01 9.623e+01 1.551e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-24 03:55:54,369 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3650, loss[loss=0.08086, simple_loss=0.1099, pruned_loss=0.01838, audio_tagging_loss=0.007521, over 14767.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09204, pruned_loss=0.01349, audio_tagging_loss=0.008939, over 3051232.07 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:56:06,534 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400450
2023-11-24 03:56:26,024 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.95 vs. limit=15.0
2023-11-24 03:56:35,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2669786.6666666665, ans=0.125
2023-11-24 03:56:44,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2669853.3333333335, ans=0.125
2023-11-24 03:56:51,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2669853.3333333335, ans=0.0
2023-11-24 03:56:55,253 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:56:56,229 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3700, loss[loss=0.06804, simple_loss=0.09203, pruned_loss=0.01501, audio_tagging_loss=0.00701, over 14509.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09224, pruned_loss=0.01347, audio_tagging_loss=0.008874, over 3054554.24 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:56:59,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.whiten.whitening_limit, batch_count=2669920.0, ans=12.0
2023-11-24 03:57:06,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2669920.0, ans=10.0
2023-11-24 03:57:10,531 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400500
2023-11-24 03:57:12,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2669986.6666666665, ans=0.0
2023-11-24 03:57:21,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2670053.3333333335, ans=0.125
2023-11-24 03:57:29,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2670053.3333333335, ans=0.125
2023-11-24 03:57:37,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2670120.0, ans=0.125
2023-11-24 03:57:45,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2670186.6666666665, ans=0.125
2023-11-24 03:57:56,290 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:58:00,096 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.336e+01 8.459e+01 9.233e+01 9.964e+01 1.289e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 03:58:00,141 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3750, loss[loss=0.07716, simple_loss=0.1108, pruned_loss=0.01509, audio_tagging_loss=0.006686, over 14237.00 frames. ], tot_loss[loss=0.0691, simple_loss=0.09316, pruned_loss=0.01371, audio_tagging_loss=0.008809, over 3049252.35 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:58:02,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2670253.3333333335, ans=0.125
2023-11-24 03:58:11,989 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400550
2023-11-24 03:58:14,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2670320.0, ans=0.0
2023-11-24 03:58:20,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2670320.0, ans=0.2
2023-11-24 03:58:24,077 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 03:58:30,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2670386.6666666665, ans=0.09899494936611666
2023-11-24 03:58:40,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.56 vs. limit=22.5
2023-11-24 03:58:41,604 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 03:59:01,178 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3800, loss[loss=0.07124, simple_loss=0.1013, pruned_loss=0.01313, audio_tagging_loss=0.007437, over 14921.00 frames. ], tot_loss[loss=0.06925, simple_loss=0.09319, pruned_loss=0.01375, audio_tagging_loss=0.008904, over 3044085.67 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 03:59:02,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2670586.6666666665, ans=0.2
2023-11-24 03:59:07,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2670586.6666666665, ans=0.125
2023-11-24 03:59:10,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2670586.6666666665, ans=0.2
2023-11-24 03:59:13,372 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400600
2023-11-24 03:59:33,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2670720.0, ans=0.0
2023-11-24 03:59:43,773 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.03 vs. limit=22.5
2023-11-24 03:59:44,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2670786.6666666665, ans=0.1
2023-11-24 04:00:03,039 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.597e+01 8.605e+01 9.126e+01 9.740e+01 1.269e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-24 04:00:03,084 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3850, loss[loss=0.04775, simple_loss=0.05946, pruned_loss=0.007766, audio_tagging_loss=0.01025, over 15328.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09366, pruned_loss=0.01369, audio_tagging_loss=0.008889, over 3044458.97 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:00:11,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.37 vs. limit=6.0
2023-11-24 04:00:16,292 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400650
2023-11-24 04:00:16,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2670986.6666666665, ans=0.0
2023-11-24 04:00:16,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2670986.6666666665, ans=0.125
2023-11-24 04:00:17,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2670986.6666666665, ans=0.0
2023-11-24 04:00:32,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2671053.3333333335, ans=0.1
2023-11-24 04:00:57,293 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0
2023-11-24 04:01:06,133 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3900, loss[loss=0.05515, simple_loss=0.06744, pruned_loss=0.01077, audio_tagging_loss=0.01066, over 14302.00 frames. ], tot_loss[loss=0.06935, simple_loss=0.09358, pruned_loss=0.01368, audio_tagging_loss=0.008879, over 3044294.98 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:01:16,736 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.61 vs. limit=15.0
2023-11-24 04:01:18,891 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400700
2023-11-24 04:01:21,906 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.53 vs. limit=15.0
2023-11-24 04:01:24,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2671320.0, ans=0.125
2023-11-24 04:02:08,916 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.467e+01 9.035e+01 9.770e+01 1.225e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 04:02:08,961 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 3950, loss[loss=0.05533, simple_loss=0.07458, pruned_loss=0.008953, audio_tagging_loss=0.009088, over 14703.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09198, pruned_loss=0.01339, audio_tagging_loss=0.009048, over 3044504.61 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:02:20,950 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400750
2023-11-24 04:02:23,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2671653.3333333335, ans=0.2
2023-11-24 04:02:24,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2671653.3333333335, ans=0.0
2023-11-24 04:02:45,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2671786.6666666665, ans=0.1
2023-11-24 04:02:54,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2671786.6666666665, ans=0.125
2023-11-24 04:03:00,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2671853.3333333335, ans=0.2
2023-11-24 04:03:02,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2671853.3333333335, ans=0.0
2023-11-24 04:03:03,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2671853.3333333335, ans=0.0
2023-11-24 04:03:10,954 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4000, loss[loss=0.07325, simple_loss=0.09496, pruned_loss=0.01399, audio_tagging_loss=0.01178, over 14743.00 frames. ], tot_loss[loss=0.06892, simple_loss=0.0926, pruned_loss=0.01355, audio_tagging_loss=0.00907, over 3045792.98 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:03:21,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2671920.0, ans=0.125
2023-11-24 04:03:24,143 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400800
2023-11-24 04:03:28,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2671986.6666666665, ans=0.125
2023-11-24 04:03:59,433 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.36 vs. limit=10.0
2023-11-24 04:04:03,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2672186.6666666665, ans=0.1
2023-11-24 04:04:12,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2672253.3333333335, ans=0.0
2023-11-24 04:04:13,888 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.813e+01 8.478e+01 9.170e+01 1.003e+02 1.153e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 04:04:13,939 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4050, loss[loss=0.05832, simple_loss=0.08552, pruned_loss=0.006705, audio_tagging_loss=0.008854, over 16321.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09348, pruned_loss=0.01372, audio_tagging_loss=0.009054, over 3048964.23 frames. ], batch size: 62, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:04:16,846 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:04:26,464 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400850
2023-11-24 04:04:30,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2672320.0, ans=0.0
2023-11-24 04:04:35,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2672320.0, ans=0.125
2023-11-24 04:04:41,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2672386.6666666665, ans=0.0
2023-11-24 04:05:08,210 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.51 vs. limit=15.0
2023-11-24 04:05:15,960 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4100, loss[loss=0.08352, simple_loss=0.1181, pruned_loss=0.01767, audio_tagging_loss=0.006813, over 15375.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09329, pruned_loss=0.01362, audio_tagging_loss=0.00904, over 3049514.44 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:05:28,560 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400900
2023-11-24 04:05:33,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2672653.3333333335, ans=0.0
2023-11-24 04:05:37,541 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.51 vs. limit=15.0
2023-11-24 04:05:38,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2672653.3333333335, ans=0.05
2023-11-24 04:06:04,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2672853.3333333335, ans=0.0
2023-11-24 04:06:16,260 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.93 vs. limit=15.0
2023-11-24 04:06:18,028 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4150, loss[loss=0.07441, simple_loss=0.1027, pruned_loss=0.01593, audio_tagging_loss=0.007123, over 15504.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09392, pruned_loss=0.01357, audio_tagging_loss=0.008827, over 3055335.66 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:06:18,882 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.30 vs. limit=22.5
2023-11-24 04:06:19,139 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.620e+01 8.718e+01 9.320e+01 1.030e+02 1.270e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 04:06:22,173 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.74 vs. limit=15.0
2023-11-24 04:06:31,268 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 400950
2023-11-24 04:06:47,900 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.30 vs. limit=15.0
2023-11-24 04:06:51,725 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.20 vs. limit=15.0
2023-11-24 04:07:02,451 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:07:04,350 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.27 vs. limit=15.0
2023-11-24 04:07:07,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2673186.6666666665, ans=15.0
2023-11-24 04:07:21,661 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4200, loss[loss=0.06363, simple_loss=0.08201, pruned_loss=0.01342, audio_tagging_loss=0.009206, over 15175.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09365, pruned_loss=0.01347, audio_tagging_loss=0.00883, over 3050452.87 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:07:21,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2673253.3333333335, ans=0.05
2023-11-24 04:07:24,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2673253.3333333335, ans=0.125
2023-11-24 04:07:34,260 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401000
2023-11-24 04:07:42,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2673320.0, ans=0.125
2023-11-24 04:07:44,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2673320.0, ans=0.0
2023-11-24 04:07:47,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.82 vs. limit=15.0
2023-11-24 04:07:55,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2673386.6666666665, ans=0.0
2023-11-24 04:07:57,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2673453.3333333335, ans=0.1
2023-11-24 04:08:00,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2673453.3333333335, ans=0.125
2023-11-24 04:08:22,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2673520.0, ans=0.125
2023-11-24 04:08:24,431 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4250, loss[loss=0.06651, simple_loss=0.09293, pruned_loss=0.01186, audio_tagging_loss=0.008189, over 16202.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09428, pruned_loss=0.01358, audio_tagging_loss=0.008692, over 3056383.13 frames. ], batch size: 59, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:08:25,552 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.323e+01 9.029e+01 9.665e+01 1.222e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 04:08:36,840 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401050
2023-11-24 04:08:59,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2673720.0, ans=0.04949747468305833
2023-11-24 04:09:26,470 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4300, loss[loss=0.0583, simple_loss=0.06605, pruned_loss=0.01297, audio_tagging_loss=0.0123, over 15898.00 frames. ], tot_loss[loss=0.06951, simple_loss=0.09446, pruned_loss=0.0136, audio_tagging_loss=0.00868, over 3064159.18 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:09:28,522 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.73 vs. limit=15.0
2023-11-24 04:09:38,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401100
2023-11-24 04:09:57,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2674053.3333333335, ans=0.125
2023-11-24 04:09:59,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2674053.3333333335, ans=0.07
2023-11-24 04:10:03,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2674120.0, ans=0.125
2023-11-24 04:10:17,557 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:10:18,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2674186.6666666665, ans=0.0
2023-11-24 04:10:25,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2674186.6666666665, ans=0.125
2023-11-24 04:10:28,555 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4350, loss[loss=0.06105, simple_loss=0.07052, pruned_loss=0.01269, audio_tagging_loss=0.0131, over 14486.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09372, pruned_loss=0.01352, audio_tagging_loss=0.008709, over 3062212.28 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:10:30,151 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.375e+01 8.839e+01 9.580e+01 1.285e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-24 04:10:41,786 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401150
2023-11-24 04:10:42,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2674320.0, ans=0.125
2023-11-24 04:10:46,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=6.70 vs. limit=12.0
2023-11-24 04:10:49,321 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.17 vs. limit=15.0
2023-11-24 04:10:50,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2674320.0, ans=0.0
2023-11-24 04:10:50,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2674320.0, ans=0.125
2023-11-24 04:10:51,820 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.05 vs. limit=15.0
2023-11-24 04:11:06,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2674453.3333333335, ans=0.0
2023-11-24 04:11:13,818 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.15 vs. limit=22.5
2023-11-24 04:11:31,221 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4400, loss[loss=0.0597, simple_loss=0.07722, pruned_loss=0.009974, audio_tagging_loss=0.01111, over 15177.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09329, pruned_loss=0.01365, audio_tagging_loss=0.008703, over 3057543.18 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:11:33,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2674586.6666666665, ans=0.0
2023-11-24 04:11:43,179 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401200
2023-11-24 04:11:50,424 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.81 vs. limit=10.0
2023-11-24 04:12:07,873 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2674786.6666666665, ans=0.2
2023-11-24 04:12:09,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2674786.6666666665, ans=0.125
2023-11-24 04:12:13,630 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.99 vs. limit=15.0
2023-11-24 04:12:19,671 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.28 vs. limit=15.0
2023-11-24 04:12:31,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2674853.3333333335, ans=0.0
2023-11-24 04:12:33,420 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4450, loss[loss=0.05577, simple_loss=0.07718, pruned_loss=0.007552, audio_tagging_loss=0.009631, over 15663.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09293, pruned_loss=0.01371, audio_tagging_loss=0.008662, over 3054572.31 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:12:34,498 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.511e+01 8.725e+01 9.329e+01 1.003e+02 1.264e+02, threshold=1.866e+02, percent-clipped=0.0
2023-11-24 04:12:37,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2674920.0, ans=0.125
2023-11-24 04:12:43,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2674920.0, ans=0.1
2023-11-24 04:12:45,913 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401250
2023-11-24 04:12:53,178 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.77 vs. limit=22.5
2023-11-24 04:12:55,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2674986.6666666665, ans=0.0
2023-11-24 04:13:09,583 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.02 vs. limit=15.0
2023-11-24 04:13:16,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2675120.0, ans=0.1
2023-11-24 04:13:35,768 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4500, loss[loss=0.06029, simple_loss=0.07835, pruned_loss=0.01086, audio_tagging_loss=0.01026, over 15098.00 frames. ], tot_loss[loss=0.06924, simple_loss=0.09382, pruned_loss=0.01374, audio_tagging_loss=0.008589, over 3053135.35 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:13:36,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2675253.3333333335, ans=0.125
2023-11-24 04:13:38,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.69 vs. limit=15.0
2023-11-24 04:13:39,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2675253.3333333335, ans=0.125
2023-11-24 04:13:49,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401300
2023-11-24 04:14:09,374 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:14:16,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2675453.3333333335, ans=0.125
2023-11-24 04:14:21,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2675453.3333333335, ans=0.125
2023-11-24 04:14:25,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2675520.0, ans=0.125
2023-11-24 04:14:38,768 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4550, loss[loss=0.08005, simple_loss=0.09935, pruned_loss=0.02016, audio_tagging_loss=0.01022, over 15282.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09358, pruned_loss=0.01377, audio_tagging_loss=0.008591, over 3049017.25 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:14:39,937 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.652e+01 8.786e+01 9.235e+01 9.923e+01 1.429e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 04:14:40,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2675586.6666666665, ans=0.125
2023-11-24 04:14:50,691 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401350
2023-11-24 04:14:57,003 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.93 vs. limit=15.0
2023-11-24 04:15:24,808 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:15:40,268 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4600, loss[loss=0.0538, simple_loss=0.06827, pruned_loss=0.008875, audio_tagging_loss=0.01079, over 14597.00 frames. ], tot_loss[loss=0.06872, simple_loss=0.09242, pruned_loss=0.01371, audio_tagging_loss=0.008802, over 3050036.14 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:15:52,228 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401400
2023-11-24 04:16:06,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2676053.3333333335, ans=0.2
2023-11-24 04:16:23,862 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.51 vs. limit=15.0
2023-11-24 04:16:24,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=6.90 vs. limit=15.0
2023-11-24 04:16:31,124 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.01 vs. limit=12.0
2023-11-24 04:16:35,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2676186.6666666665, ans=0.125
2023-11-24 04:16:35,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2676186.6666666665, ans=0.125
2023-11-24 04:16:41,497 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.21 vs. limit=12.0
2023-11-24 04:16:42,192 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4650, loss[loss=0.06891, simple_loss=0.09279, pruned_loss=0.01578, audio_tagging_loss=0.006733, over 14897.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09174, pruned_loss=0.01364, audio_tagging_loss=0.008965, over 3052072.10 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:16:43,858 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.931e+01 8.311e+01 8.909e+01 9.483e+01 1.432e+02, threshold=1.782e+02, percent-clipped=0.0
2023-11-24 04:16:45,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2676253.3333333335, ans=0.025
2023-11-24 04:16:54,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2676320.0, ans=0.1
2023-11-24 04:16:55,739 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401450
2023-11-24 04:16:59,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2676320.0, ans=0.125
2023-11-24 04:17:03,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2676320.0, ans=0.125
2023-11-24 04:17:03,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2676320.0, ans=0.0
2023-11-24 04:17:07,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2676386.6666666665, ans=0.0
2023-11-24 04:17:08,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2676386.6666666665, ans=0.0
2023-11-24 04:17:11,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2676386.6666666665, ans=0.0
2023-11-24 04:17:14,348 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:17:14,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2676386.6666666665, ans=0.125
2023-11-24 04:17:27,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2676453.3333333335, ans=0.0
2023-11-24 04:17:38,332 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.81 vs. limit=15.0
2023-11-24 04:17:45,192 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4700, loss[loss=0.06566, simple_loss=0.09075, pruned_loss=0.01322, audio_tagging_loss=0.007065, over 15092.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09204, pruned_loss=0.01362, audio_tagging_loss=0.008997, over 3049746.58 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:17:56,440 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.09 vs. limit=15.0
2023-11-24 04:17:57,276 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401500
2023-11-24 04:18:14,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2676720.0, ans=0.125
2023-11-24 04:18:42,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2676853.3333333335, ans=0.125
2023-11-24 04:18:44,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2676853.3333333335, ans=0.125
2023-11-24 04:18:46,880 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4750, loss[loss=0.07047, simple_loss=0.09058, pruned_loss=0.01654, audio_tagging_loss=0.008638, over 14731.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09223, pruned_loss=0.01375, audio_tagging_loss=0.008965, over 3046175.85 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:18:47,987 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.095e+01 8.351e+01 9.004e+01 9.816e+01 1.181e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 04:18:49,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2676920.0, ans=10.0
2023-11-24 04:18:49,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2676920.0, ans=0.0
2023-11-24 04:18:58,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401550
2023-11-24 04:19:09,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2676986.6666666665, ans=0.1
2023-11-24 04:19:16,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2677053.3333333335, ans=0.125
2023-11-24 04:19:30,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2677120.0, ans=0.125
2023-11-24 04:19:44,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2677186.6666666665, ans=0.1
2023-11-24 04:19:44,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2677186.6666666665, ans=0.1
2023-11-24 04:19:47,785 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4800, loss[loss=0.05808, simple_loss=0.08079, pruned_loss=0.008696, audio_tagging_loss=0.008984, over 15368.00 frames. ], tot_loss[loss=0.06905, simple_loss=0.09249, pruned_loss=0.0138, audio_tagging_loss=0.009008, over 3048556.26 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:20:01,491 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401600
2023-11-24 04:20:38,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.96 vs. limit=15.0
2023-11-24 04:20:52,269 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4850, loss[loss=0.06788, simple_loss=0.08731, pruned_loss=0.01449, audio_tagging_loss=0.009733, over 15416.00 frames. ], tot_loss[loss=0.06941, simple_loss=0.09281, pruned_loss=0.01384, audio_tagging_loss=0.009156, over 3052802.11 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:20:53,334 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.256e+01 8.375e+01 9.054e+01 9.821e+01 1.202e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-24 04:20:56,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2677586.6666666665, ans=0.0
2023-11-24 04:20:58,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2677586.6666666665, ans=0.1
2023-11-24 04:20:58,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2677586.6666666665, ans=0.125
2023-11-24 04:21:04,066 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401650
2023-11-24 04:21:22,391 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.44 vs. limit=15.0
2023-11-24 04:21:24,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2677720.0, ans=0.5
2023-11-24 04:21:25,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2677720.0, ans=0.125
2023-11-24 04:21:27,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2677786.6666666665, ans=0.125
2023-11-24 04:21:28,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2677786.6666666665, ans=0.125
2023-11-24 04:21:30,226 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.32 vs. limit=15.0
2023-11-24 04:21:35,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2677786.6666666665, ans=0.04949747468305833
2023-11-24 04:21:43,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2677853.3333333335, ans=0.125
2023-11-24 04:21:53,716 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4900, loss[loss=0.06044, simple_loss=0.07455, pruned_loss=0.009983, audio_tagging_loss=0.01319, over 15850.00 frames. ], tot_loss[loss=0.06936, simple_loss=0.09297, pruned_loss=0.01377, audio_tagging_loss=0.009109, over 3050727.91 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:22:05,639 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401700
2023-11-24 04:22:10,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2677986.6666666665, ans=0.125
2023-11-24 04:22:21,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2678053.3333333335, ans=0.125
2023-11-24 04:22:21,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2678053.3333333335, ans=0.125
2023-11-24 04:22:47,570 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.22 vs. limit=15.0
2023-11-24 04:22:55,414 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 4950, loss[loss=0.05356, simple_loss=0.0649, pruned_loss=0.009452, audio_tagging_loss=0.01166, over 13487.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09211, pruned_loss=0.01348, audio_tagging_loss=0.008977, over 3046043.53 frames. ], batch size: 53, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:22:56,552 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.172e+01 8.560e+01 8.976e+01 9.753e+01 1.389e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-24 04:23:00,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2678253.3333333335, ans=0.125
2023-11-24 04:23:08,552 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401750
2023-11-24 04:23:19,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2678320.0, ans=0.0
2023-11-24 04:23:31,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2678453.3333333335, ans=0.0
2023-11-24 04:23:33,182 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:23:40,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2678453.3333333335, ans=0.125
2023-11-24 04:23:57,848 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5000, loss[loss=0.05601, simple_loss=0.07249, pruned_loss=0.01134, audio_tagging_loss=0.008429, over 14469.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09155, pruned_loss=0.01338, audio_tagging_loss=0.00882, over 3040807.23 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:24:10,923 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401800
2023-11-24 04:24:18,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2678653.3333333335, ans=0.125
2023-11-24 04:24:27,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2678720.0, ans=0.125
2023-11-24 04:24:30,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2678720.0, ans=0.125
2023-11-24 04:24:56,787 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.12 vs. limit=15.0
2023-11-24 04:25:00,942 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5050, loss[loss=0.05258, simple_loss=0.07414, pruned_loss=0.008759, audio_tagging_loss=0.006758, over 16521.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09129, pruned_loss=0.01342, audio_tagging_loss=0.008828, over 3036042.42 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:25:02,047 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.608e+01 9.238e+01 9.938e+01 4.414e+02, threshold=1.848e+02, percent-clipped=1.0
2023-11-24 04:25:07,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2678920.0, ans=0.0
2023-11-24 04:25:09,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2678920.0, ans=0.125
2023-11-24 04:25:10,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2678920.0, ans=0.125
2023-11-24 04:25:11,020 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.91 vs. limit=6.0
2023-11-24 04:25:12,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401850
2023-11-24 04:25:15,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.15 vs. limit=6.0
2023-11-24 04:25:19,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2678986.6666666665, ans=0.1
2023-11-24 04:25:46,619 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.31 vs. limit=22.5
2023-11-24 04:25:49,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=13.01 vs. limit=15.0
2023-11-24 04:26:02,692 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5100, loss[loss=0.07142, simple_loss=0.09194, pruned_loss=0.0156, audio_tagging_loss=0.009842, over 16894.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09106, pruned_loss=0.01356, audio_tagging_loss=0.008889, over 3046998.25 frames. ], batch size: 63, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:26:16,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401900
2023-11-24 04:26:29,880 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.72 vs. limit=12.0
2023-11-24 04:26:34,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2679386.6666666665, ans=0.2
2023-11-24 04:26:39,362 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.66 vs. limit=22.5
2023-11-24 04:26:49,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2679453.3333333335, ans=0.1
2023-11-24 04:26:51,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2679520.0, ans=0.0
2023-11-24 04:26:53,492 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.59 vs. limit=15.0
2023-11-24 04:26:57,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2679520.0, ans=0.1
2023-11-24 04:27:05,524 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5150, loss[loss=0.07572, simple_loss=0.09946, pruned_loss=0.01761, audio_tagging_loss=0.008386, over 15208.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09106, pruned_loss=0.01351, audio_tagging_loss=0.008865, over 3043061.51 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:27:07,946 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.413e+01 8.884e+01 9.698e+01 1.235e+02, threshold=1.777e+02, percent-clipped=0.0
2023-11-24 04:27:10,688 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:27:14,880 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:27:18,906 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 401950
2023-11-24 04:27:46,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_na.min_abs, batch_count=2679786.6666666665, ans=0.02
2023-11-24 04:28:08,610 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5200, loss[loss=0.0855, simple_loss=0.1189, pruned_loss=0.01982, audio_tagging_loss=0.006213, over 14567.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09003, pruned_loss=0.01321, audio_tagging_loss=0.008925, over 3039830.60 frames. ], batch size: 54, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:28:20,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402000
2023-11-24 04:28:39,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2680053.3333333335, ans=0.0
2023-11-24 04:28:39,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2680053.3333333335, ans=0.125
2023-11-24 04:28:54,198 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:28:54,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2680120.0, ans=0.0
2023-11-24 04:29:02,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2680186.6666666665, ans=0.0
2023-11-24 04:29:07,817 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:29:09,997 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5250, loss[loss=0.1041, simple_loss=0.1434, pruned_loss=0.02789, audio_tagging_loss=0.004525, over 15359.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09077, pruned_loss=0.01324, audio_tagging_loss=0.008927, over 3036879.75 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 32.0
2023-11-24 04:29:12,361 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.432e+01 8.228e+01 8.916e+01 9.618e+01 1.278e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-24 04:29:16,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2680253.3333333335, ans=0.125
2023-11-24 04:29:18,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=2680253.3333333335, ans=10.0
2023-11-24 04:29:21,912 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402050
2023-11-24 04:29:31,576 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2680320.0, ans=0.1
2023-11-24 04:29:34,312 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.91 vs. limit=15.0
2023-11-24 04:29:36,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2680386.6666666665, ans=0.2
2023-11-24 04:29:39,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2680386.6666666665, ans=0.2
2023-11-24 04:29:40,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2680386.6666666665, ans=0.125
2023-11-24 04:29:45,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2680386.6666666665, ans=0.125
2023-11-24 04:30:00,688 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.05 vs. limit=15.0
2023-11-24 04:30:12,381 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5300, loss[loss=0.05073, simple_loss=0.06945, pruned_loss=0.009344, audio_tagging_loss=0.006657, over 14644.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09071, pruned_loss=0.01318, audio_tagging_loss=0.008847, over 3036316.26 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:30:25,285 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402100
2023-11-24 04:30:50,130 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.78 vs. limit=15.0
2023-11-24 04:31:00,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2680786.6666666665, ans=0.0
2023-11-24 04:31:15,252 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5350, loss[loss=0.06208, simple_loss=0.08861, pruned_loss=0.008194, audio_tagging_loss=0.009584, over 15392.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09148, pruned_loss=0.01326, audio_tagging_loss=0.008955, over 3042279.04 frames. ], batch size: 55, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:31:18,680 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.481e+01 8.724e+01 9.285e+01 9.851e+01 1.330e+02, threshold=1.857e+02, percent-clipped=0.0
2023-11-24 04:31:27,815 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402150
2023-11-24 04:31:31,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2680986.6666666665, ans=0.125
2023-11-24 04:31:54,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2681120.0, ans=0.1
2023-11-24 04:32:07,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.65 vs. limit=22.5
2023-11-24 04:32:16,955 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5400, loss[loss=0.05801, simple_loss=0.08151, pruned_loss=0.00948, audio_tagging_loss=0.007779, over 15008.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.0924, pruned_loss=0.0135, audio_tagging_loss=0.008969, over 3055725.13 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:32:24,575 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.85 vs. limit=12.0
2023-11-24 04:32:28,762 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402200
2023-11-24 04:32:28,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2681320.0, ans=0.0
2023-11-24 04:32:29,201 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.34 vs. limit=12.0
2023-11-24 04:33:00,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2681453.3333333335, ans=0.2
2023-11-24 04:33:18,584 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5450, loss[loss=0.05741, simple_loss=0.07715, pruned_loss=0.00863, audio_tagging_loss=0.0102, over 14933.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.0928, pruned_loss=0.01359, audio_tagging_loss=0.008989, over 3052235.99 frames. ], batch size: 56, lr: 2.00e-03, grad_scale: 8.0
2023-11-24 04:33:18,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2681586.6666666665, ans=0.0
2023-11-24 04:33:24,396 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.146e+01 8.363e+01 9.005e+01 9.739e+01 1.241e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 04:33:31,645 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402250
2023-11-24 04:33:33,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.61 vs. limit=15.0
2023-11-24 04:33:47,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2681720.0, ans=0.125
2023-11-24 04:33:51,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2681720.0, ans=0.125
2023-11-24 04:34:02,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2681786.6666666665, ans=0.0
2023-11-24 04:34:02,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2681786.6666666665, ans=0.125
2023-11-24 04:34:12,286 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.74 vs. limit=22.5
2023-11-24 04:34:20,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.11 vs. limit=15.0
2023-11-24 04:34:21,309 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5500, loss[loss=0.07577, simple_loss=0.1007, pruned_loss=0.01809, audio_tagging_loss=0.007319, over 14741.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09255, pruned_loss=0.01358, audio_tagging_loss=0.008947, over 3054105.47 frames. ], batch size: 58, lr: 2.00e-03, grad_scale: 8.0
2023-11-24 04:34:33,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402300
2023-11-24 04:34:49,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=15.0
2023-11-24 04:35:03,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2682120.0, ans=0.035
2023-11-24 04:35:08,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2682120.0, ans=0.0
2023-11-24 04:35:10,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.25 vs. limit=22.5
2023-11-24 04:35:22,501 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5550, loss[loss=0.08102, simple_loss=0.1095, pruned_loss=0.01714, audio_tagging_loss=0.009127, over 15992.00 frames. ], tot_loss[loss=0.06874, simple_loss=0.09229, pruned_loss=0.01351, audio_tagging_loss=0.009092, over 3051799.23 frames. ], batch size: 60, lr: 2.00e-03, grad_scale: 8.0
2023-11-24 04:35:27,155 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.398e+01 8.518e+01 9.178e+01 1.002e+02 1.571e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 04:35:34,794 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402350
2023-11-24 04:35:51,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2682386.6666666665, ans=0.125
2023-11-24 04:36:09,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2682453.3333333335, ans=0.125
2023-11-24 04:36:24,125 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5600, loss[loss=0.06371, simple_loss=0.08381, pruned_loss=0.009652, audio_tagging_loss=0.01215, over 15352.00 frames. ], tot_loss[loss=0.06898, simple_loss=0.09258, pruned_loss=0.01355, audio_tagging_loss=0.009145, over 3054225.29 frames. ], batch size: 57, lr: 2.00e-03, grad_scale: 16.0
2023-11-24 04:36:28,776 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0
2023-11-24 04:36:37,242 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402400
2023-11-24 04:36:46,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2682653.3333333335, ans=0.0
2023-11-24 04:37:08,387 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:37:09,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2682786.6666666665, ans=0.0
2023-11-24 04:37:21,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2682853.3333333335, ans=0.0
2023-11-24 04:37:27,407 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5650, loss[loss=0.06979, simple_loss=0.09233, pruned_loss=0.0143, audio_tagging_loss=0.009332, over 15713.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.0923, pruned_loss=0.01335, audio_tagging_loss=0.009208, over 3054971.47 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:37:33,010 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.272e+01 8.812e+01 9.562e+01 1.491e+02, threshold=1.762e+02, percent-clipped=0.0
2023-11-24 04:37:36,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2682920.0, ans=0.025
2023-11-24 04:37:38,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2682920.0, ans=0.2
2023-11-24 04:37:40,648 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402450
2023-11-24 04:37:52,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2683053.3333333335, ans=0.0
2023-11-24 04:38:04,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2683120.0, ans=0.1
2023-11-24 04:38:21,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2683186.6666666665, ans=0.125
2023-11-24 04:38:26,024 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.21 vs. limit=6.0
2023-11-24 04:38:31,384 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5700, loss[loss=0.05563, simple_loss=0.07604, pruned_loss=0.007914, audio_tagging_loss=0.0097, over 15089.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.0921, pruned_loss=0.01347, audio_tagging_loss=0.009116, over 3055992.29 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:38:31,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2683253.3333333335, ans=0.0
2023-11-24 04:38:40,297 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.18 vs. limit=22.5
2023-11-24 04:38:43,583 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402500
2023-11-24 04:38:43,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2683320.0, ans=0.125
2023-11-24 04:39:04,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2683386.6666666665, ans=0.04949747468305833
2023-11-24 04:39:16,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2683453.3333333335, ans=0.125
2023-11-24 04:39:25,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2683520.0, ans=0.125
2023-11-24 04:39:28,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2683520.0, ans=0.0
2023-11-24 04:39:31,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2683520.0, ans=0.125
2023-11-24 04:39:33,797 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5750, loss[loss=0.06718, simple_loss=0.09198, pruned_loss=0.01216, audio_tagging_loss=0.009032, over 15927.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09081, pruned_loss=0.01329, audio_tagging_loss=0.009194, over 3056510.40 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:39:39,184 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.509e+01 8.554e+01 9.219e+01 1.000e+02 1.500e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 04:39:46,861 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0
2023-11-24 04:39:47,587 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402550
2023-11-24 04:40:09,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2683720.0, ans=0.0
2023-11-24 04:40:21,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2683786.6666666665, ans=0.125
2023-11-24 04:40:21,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.78 vs. limit=15.0
2023-11-24 04:40:24,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2683853.3333333335, ans=0.125
2023-11-24 04:40:37,111 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5800, loss[loss=0.07752, simple_loss=0.1059, pruned_loss=0.01631, audio_tagging_loss=0.008253, over 14292.00 frames. ], tot_loss[loss=0.06834, simple_loss=0.09151, pruned_loss=0.01349, audio_tagging_loss=0.009103, over 3054490.84 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:40:42,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2683920.0, ans=0.0
2023-11-24 04:40:43,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2683920.0, ans=0.125
2023-11-24 04:40:49,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402600
2023-11-24 04:40:57,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2683986.6666666665, ans=0.125
2023-11-24 04:41:06,742 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.21 vs. limit=15.0
2023-11-24 04:41:14,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2684120.0, ans=0.035
2023-11-24 04:41:28,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2684186.6666666665, ans=0.125
2023-11-24 04:41:30,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2684186.6666666665, ans=0.07
2023-11-24 04:41:37,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2684186.6666666665, ans=0.05
2023-11-24 04:41:39,347 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5850, loss[loss=0.0789, simple_loss=0.1166, pruned_loss=0.01285, audio_tagging_loss=0.007744, over 16075.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09163, pruned_loss=0.01359, audio_tagging_loss=0.009028, over 3050400.08 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:41:39,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2684253.3333333335, ans=0.125
2023-11-24 04:41:44,139 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.922e+01 8.321e+01 9.009e+01 9.790e+01 1.176e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-24 04:41:45,053 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.46 vs. limit=15.0
2023-11-24 04:41:49,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2684253.3333333335, ans=0.1
2023-11-24 04:41:51,392 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402650
2023-11-24 04:42:04,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2684386.6666666665, ans=0.0
2023-11-24 04:42:20,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2684453.3333333335, ans=0.2
2023-11-24 04:42:24,001 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.95 vs. limit=12.0
2023-11-24 04:42:41,235 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5900, loss[loss=0.05648, simple_loss=0.07614, pruned_loss=0.00905, audio_tagging_loss=0.009354, over 15480.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09163, pruned_loss=0.01361, audio_tagging_loss=0.009, over 3046031.35 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:42:41,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2684586.6666666665, ans=0.025
2023-11-24 04:42:41,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2684586.6666666665, ans=0.125
2023-11-24 04:42:43,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2684586.6666666665, ans=0.125
2023-11-24 04:42:54,510 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402700
2023-11-24 04:42:55,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2684653.3333333335, ans=0.125
2023-11-24 04:43:01,272 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.50 vs. limit=5.0
2023-11-24 04:43:43,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2684920.0, ans=0.0
2023-11-24 04:43:44,350 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 5950, loss[loss=0.08336, simple_loss=0.1189, pruned_loss=0.01841, audio_tagging_loss=0.005479, over 16203.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09156, pruned_loss=0.01361, audio_tagging_loss=0.008941, over 3051109.01 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:43:49,677 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.371e+01 8.328e+01 9.136e+01 9.988e+01 1.334e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 04:43:49,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2684920.0, ans=0.1
2023-11-24 04:43:51,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2684920.0, ans=0.125
2023-11-24 04:43:53,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2684920.0, ans=0.125
2023-11-24 04:43:54,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2684920.0, ans=0.125
2023-11-24 04:43:55,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2684986.6666666665, ans=0.125
2023-11-24 04:43:56,996 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402750
2023-11-24 04:44:04,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2684986.6666666665, ans=0.0
2023-11-24 04:44:17,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2685053.3333333335, ans=0.025
2023-11-24 04:44:28,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2685120.0, ans=0.125
2023-11-24 04:44:44,039 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.69 vs. limit=22.5
2023-11-24 04:44:45,395 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.05 vs. limit=15.0
2023-11-24 04:44:45,812 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6000, loss[loss=0.07166, simple_loss=0.09129, pruned_loss=0.01811, audio_tagging_loss=0.007904, over 15629.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09124, pruned_loss=0.01363, audio_tagging_loss=0.008948, over 3047674.34 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:44:45,814 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 04:45:04,430 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.4113, 3.2869, 3.7449, 3.1068, 3.7163, 3.5308, 3.3621, 3.3882],
       device='cuda:0')
2023-11-24 04:45:26,689 INFO [train_asr.py:1253] (0/4) Epoch 34, validation: loss=0.05772, simple_loss=0.05082, pruned_loss=0.005023, audio_tagging_loss=0.02728, over 4681554.00 frames. 
2023-11-24 04:45:26,690 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 04:45:27,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2685253.3333333335, ans=0.2
2023-11-24 04:45:33,583 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:45:40,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402800
2023-11-24 04:45:43,506 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.28 vs. limit=12.0
2023-11-24 04:46:02,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2685386.6666666665, ans=0.125
2023-11-24 04:46:11,972 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 04:46:29,983 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6050, loss[loss=0.05826, simple_loss=0.0744, pruned_loss=0.01068, audio_tagging_loss=0.01038, over 15928.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09109, pruned_loss=0.01358, audio_tagging_loss=0.008878, over 3049271.24 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:46:35,237 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.324e+01 9.101e+01 9.713e+01 1.173e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 04:46:36,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2685586.6666666665, ans=0.2
2023-11-24 04:46:42,487 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402850
2023-11-24 04:46:47,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2685653.3333333335, ans=10.0
2023-11-24 04:47:10,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2685786.6666666665, ans=0.125
2023-11-24 04:47:17,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2685786.6666666665, ans=0.2
2023-11-24 04:47:19,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2685853.3333333335, ans=0.0
2023-11-24 04:47:30,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.11 vs. limit=22.5
2023-11-24 04:47:31,920 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6100, loss[loss=0.0646, simple_loss=0.08806, pruned_loss=0.0136, audio_tagging_loss=0.006973, over 14612.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09082, pruned_loss=0.01363, audio_tagging_loss=0.008954, over 3043916.24 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:47:43,834 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402900
2023-11-24 04:47:52,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2685986.6666666665, ans=0.0
2023-11-24 04:47:59,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2686053.3333333335, ans=0.0
2023-11-24 04:48:24,130 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:48:33,319 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6150, loss[loss=0.06426, simple_loss=0.08834, pruned_loss=0.01269, audio_tagging_loss=0.0074, over 14655.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09091, pruned_loss=0.01352, audio_tagging_loss=0.009013, over 3037957.61 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:48:38,030 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.509e+01 8.476e+01 9.051e+01 9.662e+01 1.133e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 04:48:43,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2686253.3333333335, ans=0.125
2023-11-24 04:48:46,516 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 402950
2023-11-24 04:48:49,484 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.23 vs. limit=6.0
2023-11-24 04:48:55,057 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:48:55,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2686320.0, ans=0.07
2023-11-24 04:48:59,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2686386.6666666665, ans=0.1
2023-11-24 04:49:15,779 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.91 vs. limit=15.0
2023-11-24 04:49:31,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2686520.0, ans=0.125
2023-11-24 04:49:36,262 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6200, loss[loss=0.05844, simple_loss=0.07262, pruned_loss=0.01324, audio_tagging_loss=0.008891, over 14918.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.0899, pruned_loss=0.01331, audio_tagging_loss=0.009058, over 3041937.41 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:49:49,217 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403000
2023-11-24 04:49:59,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2686653.3333333335, ans=0.125
2023-11-24 04:50:16,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2686786.6666666665, ans=0.09899494936611666
2023-11-24 04:50:30,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2686853.3333333335, ans=0.1
2023-11-24 04:50:39,706 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6250, loss[loss=0.06385, simple_loss=0.08257, pruned_loss=0.01282, audio_tagging_loss=0.009743, over 15235.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09047, pruned_loss=0.01338, audio_tagging_loss=0.009093, over 3041699.69 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:50:42,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2686920.0, ans=0.0
2023-11-24 04:50:45,706 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.297e+01 8.512e+01 9.019e+01 9.887e+01 1.410e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-24 04:50:51,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403050
2023-11-24 04:51:13,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2687053.3333333335, ans=0.125
2023-11-24 04:51:35,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2687186.6666666665, ans=0.2
2023-11-24 04:51:40,189 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:51:41,207 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6300, loss[loss=0.05101, simple_loss=0.06826, pruned_loss=0.007377, audio_tagging_loss=0.009504, over 15945.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09095, pruned_loss=0.01334, audio_tagging_loss=0.009194, over 3037906.75 frames. ], batch size: 62, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:51:43,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2687253.3333333335, ans=10.0
2023-11-24 04:51:47,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2687253.3333333335, ans=0.0
2023-11-24 04:51:53,706 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403100
2023-11-24 04:52:43,755 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6350, loss[loss=0.06041, simple_loss=0.07845, pruned_loss=0.01315, audio_tagging_loss=0.008035, over 15573.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09055, pruned_loss=0.01343, audio_tagging_loss=0.009261, over 3038624.35 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:52:47,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2687586.6666666665, ans=0.125
2023-11-24 04:52:50,181 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.641e+01 8.312e+01 8.895e+01 9.882e+01 1.344e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-24 04:52:56,177 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403150
2023-11-24 04:53:34,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2687853.3333333335, ans=0.0
2023-11-24 04:53:34,369 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:53:46,056 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6400, loss[loss=0.05925, simple_loss=0.08272, pruned_loss=0.007766, audio_tagging_loss=0.01013, over 16129.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09021, pruned_loss=0.0133, audio_tagging_loss=0.009329, over 3035582.16 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:53:50,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.27 vs. limit=10.0
2023-11-24 04:53:56,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2687986.6666666665, ans=0.125
2023-11-24 04:53:57,943 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403200
2023-11-24 04:54:00,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2687986.6666666665, ans=0.125
2023-11-24 04:54:00,788 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:54:16,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.73 vs. limit=22.5
2023-11-24 04:54:33,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2688120.0, ans=0.0
2023-11-24 04:54:38,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2688186.6666666665, ans=0.0
2023-11-24 04:54:47,667 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6450, loss[loss=0.06296, simple_loss=0.0805, pruned_loss=0.01212, audio_tagging_loss=0.01059, over 14034.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09023, pruned_loss=0.01324, audio_tagging_loss=0.009361, over 3034480.76 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:54:47,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2688253.3333333335, ans=0.0
2023-11-24 04:54:52,987 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.99 vs. limit=6.0
2023-11-24 04:54:53,478 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.531e+01 8.209e+01 8.936e+01 9.468e+01 2.072e+02, threshold=1.787e+02, percent-clipped=1.0
2023-11-24 04:54:54,378 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.09 vs. limit=6.0
2023-11-24 04:54:55,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2688253.3333333335, ans=0.0
2023-11-24 04:54:57,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2688253.3333333335, ans=0.0
2023-11-24 04:55:00,073 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403250
2023-11-24 04:55:00,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2688320.0, ans=0.1
2023-11-24 04:55:03,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2688320.0, ans=0.0
2023-11-24 04:55:07,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2688320.0, ans=0.125
2023-11-24 04:55:11,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2688386.6666666665, ans=0.0
2023-11-24 04:55:13,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2688386.6666666665, ans=0.0
2023-11-24 04:55:14,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2688386.6666666665, ans=15.0
2023-11-24 04:55:15,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2688386.6666666665, ans=0.125
2023-11-24 04:55:19,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.09 vs. limit=15.0
2023-11-24 04:55:24,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2688453.3333333335, ans=0.0
2023-11-24 04:55:24,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2688453.3333333335, ans=0.0
2023-11-24 04:55:30,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2688453.3333333335, ans=0.125
2023-11-24 04:55:32,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2688453.3333333335, ans=0.125
2023-11-24 04:55:49,660 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6500, loss[loss=0.05943, simple_loss=0.07356, pruned_loss=0.01081, audio_tagging_loss=0.01184, over 14824.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09031, pruned_loss=0.01319, audio_tagging_loss=0.009292, over 3034231.19 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 04:56:02,798 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403300
2023-11-24 04:56:18,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2688720.0, ans=0.0
2023-11-24 04:56:19,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2688720.0, ans=0.0
2023-11-24 04:56:38,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2688853.3333333335, ans=0.125
2023-11-24 04:56:47,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2688853.3333333335, ans=0.0
2023-11-24 04:56:49,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2688853.3333333335, ans=0.125
2023-11-24 04:56:52,703 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6550, loss[loss=0.06856, simple_loss=0.0947, pruned_loss=0.01391, audio_tagging_loss=0.007304, over 15369.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09081, pruned_loss=0.01321, audio_tagging_loss=0.009136, over 3039481.64 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:56:59,702 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.455e+01 8.633e+01 9.315e+01 9.907e+01 1.273e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 04:57:01,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2688920.0, ans=10.0
2023-11-24 04:57:02,902 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 04:57:05,044 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403350
2023-11-24 04:57:25,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2689053.3333333335, ans=0.0
2023-11-24 04:57:29,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2689120.0, ans=0.0
2023-11-24 04:57:34,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2689120.0, ans=0.0
2023-11-24 04:57:36,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2689120.0, ans=0.0
2023-11-24 04:57:54,992 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6600, loss[loss=0.06653, simple_loss=0.09123, pruned_loss=0.01167, audio_tagging_loss=0.00925, over 15648.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.0906, pruned_loss=0.01318, audio_tagging_loss=0.009018, over 3032798.01 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:57:58,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2689253.3333333335, ans=0.1
2023-11-24 04:58:06,789 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403400
2023-11-24 04:58:06,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2689320.0, ans=0.125
2023-11-24 04:58:21,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.17 vs. limit=12.0
2023-11-24 04:58:37,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2689453.3333333335, ans=0.125
2023-11-24 04:58:56,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2689586.6666666665, ans=0.125
2023-11-24 04:58:57,519 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6650, loss[loss=0.07252, simple_loss=0.1014, pruned_loss=0.01229, audio_tagging_loss=0.009521, over 14398.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09054, pruned_loss=0.01313, audio_tagging_loss=0.009021, over 3032582.55 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 04:59:05,515 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.526e+01 8.351e+01 9.016e+01 9.664e+01 1.302e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 04:59:11,195 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403450
2023-11-24 04:59:21,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.83 vs. limit=6.0
2023-11-24 04:59:22,369 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.42 vs. limit=15.0
2023-11-24 04:59:44,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2689786.6666666665, ans=0.0
2023-11-24 04:59:49,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2689853.3333333335, ans=0.125
2023-11-24 04:59:52,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2689853.3333333335, ans=0.125
2023-11-24 04:59:53,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2689853.3333333335, ans=0.1
2023-11-24 04:59:54,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2689853.3333333335, ans=0.0
2023-11-24 05:00:00,595 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6700, loss[loss=0.0784, simple_loss=0.1106, pruned_loss=0.01386, audio_tagging_loss=0.00923, over 15726.00 frames. ], tot_loss[loss=0.06727, simple_loss=0.09025, pruned_loss=0.01318, audio_tagging_loss=0.008965, over 3036316.42 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:00:07,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2689920.0, ans=0.0
2023-11-24 05:00:10,962 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.65 vs. limit=15.0
2023-11-24 05:00:12,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403500
2023-11-24 05:00:23,248 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.70 vs. limit=12.0
2023-11-24 05:00:25,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2690053.3333333335, ans=0.125
2023-11-24 05:00:26,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2690053.3333333335, ans=0.0
2023-11-24 05:00:36,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2690120.0, ans=0.1
2023-11-24 05:00:43,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2690120.0, ans=0.2
2023-11-24 05:01:01,985 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6750, loss[loss=0.07515, simple_loss=0.1077, pruned_loss=0.01617, audio_tagging_loss=0.005154, over 14961.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09065, pruned_loss=0.01337, audio_tagging_loss=0.008953, over 3041143.39 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:01:09,704 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.923e+01 8.241e+01 8.716e+01 9.523e+01 2.058e+02, threshold=1.743e+02, percent-clipped=1.0
2023-11-24 05:01:13,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2690320.0, ans=0.0
2023-11-24 05:01:14,521 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403550
2023-11-24 05:02:04,022 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6800, loss[loss=0.05516, simple_loss=0.07532, pruned_loss=0.007521, audio_tagging_loss=0.00998, over 14393.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.091, pruned_loss=0.01337, audio_tagging_loss=0.008918, over 3038742.83 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:02:09,553 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2690586.6666666665, ans=0.0
2023-11-24 05:02:13,923 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.19 vs. limit=15.0
2023-11-24 05:02:17,545 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403600
2023-11-24 05:02:25,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2690653.3333333335, ans=0.0
2023-11-24 05:02:40,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2690786.6666666665, ans=0.125
2023-11-24 05:02:46,321 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.33 vs. limit=15.0
2023-11-24 05:03:05,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2690853.3333333335, ans=0.2
2023-11-24 05:03:07,295 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6850, loss[loss=0.05245, simple_loss=0.06845, pruned_loss=0.007607, audio_tagging_loss=0.01062, over 14711.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09113, pruned_loss=0.01343, audio_tagging_loss=0.009002, over 3037927.39 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:03:15,569 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.436e+01 8.418e+01 8.943e+01 9.846e+01 1.358e+02, threshold=1.789e+02, percent-clipped=0.0
2023-11-24 05:03:19,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403650
2023-11-24 05:03:19,712 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.76 vs. limit=22.5
2023-11-24 05:03:28,709 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2690986.6666666665, ans=0.125
2023-11-24 05:03:38,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2691053.3333333335, ans=0.04949747468305833
2023-11-24 05:03:39,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=21.13 vs. limit=22.5
2023-11-24 05:03:42,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2691120.0, ans=0.125
2023-11-24 05:04:08,464 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6900, loss[loss=0.07651, simple_loss=0.1045, pruned_loss=0.01744, audio_tagging_loss=0.006801, over 14847.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09127, pruned_loss=0.01346, audio_tagging_loss=0.00903, over 3036825.68 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:04:18,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2691253.3333333335, ans=0.125
2023-11-24 05:04:20,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403700
2023-11-24 05:04:22,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2691320.0, ans=0.0
2023-11-24 05:04:45,772 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.61 vs. limit=15.0
2023-11-24 05:04:55,802 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:04:56,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2691453.3333333335, ans=0.2
2023-11-24 05:05:10,073 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 6950, loss[loss=0.07448, simple_loss=0.1037, pruned_loss=0.01687, audio_tagging_loss=0.005748, over 16297.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09115, pruned_loss=0.0134, audio_tagging_loss=0.008942, over 3039062.85 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:05:19,311 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.551e+01 9.218e+01 9.903e+01 1.445e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 05:05:23,672 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403750
2023-11-24 05:05:27,856 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:05:32,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2691653.3333333335, ans=0.0
2023-11-24 05:06:01,952 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.22 vs. limit=15.0
2023-11-24 05:06:13,669 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7000, loss[loss=0.06213, simple_loss=0.07901, pruned_loss=0.009327, audio_tagging_loss=0.0133, over 15010.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09072, pruned_loss=0.01322, audio_tagging_loss=0.008955, over 3043674.29 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:06:13,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2691920.0, ans=0.1
2023-11-24 05:06:25,499 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403800
2023-11-24 05:06:33,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2691986.6666666665, ans=0.1
2023-11-24 05:07:09,764 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:07:09,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2692186.6666666665, ans=0.125
2023-11-24 05:07:14,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2692253.3333333335, ans=0.0
2023-11-24 05:07:15,353 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7050, loss[loss=0.07954, simple_loss=0.1082, pruned_loss=0.01633, audio_tagging_loss=0.009113, over 14888.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09048, pruned_loss=0.01323, audio_tagging_loss=0.009056, over 3034439.32 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:07:23,616 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.613e+01 8.305e+01 8.871e+01 9.954e+01 1.371e+02, threshold=1.774e+02, percent-clipped=0.0
2023-11-24 05:07:27,378 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403850
2023-11-24 05:07:34,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2692320.0, ans=0.125
2023-11-24 05:07:36,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2692320.0, ans=22.5
2023-11-24 05:07:49,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.56 vs. limit=12.0
2023-11-24 05:08:08,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2692520.0, ans=0.0
2023-11-24 05:08:15,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2692586.6666666665, ans=0.1
2023-11-24 05:08:16,599 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7100, loss[loss=0.08466, simple_loss=0.1082, pruned_loss=0.01832, audio_tagging_loss=0.01226, over 15471.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.0902, pruned_loss=0.01326, audio_tagging_loss=0.009217, over 3032048.09 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:08:19,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2692586.6666666665, ans=0.125
2023-11-24 05:08:30,257 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403900
2023-11-24 05:08:31,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2692653.3333333335, ans=0.125
2023-11-24 05:08:34,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2692653.3333333335, ans=0.1
2023-11-24 05:08:40,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2692653.3333333335, ans=0.125
2023-11-24 05:08:51,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2692720.0, ans=0.2
2023-11-24 05:08:53,184 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.86 vs. limit=15.0
2023-11-24 05:09:00,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2692786.6666666665, ans=0.125
2023-11-24 05:09:06,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2692853.3333333335, ans=0.1
2023-11-24 05:09:20,747 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7150, loss[loss=0.05259, simple_loss=0.06847, pruned_loss=0.009256, audio_tagging_loss=0.0091, over 16414.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09081, pruned_loss=0.01338, audio_tagging_loss=0.00919, over 3038278.48 frames. ], batch size: 64, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:09:29,598 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.190e+01 8.812e+01 9.268e+01 1.001e+02 1.464e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 05:09:30,154 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.82 vs. limit=15.0
2023-11-24 05:09:33,374 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 403950
2023-11-24 05:09:46,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2693053.3333333335, ans=0.125
2023-11-24 05:09:49,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.52 vs. limit=15.0
2023-11-24 05:09:56,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2693120.0, ans=0.1
2023-11-24 05:10:02,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2693120.0, ans=0.1
2023-11-24 05:10:18,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.86 vs. limit=22.5
2023-11-24 05:10:22,852 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7200, loss[loss=0.06921, simple_loss=0.0848, pruned_loss=0.01434, audio_tagging_loss=0.01247, over 13972.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09085, pruned_loss=0.01327, audio_tagging_loss=0.009175, over 3031279.08 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:10:24,949 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.66 vs. limit=5.0
2023-11-24 05:10:25,686 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.00 vs. limit=15.0
2023-11-24 05:10:30,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2693253.3333333335, ans=0.1
2023-11-24 05:10:30,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2693253.3333333335, ans=0.1
2023-11-24 05:10:34,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404000
2023-11-24 05:10:36,114 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-404000.pt
2023-11-24 05:10:39,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2693320.0, ans=0.125
2023-11-24 05:10:44,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2693320.0, ans=0.0
2023-11-24 05:10:49,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2693386.6666666665, ans=0.0
2023-11-24 05:10:56,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2693386.6666666665, ans=0.04949747468305833
2023-11-24 05:11:00,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2693386.6666666665, ans=0.0
2023-11-24 05:11:15,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2693520.0, ans=0.0
2023-11-24 05:11:18,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2693520.0, ans=0.125
2023-11-24 05:11:22,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2693520.0, ans=0.2
2023-11-24 05:11:28,077 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7250, loss[loss=0.05951, simple_loss=0.08894, pruned_loss=0.007049, audio_tagging_loss=0.007997, over 13778.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09013, pruned_loss=0.01316, audio_tagging_loss=0.009294, over 3035588.74 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:11:36,163 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.320e+01 8.579e+01 9.139e+01 1.008e+02 1.154e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 05:11:40,499 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404050
2023-11-24 05:11:47,111 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.73 vs. limit=15.0
2023-11-24 05:11:58,389 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.87 vs. limit=15.0
2023-11-24 05:12:28,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2693853.3333333335, ans=0.2
2023-11-24 05:12:30,885 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7300, loss[loss=0.0754, simple_loss=0.09791, pruned_loss=0.017, audio_tagging_loss=0.009447, over 15230.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09005, pruned_loss=0.01307, audio_tagging_loss=0.009207, over 3038452.57 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:12:41,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2693920.0, ans=0.125
2023-11-24 05:12:41,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2693920.0, ans=0.1
2023-11-24 05:12:43,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404100
2023-11-24 05:12:49,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2693986.6666666665, ans=0.125
2023-11-24 05:13:07,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2694120.0, ans=0.0
2023-11-24 05:13:31,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2694186.6666666665, ans=0.125
2023-11-24 05:13:31,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2694186.6666666665, ans=0.0
2023-11-24 05:13:32,972 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7350, loss[loss=0.07509, simple_loss=0.1038, pruned_loss=0.01434, audio_tagging_loss=0.008863, over 16831.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09075, pruned_loss=0.01324, audio_tagging_loss=0.009118, over 3048415.24 frames. ], batch size: 64, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:13:43,685 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.288e+01 8.435e+01 9.060e+01 9.696e+01 1.244e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 05:13:45,022 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404150
2023-11-24 05:13:49,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2694320.0, ans=0.125
2023-11-24 05:13:58,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2694386.6666666665, ans=0.125
2023-11-24 05:14:02,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2694386.6666666665, ans=0.125
2023-11-24 05:14:03,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2694386.6666666665, ans=0.125
2023-11-24 05:14:34,634 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7400, loss[loss=0.08151, simple_loss=0.1098, pruned_loss=0.02015, audio_tagging_loss=0.006486, over 14735.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09095, pruned_loss=0.01337, audio_tagging_loss=0.008895, over 3039879.81 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:14:47,301 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404200
2023-11-24 05:15:02,557 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:15:09,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2694720.0, ans=0.125
2023-11-24 05:15:25,805 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.97 vs. limit=10.0
2023-11-24 05:15:27,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2694853.3333333335, ans=0.2
2023-11-24 05:15:31,625 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.32 vs. limit=15.0
2023-11-24 05:15:37,380 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7450, loss[loss=0.0672, simple_loss=0.08779, pruned_loss=0.01473, audio_tagging_loss=0.008579, over 16308.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09094, pruned_loss=0.01352, audio_tagging_loss=0.008761, over 3035750.05 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:15:41,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2694920.0, ans=0.0
2023-11-24 05:15:48,987 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.292e+01 8.446e+01 9.170e+01 9.700e+01 1.341e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 05:15:50,879 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404250
2023-11-24 05:15:50,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2694986.6666666665, ans=0.1
2023-11-24 05:16:15,338 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.83 vs. limit=15.0
2023-11-24 05:16:28,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2695186.6666666665, ans=0.0
2023-11-24 05:16:40,531 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7500, loss[loss=0.06687, simple_loss=0.08301, pruned_loss=0.01425, audio_tagging_loss=0.01111, over 14519.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09156, pruned_loss=0.01337, audio_tagging_loss=0.008698, over 3038227.79 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:16:48,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2695253.3333333335, ans=0.025
2023-11-24 05:16:51,866 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.38 vs. limit=15.0
2023-11-24 05:16:52,366 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404300
2023-11-24 05:16:55,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2695320.0, ans=0.1
2023-11-24 05:17:06,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2695386.6666666665, ans=0.025
2023-11-24 05:17:34,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2695520.0, ans=0.125
2023-11-24 05:17:41,563 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7550, loss[loss=0.08222, simple_loss=0.111, pruned_loss=0.0204, audio_tagging_loss=0.006303, over 14297.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09088, pruned_loss=0.01326, audio_tagging_loss=0.008803, over 3045813.86 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:17:47,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2695586.6666666665, ans=0.1
2023-11-24 05:17:51,950 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.237e+01 8.670e+01 9.167e+01 9.885e+01 1.310e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 05:17:52,361 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:17:53,357 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404350
2023-11-24 05:18:01,973 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.21 vs. limit=15.0
2023-11-24 05:18:08,771 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.72 vs. limit=22.5
2023-11-24 05:18:24,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2695786.6666666665, ans=0.125
2023-11-24 05:18:26,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2695786.6666666665, ans=0.1
2023-11-24 05:18:43,189 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7600, loss[loss=0.08206, simple_loss=0.1157, pruned_loss=0.01627, audio_tagging_loss=0.007937, over 15302.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.08998, pruned_loss=0.01301, audio_tagging_loss=0.008784, over 3047875.52 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:18:43,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2695920.0, ans=0.125
2023-11-24 05:18:56,262 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404400
2023-11-24 05:19:12,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2696053.3333333335, ans=0.0
2023-11-24 05:19:42,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2696186.6666666665, ans=0.0
2023-11-24 05:19:45,985 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7650, loss[loss=0.06786, simple_loss=0.1025, pruned_loss=0.01115, audio_tagging_loss=0.005468, over 15082.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09031, pruned_loss=0.01298, audio_tagging_loss=0.008802, over 3048640.41 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:19:57,217 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.788e+01 8.265e+01 8.997e+01 9.695e+01 1.208e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 05:19:58,565 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404450
2023-11-24 05:20:11,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2696386.6666666665, ans=0.1
2023-11-24 05:20:12,337 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.50 vs. limit=22.5
2023-11-24 05:20:47,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.46 vs. limit=22.5
2023-11-24 05:20:48,146 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7700, loss[loss=0.05533, simple_loss=0.07424, pruned_loss=0.00983, audio_tagging_loss=0.008382, over 14346.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09066, pruned_loss=0.01311, audio_tagging_loss=0.00878, over 3055659.32 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:20:53,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2696586.6666666665, ans=0.0
2023-11-24 05:20:54,959 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.53 vs. limit=8.0
2023-11-24 05:21:00,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404500
2023-11-24 05:21:01,745 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.02 vs. limit=22.5
2023-11-24 05:21:23,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2696720.0, ans=0.125
2023-11-24 05:21:50,154 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7750, loss[loss=0.06882, simple_loss=0.09643, pruned_loss=0.01408, audio_tagging_loss=0.006527, over 15992.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.09034, pruned_loss=0.01316, audio_tagging_loss=0.008792, over 3053999.28 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:21:51,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2696920.0, ans=0.0
2023-11-24 05:22:01,916 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.961e+01 8.444e+01 9.119e+01 9.673e+01 1.144e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 05:22:03,344 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404550
2023-11-24 05:22:11,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2696986.6666666665, ans=0.1
2023-11-24 05:22:23,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2697053.3333333335, ans=0.125
2023-11-24 05:22:40,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2697186.6666666665, ans=0.125
2023-11-24 05:22:53,785 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7800, loss[loss=0.05762, simple_loss=0.07334, pruned_loss=0.01351, audio_tagging_loss=0.007433, over 14798.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09046, pruned_loss=0.01323, audio_tagging_loss=0.008873, over 3052986.91 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:22:55,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2697253.3333333335, ans=0.125
2023-11-24 05:23:05,800 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404600
2023-11-24 05:23:48,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2697520.0, ans=0.0
2023-11-24 05:23:56,291 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7850, loss[loss=0.06762, simple_loss=0.08524, pruned_loss=0.01487, audio_tagging_loss=0.01013, over 15253.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.0913, pruned_loss=0.01326, audio_tagging_loss=0.008899, over 3055247.30 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:24:02,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2697586.6666666665, ans=0.125
2023-11-24 05:24:07,516 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.405e+01 8.451e+01 9.106e+01 9.579e+01 1.135e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 05:24:08,845 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404650
2023-11-24 05:24:24,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2697720.0, ans=0.0
2023-11-24 05:24:30,967 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.77 vs. limit=10.0
2023-11-24 05:24:36,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2697786.6666666665, ans=0.125
2023-11-24 05:24:37,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2697786.6666666665, ans=0.125
2023-11-24 05:24:44,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.95 vs. limit=10.0
2023-11-24 05:24:49,337 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.60 vs. limit=15.0
2023-11-24 05:24:51,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2697853.3333333335, ans=0.2
2023-11-24 05:24:52,880 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.89 vs. limit=6.0
2023-11-24 05:24:56,321 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.51 vs. limit=10.0
2023-11-24 05:24:57,997 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7900, loss[loss=0.07516, simple_loss=0.1007, pruned_loss=0.01562, audio_tagging_loss=0.009215, over 15642.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.0915, pruned_loss=0.01314, audio_tagging_loss=0.009018, over 3059837.49 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:25:11,502 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404700
2023-11-24 05:25:33,599 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:25:37,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2698120.0, ans=0.1
2023-11-24 05:25:40,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2698120.0, ans=0.125
2023-11-24 05:25:43,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2698120.0, ans=0.035
2023-11-24 05:25:50,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2698186.6666666665, ans=0.025
2023-11-24 05:25:50,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2698186.6666666665, ans=0.125
2023-11-24 05:26:01,544 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 7950, loss[loss=0.06871, simple_loss=0.09714, pruned_loss=0.0129, audio_tagging_loss=0.00724, over 15171.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.0908, pruned_loss=0.01327, audio_tagging_loss=0.009185, over 3060613.06 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:26:12,224 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.773e+01 8.346e+01 8.927e+01 9.659e+01 1.276e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-24 05:26:13,545 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404750
2023-11-24 05:26:15,946 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:27:03,772 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8000, loss[loss=0.04826, simple_loss=0.06108, pruned_loss=0.007257, audio_tagging_loss=0.01047, over 16028.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09086, pruned_loss=0.01327, audio_tagging_loss=0.009205, over 3057623.73 frames. ], batch size: 62, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:27:16,454 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404800
2023-11-24 05:27:22,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2698653.3333333335, ans=0.125
2023-11-24 05:27:25,545 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.92 vs. limit=15.0
2023-11-24 05:27:26,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2698653.3333333335, ans=0.125
2023-11-24 05:27:39,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2698720.0, ans=0.07
2023-11-24 05:28:00,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2698853.3333333335, ans=0.125
2023-11-24 05:28:06,019 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8050, loss[loss=0.06303, simple_loss=0.07657, pruned_loss=0.01505, audio_tagging_loss=0.009697, over 15569.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09124, pruned_loss=0.01345, audio_tagging_loss=0.009284, over 3049887.75 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:28:15,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2698920.0, ans=0.0
2023-11-24 05:28:17,289 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.661e+01 8.621e+01 9.093e+01 9.694e+01 1.272e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 05:28:19,192 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404850
2023-11-24 05:28:20,960 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.61 vs. limit=10.0
2023-11-24 05:28:30,913 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=13.19 vs. limit=15.0
2023-11-24 05:28:31,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2699053.3333333335, ans=0.125
2023-11-24 05:29:08,251 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0
2023-11-24 05:29:08,661 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8100, loss[loss=0.07798, simple_loss=0.1147, pruned_loss=0.01612, audio_tagging_loss=0.004498, over 15036.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09105, pruned_loss=0.01354, audio_tagging_loss=0.009192, over 3045290.50 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:29:15,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2699253.3333333335, ans=0.125
2023-11-24 05:29:21,387 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404900
2023-11-24 05:29:23,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2699320.0, ans=0.0
2023-11-24 05:29:43,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.37 vs. limit=22.5
2023-11-24 05:29:58,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2699520.0, ans=0.1
2023-11-24 05:30:08,046 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.16 vs. limit=22.5
2023-11-24 05:30:10,913 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8150, loss[loss=0.07016, simple_loss=0.08781, pruned_loss=0.01489, audio_tagging_loss=0.01137, over 15124.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09118, pruned_loss=0.01349, audio_tagging_loss=0.009031, over 3048635.33 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:30:13,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2699586.6666666665, ans=0.2
2023-11-24 05:30:22,679 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.155e+01 8.549e+01 9.218e+01 9.858e+01 1.332e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 05:30:22,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 404950
2023-11-24 05:30:34,374 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.62 vs. limit=22.5
2023-11-24 05:30:35,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2699720.0, ans=0.0
2023-11-24 05:30:48,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2699786.6666666665, ans=0.125
2023-11-24 05:30:54,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2699786.6666666665, ans=0.125
2023-11-24 05:31:01,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2699853.3333333335, ans=0.0
2023-11-24 05:31:11,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2699920.0, ans=0.0
2023-11-24 05:31:12,013 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8200, loss[loss=0.0402, simple_loss=0.05199, pruned_loss=0.005543, audio_tagging_loss=0.008664, over 14405.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09126, pruned_loss=0.01342, audio_tagging_loss=0.008905, over 3045632.19 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:31:12,069 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:31:17,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2699920.0, ans=0.125
2023-11-24 05:31:18,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2699920.0, ans=0.125
2023-11-24 05:31:24,907 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405000
2023-11-24 05:31:42,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2700053.3333333335, ans=0.125
2023-11-24 05:31:47,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2700053.3333333335, ans=0.125
2023-11-24 05:31:52,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2700120.0, ans=0.2
2023-11-24 05:31:59,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2700120.0, ans=0.125
2023-11-24 05:31:59,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2700120.0, ans=0.125
2023-11-24 05:32:04,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.whiten.whitening_limit, batch_count=2700186.6666666665, ans=12.0
2023-11-24 05:32:13,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2700253.3333333335, ans=0.2
2023-11-24 05:32:14,745 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8250, loss[loss=0.08427, simple_loss=0.114, pruned_loss=0.02022, audio_tagging_loss=0.00704, over 14780.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09144, pruned_loss=0.01342, audio_tagging_loss=0.008854, over 3050106.07 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:32:27,223 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405050
2023-11-24 05:32:28,209 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.520e+01 8.524e+01 9.156e+01 9.753e+01 1.627e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 05:32:28,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2700320.0, ans=0.1
2023-11-24 05:32:30,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2700320.0, ans=0.125
2023-11-24 05:32:44,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.66 vs. limit=15.0
2023-11-24 05:32:52,392 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:33:10,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2700520.0, ans=0.0
2023-11-24 05:33:14,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2700520.0, ans=0.125
2023-11-24 05:33:16,935 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8300, loss[loss=0.05174, simple_loss=0.07546, pruned_loss=0.006027, audio_tagging_loss=0.007986, over 14750.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.0906, pruned_loss=0.01311, audio_tagging_loss=0.008842, over 3055420.93 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:33:17,672 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.23 vs. limit=15.0
2023-11-24 05:33:18,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2700586.6666666665, ans=0.0
2023-11-24 05:33:26,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn2.whiten.whitening_limit, batch_count=2700586.6666666665, ans=22.5
2023-11-24 05:33:28,931 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405100
2023-11-24 05:33:42,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2700720.0, ans=0.125
2023-11-24 05:33:49,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2700720.0, ans=0.125
2023-11-24 05:33:54,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2700786.6666666665, ans=0.125
2023-11-24 05:34:07,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2700853.3333333335, ans=0.07
2023-11-24 05:34:18,257 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8350, loss[loss=0.05589, simple_loss=0.07768, pruned_loss=0.009268, audio_tagging_loss=0.007788, over 14763.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09141, pruned_loss=0.01328, audio_tagging_loss=0.008821, over 3050958.53 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 8.0
2023-11-24 05:34:25,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2700920.0, ans=0.125
2023-11-24 05:34:30,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405150
2023-11-24 05:34:31,207 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.980e+01 8.407e+01 9.166e+01 9.742e+01 1.258e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 05:34:49,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2701053.3333333335, ans=0.07
2023-11-24 05:35:06,722 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.80 vs. limit=22.5
2023-11-24 05:35:15,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2701186.6666666665, ans=0.2
2023-11-24 05:35:19,466 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8400, loss[loss=0.07152, simple_loss=0.09198, pruned_loss=0.01455, audio_tagging_loss=0.01098, over 16026.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09134, pruned_loss=0.0133, audio_tagging_loss=0.008769, over 3051443.40 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:35:19,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2701253.3333333335, ans=0.0
2023-11-24 05:35:32,226 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405200
2023-11-24 05:35:37,942 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.02 vs. limit=10.0
2023-11-24 05:35:41,290 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.22 vs. limit=15.0
2023-11-24 05:35:42,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2701320.0, ans=0.125
2023-11-24 05:35:48,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2701386.6666666665, ans=0.125
2023-11-24 05:35:56,517 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.60 vs. limit=15.0
2023-11-24 05:36:02,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2701453.3333333335, ans=0.2
2023-11-24 05:36:04,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2701453.3333333335, ans=0.125
2023-11-24 05:36:21,620 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8450, loss[loss=0.06839, simple_loss=0.08763, pruned_loss=0.01682, audio_tagging_loss=0.007761, over 14502.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09089, pruned_loss=0.01314, audio_tagging_loss=0.00888, over 3044643.44 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:36:31,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2701586.6666666665, ans=0.125
2023-11-24 05:36:33,797 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405250
2023-11-24 05:36:34,770 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.098e+01 8.659e+01 9.489e+01 1.219e+02, threshold=1.732e+02, percent-clipped=0.0
2023-11-24 05:36:39,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2701653.3333333335, ans=0.0
2023-11-24 05:36:44,763 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.20 vs. limit=22.5
2023-11-24 05:36:49,556 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.10 vs. limit=15.0
2023-11-24 05:37:06,275 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.85 vs. limit=22.5
2023-11-24 05:37:09,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2701786.6666666665, ans=0.125
2023-11-24 05:37:13,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2701853.3333333335, ans=0.1
2023-11-24 05:37:18,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2701853.3333333335, ans=0.125
2023-11-24 05:37:23,348 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8500, loss[loss=0.0737, simple_loss=0.1071, pruned_loss=0.01308, audio_tagging_loss=0.007087, over 14832.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09228, pruned_loss=0.01338, audio_tagging_loss=0.008791, over 3046909.03 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:37:35,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405300
2023-11-24 05:37:37,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2701986.6666666665, ans=0.125
2023-11-24 05:37:37,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2701986.6666666665, ans=0.2
2023-11-24 05:37:51,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2702053.3333333335, ans=0.025
2023-11-24 05:38:14,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.95 vs. limit=15.0
2023-11-24 05:38:24,812 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8550, loss[loss=0.06196, simple_loss=0.07924, pruned_loss=0.00971, audio_tagging_loss=0.01263, over 15888.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09303, pruned_loss=0.01355, audio_tagging_loss=0.008793, over 3047908.83 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:38:39,291 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405350
2023-11-24 05:38:40,297 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.730e+01 8.556e+01 9.186e+01 9.838e+01 1.237e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-24 05:38:40,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2702320.0, ans=0.2
2023-11-24 05:38:53,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2702386.6666666665, ans=0.125
2023-11-24 05:38:57,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2702386.6666666665, ans=0.1
2023-11-24 05:39:04,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2702453.3333333335, ans=0.125
2023-11-24 05:39:09,766 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0
2023-11-24 05:39:21,358 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2702520.0, ans=0.0
2023-11-24 05:39:29,214 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8600, loss[loss=0.07916, simple_loss=0.1075, pruned_loss=0.01611, audio_tagging_loss=0.009277, over 14937.00 frames. ], tot_loss[loss=0.06866, simple_loss=0.09251, pruned_loss=0.01351, audio_tagging_loss=0.008899, over 3042183.27 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:39:36,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2702586.6666666665, ans=0.07
2023-11-24 05:39:40,320 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.68 vs. limit=10.0
2023-11-24 05:39:41,053 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405400
2023-11-24 05:39:59,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2702720.0, ans=0.0
2023-11-24 05:40:03,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2702720.0, ans=0.2
2023-11-24 05:40:11,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2702786.6666666665, ans=0.1
2023-11-24 05:40:22,573 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.88 vs. limit=15.0
2023-11-24 05:40:29,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2702853.3333333335, ans=0.125
2023-11-24 05:40:31,247 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8650, loss[loss=0.05068, simple_loss=0.06083, pruned_loss=0.007991, audio_tagging_loss=0.01227, over 15478.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09257, pruned_loss=0.01341, audio_tagging_loss=0.008952, over 3041011.21 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:40:35,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2702920.0, ans=0.125
2023-11-24 05:40:38,852 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.23 vs. limit=15.0
2023-11-24 05:40:43,343 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405450
2023-11-24 05:40:44,410 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.097e+01 8.340e+01 8.964e+01 9.438e+01 1.251e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 05:40:49,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2702986.6666666665, ans=0.0
2023-11-24 05:41:33,411 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8700, loss[loss=0.05216, simple_loss=0.06557, pruned_loss=0.01114, audio_tagging_loss=0.00824, over 15194.00 frames. ], tot_loss[loss=0.069, simple_loss=0.09303, pruned_loss=0.01354, audio_tagging_loss=0.00894, over 3042242.35 frames. ], batch size: 61, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:41:46,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405500
2023-11-24 05:42:02,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2703386.6666666665, ans=0.0
2023-11-24 05:42:25,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2703520.0, ans=0.125
2023-11-24 05:42:25,836 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.53 vs. limit=6.0
2023-11-24 05:42:32,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2703520.0, ans=0.125
2023-11-24 05:42:36,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2703586.6666666665, ans=0.125
2023-11-24 05:42:37,531 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8750, loss[loss=0.07179, simple_loss=0.09683, pruned_loss=0.01479, audio_tagging_loss=0.008584, over 15232.00 frames. ], tot_loss[loss=0.06938, simple_loss=0.09335, pruned_loss=0.01369, audio_tagging_loss=0.009017, over 3037033.04 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:42:40,417 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.69 vs. limit=15.0
2023-11-24 05:42:41,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2703586.6666666665, ans=0.125
2023-11-24 05:42:46,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2703586.6666666665, ans=0.125
2023-11-24 05:42:50,138 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405550
2023-11-24 05:42:51,164 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.109e+01 8.699e+01 9.632e+01 1.055e+02 1.285e+02, threshold=1.926e+02, percent-clipped=0.0
2023-11-24 05:43:00,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2703720.0, ans=0.125
2023-11-24 05:43:07,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=12.65 vs. limit=15.0
2023-11-24 05:43:18,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2703786.6666666665, ans=0.04949747468305833
2023-11-24 05:43:39,337 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8800, loss[loss=0.0633, simple_loss=0.08971, pruned_loss=0.009469, audio_tagging_loss=0.008974, over 15522.00 frames. ], tot_loss[loss=0.06908, simple_loss=0.09264, pruned_loss=0.01357, audio_tagging_loss=0.009192, over 3037210.66 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:43:51,390 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405600
2023-11-24 05:44:05,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2704053.3333333335, ans=0.2
2023-11-24 05:44:08,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2704053.3333333335, ans=0.1
2023-11-24 05:44:20,463 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.06 vs. limit=22.5
2023-11-24 05:44:30,119 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.78 vs. limit=22.5
2023-11-24 05:44:41,163 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8850, loss[loss=0.07808, simple_loss=0.1173, pruned_loss=0.01428, audio_tagging_loss=0.005163, over 14498.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09195, pruned_loss=0.01328, audio_tagging_loss=0.009206, over 3040939.19 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:44:53,078 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:44:54,400 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405650
2023-11-24 05:44:55,470 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.409e+01 8.541e+01 8.982e+01 9.482e+01 1.517e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-24 05:44:55,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2704320.0, ans=0.125
2023-11-24 05:45:32,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2704520.0, ans=0.125
2023-11-24 05:45:33,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2704520.0, ans=0.125
2023-11-24 05:45:34,034 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.26 vs. limit=15.0
2023-11-24 05:45:39,524 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:45:44,008 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8900, loss[loss=0.0796, simple_loss=0.1139, pruned_loss=0.01243, audio_tagging_loss=0.01022, over 15691.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09328, pruned_loss=0.01352, audio_tagging_loss=0.008928, over 3041125.17 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:45:56,492 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405700
2023-11-24 05:46:11,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2704720.0, ans=0.0
2023-11-24 05:46:13,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2704720.0, ans=0.0
2023-11-24 05:46:14,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2704720.0, ans=0.1
2023-11-24 05:46:23,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2704786.6666666665, ans=0.1
2023-11-24 05:46:28,032 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.87 vs. limit=15.0
2023-11-24 05:46:37,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2704853.3333333335, ans=0.04949747468305833
2023-11-24 05:46:42,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2704853.3333333335, ans=0.2
2023-11-24 05:46:45,644 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 8950, loss[loss=0.07504, simple_loss=0.1067, pruned_loss=0.01607, audio_tagging_loss=0.00562, over 14880.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09275, pruned_loss=0.01328, audio_tagging_loss=0.008801, over 3040505.71 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:46:58,051 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405750
2023-11-24 05:46:59,185 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.112e+01 8.729e+01 9.252e+01 9.908e+01 1.235e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 05:47:38,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2705186.6666666665, ans=0.0
2023-11-24 05:47:47,501 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9000, loss[loss=0.06983, simple_loss=0.09228, pruned_loss=0.01643, audio_tagging_loss=0.007259, over 15551.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09297, pruned_loss=0.01346, audio_tagging_loss=0.008798, over 3047363.24 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:47:47,504 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 05:48:21,057 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.0993, 2.5523, 5.0142, 2.9586], device='cuda:0')
2023-11-24 05:48:27,862 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.0210, 3.8841, 4.8626, 4.4349], device='cuda:0')
2023-11-24 05:48:30,296 INFO [train_asr.py:1253] (0/4) Epoch 34, validation: loss=0.05898, simple_loss=0.05084, pruned_loss=0.005097, audio_tagging_loss=0.02846, over 4681554.00 frames. 
2023-11-24 05:48:30,297 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 05:48:31,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2705253.3333333335, ans=0.125
2023-11-24 05:48:32,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2705253.3333333335, ans=0.0
2023-11-24 05:48:38,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2705253.3333333335, ans=0.1
2023-11-24 05:48:38,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2705253.3333333335, ans=0.1
2023-11-24 05:48:42,191 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405800
2023-11-24 05:48:57,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2705386.6666666665, ans=0.2
2023-11-24 05:49:02,631 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=14.73 vs. limit=22.5
2023-11-24 05:49:03,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2705386.6666666665, ans=0.125
2023-11-24 05:49:03,915 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.84 vs. limit=10.0
2023-11-24 05:49:13,225 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=8.34 vs. limit=12.0
2023-11-24 05:49:32,031 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9050, loss[loss=0.07269, simple_loss=0.1024, pruned_loss=0.01442, audio_tagging_loss=0.007041, over 14230.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09219, pruned_loss=0.01331, audio_tagging_loss=0.008787, over 3043823.90 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:49:44,531 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405850
2023-11-24 05:49:46,750 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.178e+01 8.606e+01 9.162e+01 9.789e+01 1.610e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 05:50:09,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2705786.6666666665, ans=0.2
2023-11-24 05:50:34,570 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9100, loss[loss=0.06473, simple_loss=0.09007, pruned_loss=0.01219, audio_tagging_loss=0.007509, over 15404.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09161, pruned_loss=0.01329, audio_tagging_loss=0.008763, over 3045343.00 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:50:42,473 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:50:44,004 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.32 vs. limit=15.0
2023-11-24 05:50:47,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405900
2023-11-24 05:50:56,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2705986.6666666665, ans=0.125
2023-11-24 05:51:37,132 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9150, loss[loss=0.0841, simple_loss=0.1194, pruned_loss=0.01658, audio_tagging_loss=0.007824, over 16549.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09225, pruned_loss=0.01348, audio_tagging_loss=0.008726, over 3049921.62 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 05:51:42,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2706253.3333333335, ans=0.1
2023-11-24 05:51:49,181 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 405950
2023-11-24 05:51:51,501 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.335e+01 8.866e+01 9.361e+01 1.025e+02 1.380e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-24 05:52:04,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2706386.6666666665, ans=0.125
2023-11-24 05:52:07,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2706386.6666666665, ans=0.125
2023-11-24 05:52:08,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2706386.6666666665, ans=0.125
2023-11-24 05:52:18,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2706453.3333333335, ans=0.1
2023-11-24 05:52:39,406 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9200, loss[loss=0.06151, simple_loss=0.07586, pruned_loss=0.01438, audio_tagging_loss=0.0092, over 14188.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09253, pruned_loss=0.01354, audio_tagging_loss=0.00872, over 3057048.62 frames. ], batch size: 53, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:52:51,501 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406000
2023-11-24 05:52:59,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2706653.3333333335, ans=0.1
2023-11-24 05:53:25,308 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2706786.6666666665, ans=0.2
2023-11-24 05:53:26,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2706786.6666666665, ans=0.125
2023-11-24 05:53:34,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2706853.3333333335, ans=0.0
2023-11-24 05:53:41,426 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9250, loss[loss=0.07688, simple_loss=0.1038, pruned_loss=0.01664, audio_tagging_loss=0.008357, over 15562.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09199, pruned_loss=0.01344, audio_tagging_loss=0.0088, over 3062811.40 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:53:54,516 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406050
2023-11-24 05:53:57,354 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.948e+01 8.286e+01 8.929e+01 9.652e+01 1.621e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 05:54:00,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2706986.6666666665, ans=0.1
2023-11-24 05:54:01,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2706986.6666666665, ans=0.0
2023-11-24 05:54:02,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2706986.6666666665, ans=0.125
2023-11-24 05:54:11,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2707053.3333333335, ans=0.1
2023-11-24 05:54:12,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2707053.3333333335, ans=0.125
2023-11-24 05:54:45,408 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9300, loss[loss=0.05442, simple_loss=0.06966, pruned_loss=0.008827, audio_tagging_loss=0.01076, over 15264.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09195, pruned_loss=0.01349, audio_tagging_loss=0.008722, over 3059281.01 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:54:49,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2707253.3333333335, ans=0.125
2023-11-24 05:54:57,989 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406100
2023-11-24 05:54:58,514 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.00 vs. limit=15.0
2023-11-24 05:55:00,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2707320.0, ans=0.0
2023-11-24 05:55:12,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2707386.6666666665, ans=0.125
2023-11-24 05:55:27,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2707453.3333333335, ans=0.125
2023-11-24 05:55:28,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2707453.3333333335, ans=0.125
2023-11-24 05:55:36,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2707520.0, ans=0.125
2023-11-24 05:55:39,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2707520.0, ans=0.0
2023-11-24 05:55:40,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2707520.0, ans=0.1
2023-11-24 05:55:47,296 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9350, loss[loss=0.05544, simple_loss=0.08103, pruned_loss=0.006482, audio_tagging_loss=0.008447, over 15524.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09107, pruned_loss=0.01326, audio_tagging_loss=0.008784, over 3052724.10 frames. ], batch size: 59, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:55:59,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406150
2023-11-24 05:56:01,327 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.267e+01 8.542e+01 9.315e+01 1.015e+02 1.329e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 05:56:13,945 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 05:56:43,674 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.37 vs. limit=10.0
2023-11-24 05:56:43,988 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.55 vs. limit=22.5
2023-11-24 05:56:49,159 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9400, loss[loss=0.06445, simple_loss=0.08594, pruned_loss=0.01067, audio_tagging_loss=0.01081, over 15167.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09114, pruned_loss=0.01331, audio_tagging_loss=0.008921, over 3051389.29 frames. ], batch size: 58, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:56:49,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2707920.0, ans=0.125
2023-11-24 05:56:51,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2707920.0, ans=0.2
2023-11-24 05:56:53,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2707920.0, ans=0.125
2023-11-24 05:57:02,297 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406200
2023-11-24 05:57:26,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2708120.0, ans=0.025
2023-11-24 05:57:38,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2708186.6666666665, ans=0.1
2023-11-24 05:57:40,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2708186.6666666665, ans=0.125
2023-11-24 05:57:41,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2708186.6666666665, ans=0.07
2023-11-24 05:57:50,064 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 05:57:52,361 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9450, loss[loss=0.06443, simple_loss=0.09633, pruned_loss=0.008381, audio_tagging_loss=0.007882, over 15485.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09142, pruned_loss=0.01323, audio_tagging_loss=0.008966, over 3050507.13 frames. ], batch size: 55, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:58:05,455 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406250
2023-11-24 05:58:07,663 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.916e+01 8.368e+01 8.774e+01 9.342e+01 1.287e+02, threshold=1.755e+02, percent-clipped=0.0
2023-11-24 05:58:17,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2708386.6666666665, ans=0.0
2023-11-24 05:58:21,578 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.21 vs. limit=15.0
2023-11-24 05:58:32,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2708453.3333333335, ans=0.2
2023-11-24 05:58:55,173 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9500, loss[loss=0.08841, simple_loss=0.1268, pruned_loss=0.01633, audio_tagging_loss=0.008672, over 14326.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09189, pruned_loss=0.01325, audio_tagging_loss=0.009047, over 3048377.48 frames. ], batch size: 54, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 05:58:57,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2708586.6666666665, ans=0.125
2023-11-24 05:58:58,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2708586.6666666665, ans=0.125
2023-11-24 05:59:07,311 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406300
2023-11-24 05:59:07,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2708653.3333333335, ans=0.125
2023-11-24 05:59:08,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2708653.3333333335, ans=0.125
2023-11-24 05:59:14,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2708653.3333333335, ans=0.125
2023-11-24 05:59:26,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2708720.0, ans=10.0
2023-11-24 05:59:39,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2708786.6666666665, ans=0.125
2023-11-24 05:59:40,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2708786.6666666665, ans=0.0
2023-11-24 05:59:50,616 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=12.17 vs. limit=15.0
2023-11-24 05:59:56,986 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9550, loss[loss=0.09392, simple_loss=0.135, pruned_loss=0.01923, audio_tagging_loss=0.007192, over 15111.00 frames. ], tot_loss[loss=0.0692, simple_loss=0.09332, pruned_loss=0.01347, audio_tagging_loss=0.009074, over 3044312.30 frames. ], batch size: 56, lr: 1.99e-03, grad_scale: 16.0
2023-11-24 06:00:09,179 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406350
2023-11-24 06:00:13,192 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.057e+01 8.403e+01 9.161e+01 9.740e+01 1.180e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 06:00:31,751 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2709053.3333333335, ans=0.2
2023-11-24 06:00:40,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2709120.0, ans=0.09899494936611666
2023-11-24 06:00:53,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2709186.6666666665, ans=0.0
2023-11-24 06:00:58,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2709253.3333333335, ans=0.125
2023-11-24 06:00:59,407 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9600, loss[loss=0.0637, simple_loss=0.08895, pruned_loss=0.01137, audio_tagging_loss=0.007856, over 15519.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.0931, pruned_loss=0.01337, audio_tagging_loss=0.009144, over 3046591.97 frames. ], batch size: 60, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 06:01:11,854 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:01:12,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2709320.0, ans=0.125
2023-11-24 06:01:13,014 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406400
2023-11-24 06:01:20,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2709320.0, ans=0.0
2023-11-24 06:01:22,159 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.82 vs. limit=6.0
2023-11-24 06:01:24,841 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.97 vs. limit=22.5
2023-11-24 06:01:32,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2709386.6666666665, ans=0.125
2023-11-24 06:01:35,530 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.58 vs. limit=12.0
2023-11-24 06:01:37,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2709453.3333333335, ans=0.125
2023-11-24 06:01:42,567 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.65 vs. limit=15.0
2023-11-24 06:01:46,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.47 vs. limit=22.5
2023-11-24 06:01:56,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2709520.0, ans=0.125
2023-11-24 06:02:03,412 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9650, loss[loss=0.06313, simple_loss=0.09076, pruned_loss=0.01049, audio_tagging_loss=0.007261, over 15197.00 frames. ], tot_loss[loss=0.06915, simple_loss=0.09298, pruned_loss=0.01354, audio_tagging_loss=0.009121, over 3050876.87 frames. ], batch size: 57, lr: 1.99e-03, grad_scale: 32.0
2023-11-24 06:02:15,205 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406450
2023-11-24 06:02:18,606 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.885e+01 8.449e+01 8.956e+01 9.585e+01 1.411e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 06:02:35,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.70 vs. limit=15.0
2023-11-24 06:03:04,829 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9700, loss[loss=0.05606, simple_loss=0.0791, pruned_loss=0.008326, audio_tagging_loss=0.008181, over 13917.00 frames. ], tot_loss[loss=0.06919, simple_loss=0.09334, pruned_loss=0.01354, audio_tagging_loss=0.008981, over 3047756.56 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:03:13,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2709920.0, ans=0.125
2023-11-24 06:03:16,712 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406500
2023-11-24 06:03:18,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2709986.6666666665, ans=0.125
2023-11-24 06:03:28,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2710053.3333333335, ans=0.0
2023-11-24 06:03:28,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2710053.3333333335, ans=0.125
2023-11-24 06:03:41,818 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.78 vs. limit=15.0
2023-11-24 06:03:49,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2710120.0, ans=0.125
2023-11-24 06:04:06,192 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9750, loss[loss=0.06376, simple_loss=0.08689, pruned_loss=0.01178, audio_tagging_loss=0.008538, over 15301.00 frames. ], tot_loss[loss=0.06862, simple_loss=0.09285, pruned_loss=0.01341, audio_tagging_loss=0.008781, over 3047248.42 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:04:19,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406550
2023-11-24 06:04:20,568 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.93 vs. limit=15.0
2023-11-24 06:04:21,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2710320.0, ans=0.125
2023-11-24 06:04:24,965 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.565e+01 9.211e+01 9.883e+01 1.176e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 06:04:39,831 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.39 vs. limit=22.5
2023-11-24 06:04:50,282 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2710453.3333333335, ans=0.125
2023-11-24 06:04:50,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.27 vs. limit=15.0
2023-11-24 06:04:57,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2710520.0, ans=0.0
2023-11-24 06:05:07,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2710520.0, ans=0.125
2023-11-24 06:05:09,513 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9800, loss[loss=0.0506, simple_loss=0.06469, pruned_loss=0.009227, audio_tagging_loss=0.009025, over 14825.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09303, pruned_loss=0.01347, audio_tagging_loss=0.008658, over 3042486.44 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:05:11,182 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:05:22,219 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406600
2023-11-24 06:05:24,198 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.80 vs. limit=15.0
2023-11-24 06:05:46,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2710786.6666666665, ans=0.1
2023-11-24 06:05:57,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2710786.6666666665, ans=0.1
2023-11-24 06:06:04,973 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:06:12,018 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9850, loss[loss=0.05515, simple_loss=0.07583, pruned_loss=0.01007, audio_tagging_loss=0.00717, over 14652.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09272, pruned_loss=0.0134, audio_tagging_loss=0.008643, over 3042664.52 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:06:14,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2710920.0, ans=0.0
2023-11-24 06:06:23,916 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406650
2023-11-24 06:06:28,339 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.400e+01 8.541e+01 9.131e+01 9.880e+01 1.241e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 06:06:28,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2710986.6666666665, ans=0.05
2023-11-24 06:06:31,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2710986.6666666665, ans=0.125
2023-11-24 06:06:33,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2710986.6666666665, ans=0.0
2023-11-24 06:06:46,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2711053.3333333335, ans=0.125
2023-11-24 06:06:49,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2711120.0, ans=0.1
2023-11-24 06:06:49,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2711120.0, ans=0.125
2023-11-24 06:07:13,494 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9900, loss[loss=0.05876, simple_loss=0.08529, pruned_loss=0.008657, audio_tagging_loss=0.007455, over 16156.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09221, pruned_loss=0.01331, audio_tagging_loss=0.008645, over 3041866.59 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:07:27,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406700
2023-11-24 06:07:28,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2711320.0, ans=0.2
2023-11-24 06:07:42,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2711386.6666666665, ans=0.1
2023-11-24 06:08:00,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2711453.3333333335, ans=0.125
2023-11-24 06:08:02,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2711520.0, ans=0.125
2023-11-24 06:08:03,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2711520.0, ans=0.0
2023-11-24 06:08:03,804 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.57 vs. limit=10.0
2023-11-24 06:08:16,821 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 9950, loss[loss=0.09131, simple_loss=0.1123, pruned_loss=0.02733, audio_tagging_loss=0.007803, over 14177.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.092, pruned_loss=0.01314, audio_tagging_loss=0.008714, over 3044093.64 frames. ], batch size: 54, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:08:29,172 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406750
2023-11-24 06:08:34,883 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.047e+01 8.279e+01 9.180e+01 9.821e+01 1.550e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 06:08:44,870 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.31 vs. limit=12.0
2023-11-24 06:08:49,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2711720.0, ans=0.1
2023-11-24 06:08:58,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2711786.6666666665, ans=0.125
2023-11-24 06:09:08,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2711853.3333333335, ans=10.0
2023-11-24 06:09:18,682 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10000, loss[loss=0.05773, simple_loss=0.07555, pruned_loss=0.01393, audio_tagging_loss=0.006028, over 16404.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09191, pruned_loss=0.01325, audio_tagging_loss=0.008777, over 3039920.02 frames. ], batch size: 63, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:09:30,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406800
2023-11-24 06:09:43,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2712053.3333333335, ans=0.0
2023-11-24 06:09:45,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2712053.3333333335, ans=0.1
2023-11-24 06:10:01,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2712120.0, ans=0.125
2023-11-24 06:10:05,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2712120.0, ans=0.0
2023-11-24 06:10:12,763 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.95 vs. limit=15.0
2023-11-24 06:10:20,490 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10050, loss[loss=0.06327, simple_loss=0.08256, pruned_loss=0.01142, audio_tagging_loss=0.01057, over 14660.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09271, pruned_loss=0.01351, audio_tagging_loss=0.008722, over 3036736.39 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:10:22,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2712253.3333333335, ans=0.0
2023-11-24 06:10:33,493 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406850
2023-11-24 06:10:39,683 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.233e+01 8.347e+01 9.098e+01 9.653e+01 1.255e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 06:10:42,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2712320.0, ans=0.125
2023-11-24 06:10:48,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2712386.6666666665, ans=0.5
2023-11-24 06:10:50,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2712386.6666666665, ans=0.125
2023-11-24 06:11:11,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2712520.0, ans=0.125
2023-11-24 06:11:11,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2712520.0, ans=0.0
2023-11-24 06:11:22,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2712586.6666666665, ans=0.0
2023-11-24 06:11:23,348 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10100, loss[loss=0.05765, simple_loss=0.06957, pruned_loss=0.01244, audio_tagging_loss=0.01042, over 13845.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.0913, pruned_loss=0.01328, audio_tagging_loss=0.008854, over 3035568.48 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:11:35,122 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406900
2023-11-24 06:11:38,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2712653.3333333335, ans=0.0
2023-11-24 06:12:12,371 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:12:12,827 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.52 vs. limit=15.0
2023-11-24 06:12:19,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2712853.3333333335, ans=0.125
2023-11-24 06:12:23,736 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=10.14 vs. limit=12.0
2023-11-24 06:12:24,237 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10150, loss[loss=0.0638, simple_loss=0.08621, pruned_loss=0.01068, audio_tagging_loss=0.01001, over 15279.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09187, pruned_loss=0.01335, audio_tagging_loss=0.008891, over 3040816.69 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:12:25,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2712920.0, ans=0.1
2023-11-24 06:12:36,667 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 406950
2023-11-24 06:12:41,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2712986.6666666665, ans=0.95
2023-11-24 06:12:42,366 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.240e+01 8.502e+01 9.089e+01 9.719e+01 1.256e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-24 06:12:44,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2712986.6666666665, ans=0.125
2023-11-24 06:12:52,484 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:12:56,212 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:13:04,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2713120.0, ans=0.125
2023-11-24 06:13:26,450 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10200, loss[loss=0.05551, simple_loss=0.06881, pruned_loss=0.008883, audio_tagging_loss=0.01222, over 16247.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09152, pruned_loss=0.01336, audio_tagging_loss=0.008959, over 3037479.05 frames. ], batch size: 61, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:13:37,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2713253.3333333335, ans=0.125
2023-11-24 06:13:39,168 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407000
2023-11-24 06:13:42,506 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:13:51,004 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:13:57,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2713386.6666666665, ans=0.125
2023-11-24 06:14:04,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.27 vs. limit=12.0
2023-11-24 06:14:29,222 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10250, loss[loss=0.06831, simple_loss=0.08817, pruned_loss=0.01506, audio_tagging_loss=0.009163, over 14547.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.0917, pruned_loss=0.01335, audio_tagging_loss=0.009094, over 3044272.57 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:14:35,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2713586.6666666665, ans=0.2
2023-11-24 06:14:41,633 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407050
2023-11-24 06:14:47,354 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.429e+01 8.707e+01 9.386e+01 1.043e+02 1.340e+02, threshold=1.877e+02, percent-clipped=0.0
2023-11-24 06:14:48,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2713653.3333333335, ans=0.1
2023-11-24 06:15:07,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2713786.6666666665, ans=0.0
2023-11-24 06:15:11,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2713786.6666666665, ans=0.2
2023-11-24 06:15:28,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2713853.3333333335, ans=0.125
2023-11-24 06:15:30,890 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10300, loss[loss=0.07148, simple_loss=0.09786, pruned_loss=0.01381, audio_tagging_loss=0.008742, over 15232.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09163, pruned_loss=0.01337, audio_tagging_loss=0.009123, over 3044985.66 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:15:42,732 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407100
2023-11-24 06:15:52,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2713986.6666666665, ans=0.125
2023-11-24 06:16:01,153 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:16:32,192 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10350, loss[loss=0.07961, simple_loss=0.1136, pruned_loss=0.01699, audio_tagging_loss=0.005819, over 15496.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09207, pruned_loss=0.01333, audio_tagging_loss=0.009226, over 3051693.69 frames. ], batch size: 54, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:16:45,334 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407150
2023-11-24 06:16:52,216 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.280e+01 8.459e+01 9.063e+01 9.492e+01 1.258e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-24 06:16:52,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2714320.0, ans=0.125
2023-11-24 06:17:00,305 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.18 vs. limit=8.0
2023-11-24 06:17:14,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2714453.3333333335, ans=0.0
2023-11-24 06:17:17,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2714453.3333333335, ans=0.125
2023-11-24 06:17:22,507 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.07 vs. limit=15.0
2023-11-24 06:17:31,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2714520.0, ans=0.125
2023-11-24 06:17:35,060 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10400, loss[loss=0.06988, simple_loss=0.1003, pruned_loss=0.01223, audio_tagging_loss=0.007499, over 15364.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09071, pruned_loss=0.01312, audio_tagging_loss=0.009325, over 3042205.16 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:17:41,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2714586.6666666665, ans=0.1
2023-11-24 06:17:46,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2714653.3333333335, ans=0.125
2023-11-24 06:17:47,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407200
2023-11-24 06:18:02,114 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.57 vs. limit=15.0
2023-11-24 06:18:02,975 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.27 vs. limit=6.0
2023-11-24 06:18:28,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2714853.3333333335, ans=0.125
2023-11-24 06:18:32,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2714853.3333333335, ans=0.07
2023-11-24 06:18:38,040 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10450, loss[loss=0.0628, simple_loss=0.08567, pruned_loss=0.01128, audio_tagging_loss=0.008675, over 14965.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09024, pruned_loss=0.01308, audio_tagging_loss=0.009317, over 3037051.12 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:18:45,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2714920.0, ans=0.0
2023-11-24 06:18:49,929 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407250
2023-11-24 06:18:52,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2714986.6666666665, ans=0.125
2023-11-24 06:18:55,554 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.473e+01 9.221e+01 1.007e+02 1.315e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 06:19:10,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2715053.3333333335, ans=0.125
2023-11-24 06:19:33,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2715186.6666666665, ans=0.125
2023-11-24 06:19:35,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2715186.6666666665, ans=0.125
2023-11-24 06:19:38,980 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10500, loss[loss=0.06995, simple_loss=0.08301, pruned_loss=0.01681, audio_tagging_loss=0.01164, over 15646.00 frames. ], tot_loss[loss=0.06663, simple_loss=0.08903, pruned_loss=0.01282, audio_tagging_loss=0.0093, over 3030545.67 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:19:39,653 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.66 vs. limit=15.0
2023-11-24 06:19:45,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2715253.3333333335, ans=0.125
2023-11-24 06:19:48,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.46 vs. limit=15.0
2023-11-24 06:19:51,555 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407300
2023-11-24 06:20:19,202 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2715453.3333333335, ans=0.125
2023-11-24 06:20:33,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2715520.0, ans=0.125
2023-11-24 06:20:41,122 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10550, loss[loss=0.07486, simple_loss=0.09827, pruned_loss=0.01454, audio_tagging_loss=0.01118, over 15781.00 frames. ], tot_loss[loss=0.0665, simple_loss=0.08889, pruned_loss=0.01281, audio_tagging_loss=0.009251, over 3037068.99 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:20:45,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2715586.6666666665, ans=0.125
2023-11-24 06:20:54,300 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407350
2023-11-24 06:20:54,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=2715653.3333333335, ans=0.1
2023-11-24 06:21:01,302 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.247e+01 8.501e+01 9.231e+01 9.962e+01 1.175e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 06:21:33,482 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2715853.3333333335, ans=0.125
2023-11-24 06:21:39,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2715853.3333333335, ans=0.125
2023-11-24 06:21:43,200 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10600, loss[loss=0.05347, simple_loss=0.07081, pruned_loss=0.008042, audio_tagging_loss=0.01002, over 13920.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.08926, pruned_loss=0.01297, audio_tagging_loss=0.009219, over 3039128.45 frames. ], batch size: 54, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:21:53,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff3.min_abs, batch_count=2715986.6666666665, ans=0.2
2023-11-24 06:21:55,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407400
2023-11-24 06:22:09,364 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.66 vs. limit=15.0
2023-11-24 06:22:23,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2716120.0, ans=0.2
2023-11-24 06:22:45,106 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10650, loss[loss=0.05742, simple_loss=0.07256, pruned_loss=0.008924, audio_tagging_loss=0.01222, over 14584.00 frames. ], tot_loss[loss=0.06661, simple_loss=0.089, pruned_loss=0.01298, audio_tagging_loss=0.009133, over 3033830.15 frames. ], batch size: 54, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:22:48,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2716253.3333333335, ans=0.125
2023-11-24 06:22:56,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.whiten.whitening_limit, batch_count=2716320.0, ans=12.0
2023-11-24 06:22:56,886 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407450
2023-11-24 06:22:57,435 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.03 vs. limit=15.0
2023-11-24 06:22:59,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2716320.0, ans=0.0
2023-11-24 06:23:04,946 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.602e+01 8.300e+01 8.906e+01 9.684e+01 1.178e+02, threshold=1.781e+02, percent-clipped=0.0
2023-11-24 06:23:45,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2716586.6666666665, ans=0.125
2023-11-24 06:23:46,564 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10700, loss[loss=0.06363, simple_loss=0.07666, pruned_loss=0.01387, audio_tagging_loss=0.01143, over 14870.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.08989, pruned_loss=0.01295, audio_tagging_loss=0.009058, over 3032246.57 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:23:46,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2716586.6666666665, ans=0.125
2023-11-24 06:23:51,327 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.15 vs. limit=15.0
2023-11-24 06:23:55,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2716586.6666666665, ans=0.125
2023-11-24 06:24:00,608 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407500
2023-11-24 06:24:06,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2716653.3333333335, ans=0.0
2023-11-24 06:24:19,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2716720.0, ans=0.125
2023-11-24 06:24:30,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2716786.6666666665, ans=0.125
2023-11-24 06:24:31,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2716786.6666666665, ans=0.1
2023-11-24 06:24:32,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2716786.6666666665, ans=0.125
2023-11-24 06:24:46,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2716853.3333333335, ans=0.125
2023-11-24 06:24:50,043 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10750, loss[loss=0.0783, simple_loss=0.1057, pruned_loss=0.01917, audio_tagging_loss=0.006259, over 15126.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.08992, pruned_loss=0.01302, audio_tagging_loss=0.009061, over 3035811.29 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:25:02,088 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407550
2023-11-24 06:25:09,203 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.785e+01 8.501e+01 8.981e+01 9.807e+01 1.307e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-24 06:25:15,385 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2717053.3333333335, ans=0.0
2023-11-24 06:25:16,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2717053.3333333335, ans=0.1
2023-11-24 06:25:51,845 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10800, loss[loss=0.05116, simple_loss=0.06685, pruned_loss=0.008288, audio_tagging_loss=0.00945, over 16858.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.08982, pruned_loss=0.01288, audio_tagging_loss=0.008974, over 3037523.49 frames. ], batch size: 66, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:25:55,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2717253.3333333335, ans=0.09899494936611666
2023-11-24 06:26:04,061 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407600
2023-11-24 06:26:07,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2717320.0, ans=0.125
2023-11-24 06:26:12,547 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.34 vs. limit=22.5
2023-11-24 06:26:30,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2717453.3333333335, ans=0.125
2023-11-24 06:26:31,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2717453.3333333335, ans=0.125
2023-11-24 06:26:34,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2717453.3333333335, ans=0.125
2023-11-24 06:26:38,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2717453.3333333335, ans=0.125
2023-11-24 06:26:39,342 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.24 vs. limit=15.0
2023-11-24 06:26:40,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2717453.3333333335, ans=0.125
2023-11-24 06:26:44,021 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.45 vs. limit=15.0
2023-11-24 06:26:54,168 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10850, loss[loss=0.08355, simple_loss=0.1164, pruned_loss=0.01692, audio_tagging_loss=0.008448, over 14566.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09012, pruned_loss=0.01302, audio_tagging_loss=0.008953, over 3034654.61 frames. ], batch size: 53, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:27:03,319 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:27:07,797 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407650
2023-11-24 06:27:08,473 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.07 vs. limit=15.0
2023-11-24 06:27:15,320 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.695e+01 9.228e+01 1.009e+02 1.373e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 06:27:27,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2717720.0, ans=0.2
2023-11-24 06:27:54,162 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:27:57,724 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10900, loss[loss=0.08033, simple_loss=0.1019, pruned_loss=0.01865, audio_tagging_loss=0.01075, over 15839.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.0902, pruned_loss=0.01293, audio_tagging_loss=0.008904, over 3045897.68 frames. ], batch size: 61, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:28:04,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2717920.0, ans=0.125
2023-11-24 06:28:10,233 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407700
2023-11-24 06:28:32,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_na.min_abs, batch_count=2718120.0, ans=0.02
2023-11-24 06:28:46,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2718186.6666666665, ans=0.125
2023-11-24 06:28:58,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2718253.3333333335, ans=0.125
2023-11-24 06:28:59,463 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 10950, loss[loss=0.04096, simple_loss=0.05142, pruned_loss=0.007105, audio_tagging_loss=0.008139, over 13956.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.08989, pruned_loss=0.01295, audio_tagging_loss=0.008934, over 3042549.05 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:29:11,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407750
2023-11-24 06:29:20,638 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.403e+01 8.864e+01 9.655e+01 1.476e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-24 06:29:35,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2718453.3333333335, ans=0.125
2023-11-24 06:29:36,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.20 vs. limit=10.0
2023-11-24 06:29:38,860 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:29:41,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2718453.3333333335, ans=0.1
2023-11-24 06:29:59,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2718586.6666666665, ans=0.125
2023-11-24 06:30:00,924 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11000, loss[loss=0.06172, simple_loss=0.08547, pruned_loss=0.009544, audio_tagging_loss=0.00944, over 15456.00 frames. ], tot_loss[loss=0.06661, simple_loss=0.08929, pruned_loss=0.01284, audio_tagging_loss=0.009126, over 3047730.30 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:30:08,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2718586.6666666665, ans=0.125
2023-11-24 06:30:11,001 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:30:14,010 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407800
2023-11-24 06:30:24,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2718653.3333333335, ans=0.09899494936611666
2023-11-24 06:30:32,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2718720.0, ans=0.125
2023-11-24 06:30:32,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2718720.0, ans=0.1
2023-11-24 06:30:48,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2718786.6666666665, ans=0.2
2023-11-24 06:31:04,754 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11050, loss[loss=0.05807, simple_loss=0.07559, pruned_loss=0.009745, audio_tagging_loss=0.01053, over 14895.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.09036, pruned_loss=0.01304, audio_tagging_loss=0.009059, over 3046949.82 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:31:06,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2718920.0, ans=0.0
2023-11-24 06:31:17,362 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407850
2023-11-24 06:31:26,792 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.384e+01 8.514e+01 8.990e+01 9.675e+01 1.640e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 06:31:30,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2719053.3333333335, ans=0.125
2023-11-24 06:31:55,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2719186.6666666665, ans=0.07
2023-11-24 06:32:01,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2719186.6666666665, ans=0.0
2023-11-24 06:32:06,875 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11100, loss[loss=0.08505, simple_loss=0.1093, pruned_loss=0.0195, audio_tagging_loss=0.01089, over 13978.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09124, pruned_loss=0.01323, audio_tagging_loss=0.009103, over 3048929.26 frames. ], batch size: 53, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:32:19,019 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407900
2023-11-24 06:32:27,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2719320.0, ans=0.1
2023-11-24 06:33:03,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2719520.0, ans=0.125
2023-11-24 06:33:05,652 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.23 vs. limit=22.5
2023-11-24 06:33:08,623 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11150, loss[loss=0.05089, simple_loss=0.06248, pruned_loss=0.009601, audio_tagging_loss=0.01005, over 14651.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09094, pruned_loss=0.01326, audio_tagging_loss=0.009174, over 3047325.63 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:33:21,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 407950
2023-11-24 06:33:21,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2719653.3333333335, ans=0.125
2023-11-24 06:33:26,099 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:33:31,713 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.721e+01 8.657e+01 9.225e+01 9.905e+01 1.226e+02, threshold=1.845e+02, percent-clipped=0.0
2023-11-24 06:33:37,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2719720.0, ans=0.125
2023-11-24 06:33:42,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2719720.0, ans=0.2
2023-11-24 06:34:02,099 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=8.37 vs. limit=12.0
2023-11-24 06:34:10,878 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11200, loss[loss=0.08395, simple_loss=0.1076, pruned_loss=0.0194, audio_tagging_loss=0.01075, over 15867.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09157, pruned_loss=0.01343, audio_tagging_loss=0.009146, over 3047464.63 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:34:13,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.07 vs. limit=6.0
2023-11-24 06:34:15,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2719920.0, ans=0.125
2023-11-24 06:34:15,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2719920.0, ans=0.125
2023-11-24 06:34:23,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408000
2023-11-24 06:34:24,613 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-408000.pt
2023-11-24 06:35:16,698 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11250, loss[loss=0.08109, simple_loss=0.1011, pruned_loss=0.02144, audio_tagging_loss=0.009096, over 14644.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09171, pruned_loss=0.01354, audio_tagging_loss=0.009143, over 3046059.75 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:35:24,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2720253.3333333335, ans=0.1
2023-11-24 06:35:26,000 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.59 vs. limit=15.0
2023-11-24 06:35:29,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408050
2023-11-24 06:35:37,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2720320.0, ans=0.125
2023-11-24 06:35:39,636 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.153e+01 8.539e+01 9.181e+01 9.755e+01 1.709e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 06:36:03,221 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.44 vs. limit=15.0
2023-11-24 06:36:06,139 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:36:07,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2720520.0, ans=0.125
2023-11-24 06:36:18,131 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11300, loss[loss=0.0519, simple_loss=0.06654, pruned_loss=0.01009, audio_tagging_loss=0.008544, over 16164.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09176, pruned_loss=0.0134, audio_tagging_loss=0.009047, over 3051776.35 frames. ], batch size: 63, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:36:30,487 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408100
2023-11-24 06:36:38,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2720653.3333333335, ans=0.05
2023-11-24 06:36:41,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2720653.3333333335, ans=0.125
2023-11-24 06:36:41,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2720653.3333333335, ans=0.125
2023-11-24 06:37:19,934 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11350, loss[loss=0.07905, simple_loss=0.09703, pruned_loss=0.02111, audio_tagging_loss=0.009427, over 14527.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09137, pruned_loss=0.01337, audio_tagging_loss=0.009006, over 3045333.68 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:37:26,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2720920.0, ans=0.125
2023-11-24 06:37:33,086 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408150
2023-11-24 06:37:39,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2720986.6666666665, ans=0.1
2023-11-24 06:37:43,672 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.886e+01 8.373e+01 9.046e+01 9.724e+01 1.113e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 06:37:46,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2721053.3333333335, ans=0.04949747468305833
2023-11-24 06:37:49,940 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.34 vs. limit=15.0
2023-11-24 06:37:56,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2721120.0, ans=0.1
2023-11-24 06:37:58,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2721120.0, ans=0.1
2023-11-24 06:37:58,187 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.32 vs. limit=12.0
2023-11-24 06:38:04,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2721120.0, ans=0.125
2023-11-24 06:38:04,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2721120.0, ans=0.0
2023-11-24 06:38:22,790 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11400, loss[loss=0.08733, simple_loss=0.122, pruned_loss=0.02038, audio_tagging_loss=0.005968, over 13785.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09169, pruned_loss=0.01334, audio_tagging_loss=0.008935, over 3039339.18 frames. ], batch size: 53, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:38:31,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2721253.3333333335, ans=0.0
2023-11-24 06:38:33,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2721320.0, ans=0.2
2023-11-24 06:38:34,687 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408200
2023-11-24 06:39:01,373 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.83 vs. limit=15.0
2023-11-24 06:39:12,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2721520.0, ans=0.125
2023-11-24 06:39:16,235 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:39:24,332 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11450, loss[loss=0.06968, simple_loss=0.09377, pruned_loss=0.01565, audio_tagging_loss=0.007144, over 15451.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09196, pruned_loss=0.01341, audio_tagging_loss=0.008805, over 3038224.91 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:39:33,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2721586.6666666665, ans=0.2
2023-11-24 06:39:34,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2721586.6666666665, ans=0.125
2023-11-24 06:39:36,971 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408250
2023-11-24 06:39:40,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2721653.3333333335, ans=0.2
2023-11-24 06:39:41,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2721653.3333333335, ans=0.0
2023-11-24 06:39:48,648 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.611e+01 9.148e+01 9.766e+01 1.313e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 06:39:53,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2721720.0, ans=0.2
2023-11-24 06:40:03,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2721786.6666666665, ans=0.1
2023-11-24 06:40:06,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2721786.6666666665, ans=0.125
2023-11-24 06:40:26,823 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11500, loss[loss=0.05898, simple_loss=0.07267, pruned_loss=0.00948, audio_tagging_loss=0.01316, over 15470.00 frames. ], tot_loss[loss=0.06855, simple_loss=0.09228, pruned_loss=0.01354, audio_tagging_loss=0.00887, over 3043585.35 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:40:39,541 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408300
2023-11-24 06:40:44,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2721986.6666666665, ans=0.1
2023-11-24 06:40:55,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2722053.3333333335, ans=0.1
2023-11-24 06:41:06,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2722120.0, ans=0.1
2023-11-24 06:41:07,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2722120.0, ans=0.125
2023-11-24 06:41:27,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2722253.3333333335, ans=0.125
2023-11-24 06:41:28,739 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11550, loss[loss=0.07848, simple_loss=0.1026, pruned_loss=0.01573, audio_tagging_loss=0.01142, over 14758.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09124, pruned_loss=0.01338, audio_tagging_loss=0.008868, over 3039928.00 frames. ], batch size: 55, lr: 1.98e-03, grad_scale: 8.0
2023-11-24 06:41:40,687 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408350
2023-11-24 06:41:45,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2722320.0, ans=0.125
2023-11-24 06:41:51,960 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.495e+01 8.414e+01 9.031e+01 9.585e+01 1.239e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 06:41:57,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2722386.6666666665, ans=0.125
2023-11-24 06:41:58,354 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2722386.6666666665, ans=0.2
2023-11-24 06:42:07,497 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 06:42:29,726 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11600, loss[loss=0.08371, simple_loss=0.121, pruned_loss=0.01542, audio_tagging_loss=0.0078, over 15248.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09182, pruned_loss=0.01337, audio_tagging_loss=0.008817, over 3043637.47 frames. ], batch size: 57, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:42:42,243 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408400
2023-11-24 06:43:24,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2722853.3333333335, ans=0.0
2023-11-24 06:43:24,817 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.94 vs. limit=10.0
2023-11-24 06:43:29,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2722853.3333333335, ans=0.1
2023-11-24 06:43:31,405 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.27 vs. limit=15.0
2023-11-24 06:43:31,829 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11650, loss[loss=0.05959, simple_loss=0.08352, pruned_loss=0.008578, audio_tagging_loss=0.009255, over 15284.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09184, pruned_loss=0.01339, audio_tagging_loss=0.008878, over 3040629.19 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:43:32,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2722920.0, ans=0.07
2023-11-24 06:43:43,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2722920.0, ans=0.0
2023-11-24 06:43:45,515 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408450
2023-11-24 06:43:55,840 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.490e+01 8.921e+01 9.548e+01 1.142e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-24 06:44:29,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.65 vs. limit=22.5
2023-11-24 06:44:34,537 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11700, loss[loss=0.0617, simple_loss=0.08075, pruned_loss=0.008319, audio_tagging_loss=0.013, over 14469.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09109, pruned_loss=0.01322, audio_tagging_loss=0.008992, over 3042456.40 frames. ], batch size: 53, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:44:39,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2723253.3333333335, ans=0.125
2023-11-24 06:44:42,086 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.14 vs. limit=15.0
2023-11-24 06:44:46,462 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408500
2023-11-24 06:44:49,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2723320.0, ans=0.125
2023-11-24 06:44:53,662 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:44:53,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2723320.0, ans=0.0
2023-11-24 06:45:02,389 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.77 vs. limit=22.5
2023-11-24 06:45:35,970 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11750, loss[loss=0.07001, simple_loss=0.09911, pruned_loss=0.01325, audio_tagging_loss=0.007202, over 15753.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09093, pruned_loss=0.01329, audio_tagging_loss=0.009011, over 3039708.91 frames. ], batch size: 58, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:45:41,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2723586.6666666665, ans=0.125
2023-11-24 06:45:48,014 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408550
2023-11-24 06:45:59,492 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.522e+01 8.698e+01 9.400e+01 1.049e+02 1.274e+02, threshold=1.880e+02, percent-clipped=0.0
2023-11-24 06:46:18,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2723786.6666666665, ans=0.125
2023-11-24 06:46:27,986 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.51 vs. limit=15.0
2023-11-24 06:46:30,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2723853.3333333335, ans=0.125
2023-11-24 06:46:36,807 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11800, loss[loss=0.0527, simple_loss=0.06952, pruned_loss=0.00877, audio_tagging_loss=0.009173, over 15353.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09126, pruned_loss=0.01336, audio_tagging_loss=0.00902, over 3036062.85 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:46:51,046 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408600
2023-11-24 06:47:13,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.45 vs. limit=15.0
2023-11-24 06:47:22,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2724120.0, ans=0.125
2023-11-24 06:47:40,769 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11850, loss[loss=0.05975, simple_loss=0.07568, pruned_loss=0.01201, audio_tagging_loss=0.009898, over 15981.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09085, pruned_loss=0.0132, audio_tagging_loss=0.009129, over 3038008.41 frames. ], batch size: 59, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:47:52,787 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408650
2023-11-24 06:48:02,997 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.11 vs. limit=15.0
2023-11-24 06:48:03,386 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.550e+01 9.032e+01 9.774e+01 1.343e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 06:48:13,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2724386.6666666665, ans=0.125
2023-11-24 06:48:18,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2724453.3333333335, ans=0.1
2023-11-24 06:48:28,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2724453.3333333335, ans=0.1
2023-11-24 06:48:29,315 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2724520.0, ans=0.2
2023-11-24 06:48:42,008 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11900, loss[loss=0.04712, simple_loss=0.06233, pruned_loss=0.006135, audio_tagging_loss=0.009821, over 16458.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09035, pruned_loss=0.01314, audio_tagging_loss=0.009268, over 3040509.09 frames. ], batch size: 62, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:48:50,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2724586.6666666665, ans=0.125
2023-11-24 06:48:54,122 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408700
2023-11-24 06:49:11,844 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.95 vs. limit=15.0
2023-11-24 06:49:35,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2724853.3333333335, ans=0.125
2023-11-24 06:49:38,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2724853.3333333335, ans=0.125
2023-11-24 06:49:40,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2724853.3333333335, ans=0.0
2023-11-24 06:49:42,888 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 11950, loss[loss=0.07524, simple_loss=0.09955, pruned_loss=0.01561, audio_tagging_loss=0.009859, over 15249.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09084, pruned_loss=0.01322, audio_tagging_loss=0.009288, over 3044971.13 frames. ], batch size: 56, lr: 1.98e-03, grad_scale: 16.0
2023-11-24 06:49:56,361 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408750
2023-11-24 06:49:59,872 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.08 vs. limit=15.0
2023-11-24 06:50:04,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2724986.6666666665, ans=0.125
2023-11-24 06:50:07,404 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.035e+01 8.293e+01 8.993e+01 9.649e+01 1.275e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 06:50:10,519 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.78 vs. limit=15.0
2023-11-24 06:50:16,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2725053.3333333335, ans=0.125
2023-11-24 06:50:17,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2725053.3333333335, ans=0.125
2023-11-24 06:50:21,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2725120.0, ans=0.125
2023-11-24 06:50:28,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2725120.0, ans=0.125
2023-11-24 06:50:42,953 INFO [train_asr.py:1221] (0/4) Epoch 34, batch 12000, loss[loss=0.06604, simple_loss=0.08452, pruned_loss=0.01438, audio_tagging_loss=0.009409, over 16116.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09035, pruned_loss=0.01318, audio_tagging_loss=0.009333, over 3042038.39 frames. ], batch size: 60, lr: 1.98e-03, grad_scale: 32.0
2023-11-24 06:50:42,956 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 06:51:05,120 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.0850, 3.7739, 3.3063, 3.6822], device='cuda:0')
2023-11-24 06:51:19,678 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.1350, 2.5348, 4.9081, 2.8638], device='cuda:0')
2023-11-24 06:51:25,751 INFO [train_asr.py:1253] (0/4) Epoch 34, validation: loss=0.05837, simple_loss=0.05087, pruned_loss=0.005158, audio_tagging_loss=0.02778, over 4681554.00 frames. 
2023-11-24 06:51:25,752 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 06:51:36,890 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408800
2023-11-24 06:51:52,561 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-34.pt
2023-11-24 06:52:25,020 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 0, loss[loss=0.05828, simple_loss=0.05877, pruned_loss=0.005808, audio_tagging_loss=0.02309, over 14425.00 frames. ], tot_loss[loss=0.05828, simple_loss=0.05877, pruned_loss=0.005808, audio_tagging_loss=0.02309, over 14425.00 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:52:25,023 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 06:52:55,597 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([2.2476, 3.0859, 3.3384, 2.9946, 3.6974, 3.7684, 3.3313, 3.1910],
       device='cuda:0')
2023-11-24 06:53:00,550 INFO [train_asr.py:1253] (0/4) Epoch 35, validation: loss=0.05805, simple_loss=0.05089, pruned_loss=0.005144, audio_tagging_loss=0.02746, over 4681554.00 frames. 
2023-11-24 06:53:00,551 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 06:53:14,662 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:53:15,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2725473.3333333335, ans=0.1
2023-11-24 06:53:24,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff2.min_abs, batch_count=2725473.3333333335, ans=0.1
2023-11-24 06:53:28,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.16 vs. limit=15.0
2023-11-24 06:53:47,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408850
2023-11-24 06:53:51,087 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.93 vs. limit=6.0
2023-11-24 06:53:57,723 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.642e+01 8.972e+01 1.010e+02 1.109e+02 1.515e+02, threshold=2.019e+02, percent-clipped=0.0
2023-11-24 06:54:03,124 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 50, loss[loss=0.07593, simple_loss=0.09026, pruned_loss=0.01146, audio_tagging_loss=0.01934, over 14385.00 frames. ], tot_loss[loss=0.07451, simple_loss=0.08807, pruned_loss=0.01271, audio_tagging_loss=0.01777, over 686399.71 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:54:12,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2725740.0, ans=0.125
2023-11-24 06:54:50,015 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408900
2023-11-24 06:55:04,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2726006.6666666665, ans=0.0
2023-11-24 06:55:05,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2726073.3333333335, ans=0.1
2023-11-24 06:55:06,614 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 100, loss[loss=0.06658, simple_loss=0.08501, pruned_loss=0.008678, audio_tagging_loss=0.01539, over 15573.00 frames. ], tot_loss[loss=0.07473, simple_loss=0.08987, pruned_loss=0.01301, audio_tagging_loss=0.01678, over 1202656.17 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:55:21,080 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.81 vs. limit=15.0
2023-11-24 06:55:53,050 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 408950
2023-11-24 06:56:01,244 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.33 vs. limit=12.0
2023-11-24 06:56:04,090 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.733e+01 9.169e+01 9.656e+01 1.031e+02 1.256e+02, threshold=1.931e+02, percent-clipped=0.0
2023-11-24 06:56:08,804 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 150, loss[loss=0.06111, simple_loss=0.07884, pruned_loss=0.01181, audio_tagging_loss=0.009877, over 15119.00 frames. ], tot_loss[loss=0.0733, simple_loss=0.09014, pruned_loss=0.01323, audio_tagging_loss=0.01499, over 1607212.19 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:56:12,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2726406.6666666665, ans=0.035
2023-11-24 06:56:13,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2726406.6666666665, ans=0.0
2023-11-24 06:56:27,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2726473.3333333335, ans=0.125
2023-11-24 06:56:28,610 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=8.27 vs. limit=15.0
2023-11-24 06:56:36,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2726540.0, ans=0.125
2023-11-24 06:56:54,758 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409000
2023-11-24 06:57:05,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2726673.3333333335, ans=0.125
2023-11-24 06:57:11,059 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 200, loss[loss=0.0513, simple_loss=0.06074, pruned_loss=0.008272, audio_tagging_loss=0.01266, over 15040.00 frames. ], tot_loss[loss=0.07073, simple_loss=0.08922, pruned_loss=0.01289, audio_tagging_loss=0.01323, over 1925908.67 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 06:57:15,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2726740.0, ans=0.125
2023-11-24 06:57:20,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2726740.0, ans=0.0
2023-11-24 06:57:56,228 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409050
2023-11-24 06:57:56,837 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.58 vs. limit=22.5
2023-11-24 06:58:06,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2727006.6666666665, ans=0.1
2023-11-24 06:58:09,528 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.892e+01 8.427e+01 9.100e+01 9.790e+01 1.266e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 06:58:13,115 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 250, loss[loss=0.05787, simple_loss=0.08089, pruned_loss=0.008704, audio_tagging_loss=0.00872, over 16105.00 frames. ], tot_loss[loss=0.07013, simple_loss=0.0903, pruned_loss=0.01292, audio_tagging_loss=0.01206, over 2171713.04 frames. ], batch size: 62, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 06:58:14,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2727073.3333333335, ans=0.2
2023-11-24 06:58:20,780 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.28 vs. limit=15.0
2023-11-24 06:58:26,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2727140.0, ans=0.0
2023-11-24 06:58:33,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2727140.0, ans=0.1
2023-11-24 06:58:38,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2727206.6666666665, ans=0.125
2023-11-24 06:58:48,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2727273.3333333335, ans=0.125
2023-11-24 06:58:59,012 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409100
2023-11-24 06:59:14,740 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 300, loss[loss=0.05303, simple_loss=0.06867, pruned_loss=0.01074, audio_tagging_loss=0.007952, over 14458.00 frames. ], tot_loss[loss=0.06967, simple_loss=0.09079, pruned_loss=0.01305, audio_tagging_loss=0.01123, over 2369580.11 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 06:59:16,378 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 06:59:16,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.96 vs. limit=15.0
2023-11-24 06:59:36,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2727473.3333333335, ans=0.125
2023-11-24 06:59:48,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2727540.0, ans=0.0
2023-11-24 07:00:00,739 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409150
2023-11-24 07:00:04,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2727673.3333333335, ans=0.125
2023-11-24 07:00:08,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2727673.3333333335, ans=0.125
2023-11-24 07:00:13,704 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.065e+01 8.645e+01 9.268e+01 9.915e+01 1.259e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 07:00:15,462 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.41 vs. limit=12.0
2023-11-24 07:00:16,048 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 350, loss[loss=0.06425, simple_loss=0.08992, pruned_loss=0.01193, audio_tagging_loss=0.007362, over 16500.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09111, pruned_loss=0.01316, audio_tagging_loss=0.01055, over 2525453.07 frames. ], batch size: 62, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:01:02,304 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409200
2023-11-24 07:01:05,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2728006.6666666665, ans=0.0
2023-11-24 07:01:09,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2728006.6666666665, ans=0.125
2023-11-24 07:01:19,549 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 400, loss[loss=0.07211, simple_loss=0.0994, pruned_loss=0.01473, audio_tagging_loss=0.007675, over 14307.00 frames. ], tot_loss[loss=0.06857, simple_loss=0.09043, pruned_loss=0.0132, audio_tagging_loss=0.01017, over 2636778.28 frames. ], batch size: 53, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:01:23,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2728073.3333333335, ans=0.125
2023-11-24 07:01:28,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2728073.3333333335, ans=0.125
2023-11-24 07:01:58,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2728273.3333333335, ans=0.125
2023-11-24 07:02:05,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409250
2023-11-24 07:02:07,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2728273.3333333335, ans=0.0
2023-11-24 07:02:08,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2728340.0, ans=0.0
2023-11-24 07:02:18,963 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.131e+01 8.492e+01 8.964e+01 9.627e+01 1.454e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 07:02:19,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2728340.0, ans=0.0
2023-11-24 07:02:21,382 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 450, loss[loss=0.06366, simple_loss=0.08345, pruned_loss=0.01103, audio_tagging_loss=0.0109, over 14921.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.0911, pruned_loss=0.01332, audio_tagging_loss=0.009942, over 2726406.20 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:02:40,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2728473.3333333335, ans=0.0
2023-11-24 07:02:57,128 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.53 vs. limit=15.0
2023-11-24 07:03:05,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2728606.6666666665, ans=0.125
2023-11-24 07:03:05,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2728606.6666666665, ans=0.125
2023-11-24 07:03:06,421 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.22 vs. limit=15.0
2023-11-24 07:03:08,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409300
2023-11-24 07:03:13,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2728673.3333333335, ans=0.125
2023-11-24 07:03:24,183 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 500, loss[loss=0.05846, simple_loss=0.07474, pruned_loss=0.01288, audio_tagging_loss=0.008222, over 13919.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09088, pruned_loss=0.01328, audio_tagging_loss=0.009717, over 2794382.98 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:03:24,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2728740.0, ans=0.1
2023-11-24 07:03:46,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2728806.6666666665, ans=0.125
2023-11-24 07:03:50,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2728873.3333333335, ans=0.0
2023-11-24 07:03:56,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2728873.3333333335, ans=0.125
2023-11-24 07:04:10,160 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409350
2023-11-24 07:04:15,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2729006.6666666665, ans=0.1
2023-11-24 07:04:15,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2729006.6666666665, ans=0.0
2023-11-24 07:04:18,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2729006.6666666665, ans=0.1
2023-11-24 07:04:24,161 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.423e+01 9.068e+01 9.980e+01 1.380e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 07:04:27,226 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 550, loss[loss=0.07663, simple_loss=0.0987, pruned_loss=0.02086, audio_tagging_loss=0.006421, over 14387.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09095, pruned_loss=0.01336, audio_tagging_loss=0.009487, over 2844126.21 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:04:42,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2729140.0, ans=0.04949747468305833
2023-11-24 07:04:44,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.34 vs. limit=12.0
2023-11-24 07:04:58,565 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2729206.6666666665, ans=0.1
2023-11-24 07:05:03,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2729273.3333333335, ans=0.125
2023-11-24 07:05:08,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2729273.3333333335, ans=0.2
2023-11-24 07:05:12,325 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409400
2023-11-24 07:05:26,703 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.60 vs. limit=15.0
2023-11-24 07:05:28,489 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 600, loss[loss=0.05746, simple_loss=0.07243, pruned_loss=0.01321, audio_tagging_loss=0.008028, over 14473.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09071, pruned_loss=0.01324, audio_tagging_loss=0.00937, over 2883128.70 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:06:01,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2729540.0, ans=0.125
2023-11-24 07:06:08,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2729606.6666666665, ans=0.125
2023-11-24 07:06:10,403 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.64 vs. limit=15.0
2023-11-24 07:06:14,457 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409450
2023-11-24 07:06:23,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2729673.3333333335, ans=10.0
2023-11-24 07:06:27,513 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.229e+01 8.791e+01 9.631e+01 1.307e+02, threshold=1.758e+02, percent-clipped=0.0
2023-11-24 07:06:29,981 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 650, loss[loss=0.07255, simple_loss=0.1007, pruned_loss=0.01408, audio_tagging_loss=0.008102, over 14759.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09022, pruned_loss=0.01325, audio_tagging_loss=0.00934, over 2911674.95 frames. ], batch size: 53, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:07:02,162 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.42 vs. limit=22.5
2023-11-24 07:07:12,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2729940.0, ans=0.0
2023-11-24 07:07:15,818 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409500
2023-11-24 07:07:18,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2730006.6666666665, ans=0.2
2023-11-24 07:07:32,859 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 700, loss[loss=0.05149, simple_loss=0.06828, pruned_loss=0.00653, audio_tagging_loss=0.01082, over 16615.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09086, pruned_loss=0.01324, audio_tagging_loss=0.009289, over 2942653.98 frames. ], batch size: 67, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:08:00,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2730206.6666666665, ans=0.1
2023-11-24 07:08:08,877 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.09 vs. limit=12.0
2023-11-24 07:08:13,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2730273.3333333335, ans=0.0
2023-11-24 07:08:15,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2730273.3333333335, ans=0.125
2023-11-24 07:08:19,032 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409550
2023-11-24 07:08:24,704 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.17 vs. limit=15.0
2023-11-24 07:08:32,407 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.410e+01 9.143e+01 1.010e+02 1.194e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 07:08:34,825 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 750, loss[loss=0.06282, simple_loss=0.08592, pruned_loss=0.009627, audio_tagging_loss=0.01024, over 14881.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09105, pruned_loss=0.01322, audio_tagging_loss=0.009276, over 2970128.43 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:08:55,719 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.29 vs. limit=22.5
2023-11-24 07:09:01,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2730540.0, ans=0.035
2023-11-24 07:09:16,787 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.76 vs. limit=15.0
2023-11-24 07:09:20,992 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409600
2023-11-24 07:09:29,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2730673.3333333335, ans=0.125
2023-11-24 07:09:36,493 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 800, loss[loss=0.08014, simple_loss=0.1108, pruned_loss=0.01743, audio_tagging_loss=0.007321, over 15558.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.0922, pruned_loss=0.01343, audio_tagging_loss=0.009232, over 2989111.19 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:09:39,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2730740.0, ans=0.125
2023-11-24 07:10:17,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.23 vs. limit=22.5
2023-11-24 07:10:22,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409650
2023-11-24 07:10:35,977 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.567e+01 9.382e+01 1.008e+02 1.693e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 07:10:38,338 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 850, loss[loss=0.06446, simple_loss=0.07879, pruned_loss=0.013, audio_tagging_loss=0.01206, over 15494.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09183, pruned_loss=0.01331, audio_tagging_loss=0.009361, over 2999903.32 frames. ], batch size: 60, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:10:51,006 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.88 vs. limit=15.0
2023-11-24 07:10:52,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2731140.0, ans=0.125
2023-11-24 07:11:24,123 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409700
2023-11-24 07:11:27,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2731340.0, ans=0.125
2023-11-24 07:11:29,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2731340.0, ans=0.125
2023-11-24 07:11:37,632 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=22.17 vs. limit=22.5
2023-11-24 07:11:40,601 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 900, loss[loss=0.07612, simple_loss=0.09671, pruned_loss=0.01659, audio_tagging_loss=0.01117, over 15046.00 frames. ], tot_loss[loss=0.06954, simple_loss=0.0933, pruned_loss=0.01354, audio_tagging_loss=0.00935, over 3009511.12 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:11:43,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2731406.6666666665, ans=0.0
2023-11-24 07:11:51,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_positive, batch_count=2731473.3333333335, ans=0.05
2023-11-24 07:12:14,612 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.24 vs. limit=6.0
2023-11-24 07:12:20,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2731606.6666666665, ans=0.2
2023-11-24 07:12:26,858 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409750
2023-11-24 07:12:39,637 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.971e+01 8.786e+01 9.264e+01 9.713e+01 1.175e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 07:12:42,033 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 950, loss[loss=0.06314, simple_loss=0.08331, pruned_loss=0.01247, audio_tagging_loss=0.009013, over 14228.00 frames. ], tot_loss[loss=0.06953, simple_loss=0.09348, pruned_loss=0.01362, audio_tagging_loss=0.00917, over 3009506.65 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:12:48,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2731740.0, ans=0.0
2023-11-24 07:12:53,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2731806.6666666665, ans=0.0
2023-11-24 07:13:06,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=4.35 vs. limit=15.0
2023-11-24 07:13:08,698 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.93 vs. limit=15.0
2023-11-24 07:13:09,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2731873.3333333335, ans=0.0
2023-11-24 07:13:12,295 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.91 vs. limit=12.0
2023-11-24 07:13:14,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2731873.3333333335, ans=0.0
2023-11-24 07:13:21,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2731940.0, ans=0.0
2023-11-24 07:13:27,227 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:13:28,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409800
2023-11-24 07:13:33,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2732006.6666666665, ans=0.1
2023-11-24 07:13:44,718 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1000, loss[loss=0.08139, simple_loss=0.114, pruned_loss=0.01801, audio_tagging_loss=0.006404, over 15918.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09221, pruned_loss=0.01358, audio_tagging_loss=0.009064, over 3021220.17 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:14:02,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2732140.0, ans=0.2
2023-11-24 07:14:11,511 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:14:28,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2732273.3333333335, ans=0.0
2023-11-24 07:14:31,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409850
2023-11-24 07:14:41,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2732340.0, ans=0.2
2023-11-24 07:14:46,887 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.227e+01 8.432e+01 9.037e+01 9.609e+01 1.581e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 07:14:48,084 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1050, loss[loss=0.09234, simple_loss=0.1224, pruned_loss=0.02398, audio_tagging_loss=0.007143, over 15221.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09287, pruned_loss=0.01368, audio_tagging_loss=0.008969, over 3032803.54 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:15:11,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2732540.0, ans=0.125
2023-11-24 07:15:16,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2732540.0, ans=0.0
2023-11-24 07:15:30,172 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:15:34,122 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409900
2023-11-24 07:15:35,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2732606.6666666665, ans=0.125
2023-11-24 07:15:39,670 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.87 vs. limit=6.0
2023-11-24 07:15:49,681 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1100, loss[loss=0.07108, simple_loss=0.1036, pruned_loss=0.01234, audio_tagging_loss=0.00695, over 15377.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.0924, pruned_loss=0.01355, audio_tagging_loss=0.008944, over 3035012.73 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:15:51,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2732740.0, ans=0.125
2023-11-24 07:15:52,070 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:15:59,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2732740.0, ans=0.07
2023-11-24 07:16:05,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2732806.6666666665, ans=0.0
2023-11-24 07:16:05,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2732806.6666666665, ans=0.2
2023-11-24 07:16:12,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2732806.6666666665, ans=0.125
2023-11-24 07:16:20,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.15 vs. limit=22.5
2023-11-24 07:16:30,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2732940.0, ans=0.0
2023-11-24 07:16:31,754 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.68 vs. limit=15.0
2023-11-24 07:16:35,748 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 409950
2023-11-24 07:16:49,758 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.554e+01 8.985e+01 9.572e+01 1.203e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-24 07:16:50,952 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1150, loss[loss=0.07223, simple_loss=0.1042, pruned_loss=0.0128, audio_tagging_loss=0.007347, over 15063.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09147, pruned_loss=0.01327, audio_tagging_loss=0.008954, over 3039106.00 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:17:17,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2733206.6666666665, ans=0.125
2023-11-24 07:17:33,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2733273.3333333335, ans=0.1
2023-11-24 07:17:37,332 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410000
2023-11-24 07:17:51,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2733340.0, ans=0.1
2023-11-24 07:17:54,386 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1200, loss[loss=0.08167, simple_loss=0.1153, pruned_loss=0.01689, audio_tagging_loss=0.007149, over 14469.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09126, pruned_loss=0.01328, audio_tagging_loss=0.008937, over 3030353.12 frames. ], batch size: 53, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:18:01,887 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.49 vs. limit=22.5
2023-11-24 07:18:08,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2733473.3333333335, ans=0.125
2023-11-24 07:18:13,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2733473.3333333335, ans=0.125
2023-11-24 07:18:23,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2733540.0, ans=0.0
2023-11-24 07:18:27,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2733540.0, ans=0.125
2023-11-24 07:18:40,704 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410050
2023-11-24 07:18:43,267 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:18:44,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2733673.3333333335, ans=0.125
2023-11-24 07:18:55,476 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.712e+01 8.610e+01 9.345e+01 9.890e+01 1.292e+02, threshold=1.869e+02, percent-clipped=0.0
2023-11-24 07:18:56,695 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1250, loss[loss=0.06767, simple_loss=0.08937, pruned_loss=0.01566, audio_tagging_loss=0.007324, over 14174.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09173, pruned_loss=0.01337, audio_tagging_loss=0.008922, over 3039258.32 frames. ], batch size: 53, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:19:06,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2733740.0, ans=0.0
2023-11-24 07:19:12,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.61 vs. limit=15.0
2023-11-24 07:19:18,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2733806.6666666665, ans=0.07
2023-11-24 07:19:28,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2733873.3333333335, ans=0.0
2023-11-24 07:19:34,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2733940.0, ans=0.125
2023-11-24 07:19:43,172 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410100
2023-11-24 07:19:46,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2734006.6666666665, ans=0.125
2023-11-24 07:19:58,438 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1300, loss[loss=0.06814, simple_loss=0.08297, pruned_loss=0.01425, audio_tagging_loss=0.0124, over 13959.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09169, pruned_loss=0.01328, audio_tagging_loss=0.008835, over 3047107.96 frames. ], batch size: 54, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:20:09,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2734073.3333333335, ans=0.025
2023-11-24 07:20:14,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.65 vs. limit=15.0
2023-11-24 07:20:28,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2734206.6666666665, ans=0.2
2023-11-24 07:20:34,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2734206.6666666665, ans=0.125
2023-11-24 07:20:35,020 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.99 vs. limit=15.0
2023-11-24 07:20:45,138 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410150
2023-11-24 07:20:52,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2734340.0, ans=0.1
2023-11-24 07:20:53,282 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=7.45 vs. limit=15.0
2023-11-24 07:20:55,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2734340.0, ans=0.125
2023-11-24 07:21:01,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2734406.6666666665, ans=0.125
2023-11-24 07:21:01,938 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.342e+01 8.312e+01 8.919e+01 9.626e+01 1.151e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-24 07:21:01,991 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1350, loss[loss=0.05231, simple_loss=0.06868, pruned_loss=0.008052, audio_tagging_loss=0.009919, over 15595.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09123, pruned_loss=0.01319, audio_tagging_loss=0.00891, over 3049703.09 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:21:10,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2734406.6666666665, ans=0.0
2023-11-24 07:21:47,032 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:21:48,353 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410200
2023-11-24 07:21:51,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2734673.3333333335, ans=0.125
2023-11-24 07:21:59,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2734673.3333333335, ans=0.125
2023-11-24 07:22:04,505 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1400, loss[loss=0.06099, simple_loss=0.0791, pruned_loss=0.01133, audio_tagging_loss=0.01012, over 15110.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09223, pruned_loss=0.01327, audio_tagging_loss=0.008822, over 3055124.57 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:22:06,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2734740.0, ans=0.125
2023-11-24 07:22:08,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2734740.0, ans=0.1
2023-11-24 07:22:16,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2734806.6666666665, ans=0.1
2023-11-24 07:22:50,203 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410250
2023-11-24 07:22:54,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2735006.6666666665, ans=0.125
2023-11-24 07:23:06,023 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.310e+01 8.227e+01 8.989e+01 9.587e+01 1.141e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 07:23:06,069 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1450, loss[loss=0.06678, simple_loss=0.09627, pruned_loss=0.01097, audio_tagging_loss=0.007677, over 14926.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09222, pruned_loss=0.01331, audio_tagging_loss=0.008946, over 3047708.64 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:23:08,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2735073.3333333335, ans=0.0
2023-11-24 07:23:35,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.40 vs. limit=6.0
2023-11-24 07:23:39,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2735206.6666666665, ans=0.5
2023-11-24 07:23:41,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2735206.6666666665, ans=0.0
2023-11-24 07:23:52,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410300
2023-11-24 07:23:59,119 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0
2023-11-24 07:24:08,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2735406.6666666665, ans=0.2
2023-11-24 07:24:09,065 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1500, loss[loss=0.05855, simple_loss=0.07518, pruned_loss=0.01027, audio_tagging_loss=0.0107, over 16162.00 frames. ], tot_loss[loss=0.06858, simple_loss=0.09252, pruned_loss=0.01329, audio_tagging_loss=0.009035, over 3049259.23 frames. ], batch size: 62, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:24:13,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2735406.6666666665, ans=0.0
2023-11-24 07:24:22,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2735473.3333333335, ans=0.125
2023-11-24 07:24:27,358 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.65 vs. limit=15.0
2023-11-24 07:24:34,063 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.79 vs. limit=6.0
2023-11-24 07:24:41,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2735540.0, ans=0.0
2023-11-24 07:24:42,972 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2735540.0, ans=0.125
2023-11-24 07:24:50,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2735606.6666666665, ans=0.125
2023-11-24 07:24:54,992 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410350
2023-11-24 07:25:05,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2735673.3333333335, ans=0.0
2023-11-24 07:25:10,423 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.411e+01 8.532e+01 9.153e+01 9.693e+01 1.540e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 07:25:10,467 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1550, loss[loss=0.05617, simple_loss=0.08372, pruned_loss=0.007689, audio_tagging_loss=0.006622, over 14889.00 frames. ], tot_loss[loss=0.06918, simple_loss=0.09314, pruned_loss=0.01354, audio_tagging_loss=0.009081, over 3053137.72 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:25:36,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2735873.3333333335, ans=0.05
2023-11-24 07:25:50,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2735940.0, ans=0.125
2023-11-24 07:25:52,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2735940.0, ans=0.0
2023-11-24 07:25:57,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410400
2023-11-24 07:26:03,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2736006.6666666665, ans=0.0
2023-11-24 07:26:05,187 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.74 vs. limit=12.0
2023-11-24 07:26:13,371 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1600, loss[loss=0.07467, simple_loss=0.1008, pruned_loss=0.01572, audio_tagging_loss=0.008535, over 15776.00 frames. ], tot_loss[loss=0.0693, simple_loss=0.09322, pruned_loss=0.01363, audio_tagging_loss=0.009063, over 3051210.76 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 32.0
2023-11-24 07:26:59,526 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410450
2023-11-24 07:27:02,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2736340.0, ans=0.1
2023-11-24 07:27:09,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2736340.0, ans=0.0
2023-11-24 07:27:15,444 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1650, loss[loss=0.05788, simple_loss=0.07185, pruned_loss=0.009736, audio_tagging_loss=0.01222, over 15280.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09198, pruned_loss=0.01351, audio_tagging_loss=0.009129, over 3048647.66 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:27:17,169 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.572e+01 9.096e+01 9.732e+01 1.252e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 07:27:18,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2736406.6666666665, ans=0.07
2023-11-24 07:27:20,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.91 vs. limit=15.0
2023-11-24 07:27:30,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=2736473.3333333335, ans=0.025
2023-11-24 07:28:01,161 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410500
2023-11-24 07:28:01,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2736606.6666666665, ans=0.125
2023-11-24 07:28:05,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2736673.3333333335, ans=0.125
2023-11-24 07:28:11,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2736673.3333333335, ans=0.1
2023-11-24 07:28:16,965 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1700, loss[loss=0.06247, simple_loss=0.08651, pruned_loss=0.01127, audio_tagging_loss=0.007948, over 14946.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09159, pruned_loss=0.01338, audio_tagging_loss=0.009228, over 3053113.48 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:28:36,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2736806.6666666665, ans=0.0
2023-11-24 07:28:51,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2736873.3333333335, ans=0.015
2023-11-24 07:28:58,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2736940.0, ans=0.2
2023-11-24 07:29:03,171 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410550
2023-11-24 07:29:15,257 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:29:18,496 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1750, loss[loss=0.06995, simple_loss=0.08959, pruned_loss=0.01504, audio_tagging_loss=0.01011, over 15062.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09099, pruned_loss=0.0132, audio_tagging_loss=0.009119, over 3046256.91 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:29:19,633 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.653e+01 8.559e+01 9.195e+01 9.925e+01 1.188e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 07:29:44,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2737206.6666666665, ans=0.125
2023-11-24 07:29:44,679 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.42 vs. limit=15.0
2023-11-24 07:30:04,461 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410600
2023-11-24 07:30:10,798 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.27 vs. limit=6.0
2023-11-24 07:30:21,254 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1800, loss[loss=0.04919, simple_loss=0.06447, pruned_loss=0.008481, audio_tagging_loss=0.008475, over 16035.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09031, pruned_loss=0.01306, audio_tagging_loss=0.009005, over 3046517.94 frames. ], batch size: 60, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:30:57,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2737606.6666666665, ans=0.025
2023-11-24 07:30:59,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2737606.6666666665, ans=0.125
2023-11-24 07:30:59,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.30 vs. limit=15.0
2023-11-24 07:31:06,447 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410650
2023-11-24 07:31:21,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2737673.3333333335, ans=0.0
2023-11-24 07:31:23,085 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1850, loss[loss=0.04227, simple_loss=0.06034, pruned_loss=0.004795, audio_tagging_loss=0.007305, over 14016.00 frames. ], tot_loss[loss=0.06674, simple_loss=0.08989, pruned_loss=0.01287, audio_tagging_loss=0.008931, over 3041618.84 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:31:24,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2737740.0, ans=0.1
2023-11-24 07:31:25,382 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.894e+01 8.283e+01 8.932e+01 9.471e+01 1.128e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 07:31:25,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2737740.0, ans=0.125
2023-11-24 07:31:38,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten.whitening_limit, batch_count=2737806.6666666665, ans=22.5
2023-11-24 07:32:00,382 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2737940.0, ans=0.125
2023-11-24 07:32:09,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410700
2023-11-24 07:32:13,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2738006.6666666665, ans=0.2
2023-11-24 07:32:22,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2738006.6666666665, ans=0.2
2023-11-24 07:32:24,367 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1900, loss[loss=0.07245, simple_loss=0.1013, pruned_loss=0.01329, audio_tagging_loss=0.008493, over 15742.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.08969, pruned_loss=0.01291, audio_tagging_loss=0.008866, over 3040286.11 frames. ], batch size: 59, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:32:29,593 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.31 vs. limit=10.0
2023-11-24 07:32:50,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff3.min_abs, batch_count=2738206.6666666665, ans=0.2
2023-11-24 07:33:02,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2738273.3333333335, ans=0.09899494936611666
2023-11-24 07:33:02,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2738273.3333333335, ans=0.125
2023-11-24 07:33:07,710 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.11 vs. limit=15.0
2023-11-24 07:33:10,526 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410750
2023-11-24 07:33:17,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2738340.0, ans=0.125
2023-11-24 07:33:19,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2738340.0, ans=0.0
2023-11-24 07:33:26,608 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 1950, loss[loss=0.08095, simple_loss=0.1075, pruned_loss=0.01746, audio_tagging_loss=0.009728, over 15508.00 frames. ], tot_loss[loss=0.06674, simple_loss=0.08964, pruned_loss=0.01305, audio_tagging_loss=0.008876, over 3045177.53 frames. ], batch size: 56, lr: 1.95e-03, grad_scale: 8.0
2023-11-24 07:33:28,936 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.828e+01 8.505e+01 9.149e+01 9.609e+01 1.191e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 07:33:29,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2738406.6666666665, ans=0.125
2023-11-24 07:33:37,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2738406.6666666665, ans=0.025
2023-11-24 07:33:49,487 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.89 vs. limit=15.0
2023-11-24 07:34:02,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2738606.6666666665, ans=0.0
2023-11-24 07:34:12,057 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410800
2023-11-24 07:34:25,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2738673.3333333335, ans=0.125
2023-11-24 07:34:28,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2738740.0, ans=0.125
2023-11-24 07:34:28,852 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2000, loss[loss=0.07917, simple_loss=0.1011, pruned_loss=0.01733, audio_tagging_loss=0.01127, over 15688.00 frames. ], tot_loss[loss=0.06631, simple_loss=0.08899, pruned_loss=0.01295, audio_tagging_loss=0.008866, over 3046159.94 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:34:30,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2738740.0, ans=0.125
2023-11-24 07:34:57,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2738873.3333333335, ans=0.125
2023-11-24 07:35:15,236 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410850
2023-11-24 07:35:21,274 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:35:30,577 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2050, loss[loss=0.06969, simple_loss=0.09782, pruned_loss=0.01257, audio_tagging_loss=0.008208, over 14486.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.0903, pruned_loss=0.01322, audio_tagging_loss=0.008812, over 3040861.11 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:35:32,817 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 8.370e+01 9.057e+01 9.754e+01 1.235e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-24 07:35:37,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2739073.3333333335, ans=0.125
2023-11-24 07:35:39,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.23 vs. limit=12.0
2023-11-24 07:35:47,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2739140.0, ans=0.0
2023-11-24 07:35:58,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2739206.6666666665, ans=0.125
2023-11-24 07:36:08,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2739273.3333333335, ans=0.0
2023-11-24 07:36:14,927 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.25 vs. limit=12.0
2023-11-24 07:36:16,695 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410900
2023-11-24 07:36:31,827 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2100, loss[loss=0.05857, simple_loss=0.07707, pruned_loss=0.01186, audio_tagging_loss=0.00817, over 15125.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09059, pruned_loss=0.01327, audio_tagging_loss=0.00874, over 3036619.94 frames. ], batch size: 58, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:36:36,635 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.90 vs. limit=15.0
2023-11-24 07:36:39,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2739406.6666666665, ans=0.125
2023-11-24 07:36:39,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2739406.6666666665, ans=0.125
2023-11-24 07:37:18,325 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 410950
2023-11-24 07:37:27,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2739673.3333333335, ans=10.0
2023-11-24 07:37:35,231 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2150, loss[loss=0.05002, simple_loss=0.06835, pruned_loss=0.007541, audio_tagging_loss=0.008303, over 15128.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09012, pruned_loss=0.0132, audio_tagging_loss=0.008842, over 3039954.26 frames. ], batch size: 57, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:37:37,589 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.614e+01 8.763e+01 9.326e+01 1.012e+02 1.387e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 07:37:45,067 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.71 vs. limit=15.0
2023-11-24 07:38:10,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2739940.0, ans=0.2
2023-11-24 07:38:11,455 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:38:21,502 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411000
2023-11-24 07:38:33,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2740006.6666666665, ans=0.125
2023-11-24 07:38:37,196 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2200, loss[loss=0.07866, simple_loss=0.1173, pruned_loss=0.01259, audio_tagging_loss=0.007421, over 15287.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09046, pruned_loss=0.01327, audio_tagging_loss=0.008835, over 3042948.60 frames. ], batch size: 55, lr: 1.95e-03, grad_scale: 16.0
2023-11-24 07:39:07,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2740206.6666666665, ans=0.125
2023-11-24 07:39:22,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2740273.3333333335, ans=0.025
2023-11-24 07:39:23,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411050
2023-11-24 07:39:38,466 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2250, loss[loss=0.04918, simple_loss=0.06042, pruned_loss=0.007428, audio_tagging_loss=0.01154, over 14479.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09118, pruned_loss=0.01343, audio_tagging_loss=0.008887, over 3043615.05 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:39:40,809 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.314e+01 8.864e+01 9.346e+01 9.896e+01 1.400e+02, threshold=1.869e+02, percent-clipped=0.0
2023-11-24 07:39:48,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2740406.6666666665, ans=0.125
2023-11-24 07:39:58,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2740473.3333333335, ans=0.0
2023-11-24 07:40:01,629 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.95 vs. limit=6.0
2023-11-24 07:40:12,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2740540.0, ans=0.1
2023-11-24 07:40:25,273 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411100
2023-11-24 07:40:25,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2740606.6666666665, ans=0.0
2023-11-24 07:40:27,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2740673.3333333335, ans=0.1
2023-11-24 07:40:42,289 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2300, loss[loss=0.07949, simple_loss=0.1066, pruned_loss=0.01674, audio_tagging_loss=0.009434, over 15163.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09031, pruned_loss=0.01323, audio_tagging_loss=0.009005, over 3045913.08 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:40:46,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.06 vs. limit=22.5
2023-11-24 07:40:56,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2740806.6666666665, ans=0.025
2023-11-24 07:41:28,763 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411150
2023-11-24 07:41:37,510 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:41:43,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2741073.3333333335, ans=10.0
2023-11-24 07:41:44,646 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2350, loss[loss=0.06515, simple_loss=0.0949, pruned_loss=0.01043, audio_tagging_loss=0.007267, over 14998.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09113, pruned_loss=0.01328, audio_tagging_loss=0.009064, over 3044270.49 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:41:47,090 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.561e+01 8.408e+01 9.223e+01 9.902e+01 1.512e+02, threshold=1.845e+02, percent-clipped=0.0
2023-11-24 07:42:06,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2741140.0, ans=0.0
2023-11-24 07:42:07,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2741140.0, ans=0.025
2023-11-24 07:42:13,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2741206.6666666665, ans=0.125
2023-11-24 07:42:31,344 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411200
2023-11-24 07:42:33,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2741273.3333333335, ans=0.125
2023-11-24 07:42:47,010 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2400, loss[loss=0.05984, simple_loss=0.07653, pruned_loss=0.009787, audio_tagging_loss=0.01179, over 14983.00 frames. ], tot_loss[loss=0.06861, simple_loss=0.09197, pruned_loss=0.01345, audio_tagging_loss=0.009182, over 3041588.77 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:42:55,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2741406.6666666665, ans=0.0
2023-11-24 07:43:04,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2741473.3333333335, ans=0.125
2023-11-24 07:43:19,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2741540.0, ans=0.125
2023-11-24 07:43:33,280 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411250
2023-11-24 07:43:49,970 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2450, loss[loss=0.07098, simple_loss=0.09864, pruned_loss=0.01405, audio_tagging_loss=0.00761, over 13522.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.0914, pruned_loss=0.01327, audio_tagging_loss=0.009256, over 3042184.74 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:43:52,873 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.793e+01 8.523e+01 9.052e+01 9.834e+01 1.481e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 07:44:04,854 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.84 vs. limit=6.0
2023-11-24 07:44:17,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2741873.3333333335, ans=0.1
2023-11-24 07:44:26,372 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.18 vs. limit=15.0
2023-11-24 07:44:32,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2741940.0, ans=0.125
2023-11-24 07:44:36,389 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411300
2023-11-24 07:44:53,083 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2500, loss[loss=0.0506, simple_loss=0.06107, pruned_loss=0.009423, audio_tagging_loss=0.01064, over 15073.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09164, pruned_loss=0.01333, audio_tagging_loss=0.009201, over 3039934.19 frames. ], batch size: 61, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:45:18,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2742206.6666666665, ans=0.125
2023-11-24 07:45:39,719 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411350
2023-11-24 07:45:43,839 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.88 vs. limit=15.0
2023-11-24 07:45:54,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2742406.6666666665, ans=0.09899494936611666
2023-11-24 07:45:55,250 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2550, loss[loss=0.0707, simple_loss=0.09035, pruned_loss=0.01528, audio_tagging_loss=0.01025, over 13702.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09169, pruned_loss=0.01337, audio_tagging_loss=0.00904, over 3039641.68 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:45:55,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2742406.6666666665, ans=0.0
2023-11-24 07:45:57,674 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.339e+01 8.522e+01 9.145e+01 9.945e+01 1.198e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 07:46:33,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2742606.6666666665, ans=0.1
2023-11-24 07:46:42,051 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411400
2023-11-24 07:46:47,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2742673.3333333335, ans=15.0
2023-11-24 07:46:49,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2742673.3333333335, ans=0.125
2023-11-24 07:46:55,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2742673.3333333335, ans=0.125
2023-11-24 07:46:57,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=2742740.0, ans=0.05
2023-11-24 07:46:58,266 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2600, loss[loss=0.06359, simple_loss=0.08531, pruned_loss=0.01315, audio_tagging_loss=0.007792, over 15233.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09178, pruned_loss=0.01329, audio_tagging_loss=0.008898, over 3037507.57 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:47:15,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2742806.6666666665, ans=0.125
2023-11-24 07:47:22,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2742873.3333333335, ans=0.125
2023-11-24 07:47:28,989 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.71 vs. limit=12.0
2023-11-24 07:47:44,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411450
2023-11-24 07:47:50,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2743006.6666666665, ans=0.5
2023-11-24 07:47:50,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2743006.6666666665, ans=0.125
2023-11-24 07:47:59,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2743006.6666666665, ans=0.0
2023-11-24 07:48:01,093 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2650, loss[loss=0.07694, simple_loss=0.1011, pruned_loss=0.01737, audio_tagging_loss=0.009018, over 15592.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09227, pruned_loss=0.01354, audio_tagging_loss=0.008827, over 3036055.69 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:48:03,408 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.074e+01 8.476e+01 9.136e+01 9.874e+01 1.198e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 07:48:05,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.70 vs. limit=15.0
2023-11-24 07:48:24,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2743206.6666666665, ans=0.0
2023-11-24 07:48:27,475 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.79 vs. limit=22.5
2023-11-24 07:48:47,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411500
2023-11-24 07:49:03,300 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2700, loss[loss=0.05139, simple_loss=0.07029, pruned_loss=0.009517, audio_tagging_loss=0.006725, over 15215.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09058, pruned_loss=0.01332, audio_tagging_loss=0.00885, over 3027920.94 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:49:16,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2743473.3333333335, ans=0.035
2023-11-24 07:49:32,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2743540.0, ans=0.125
2023-11-24 07:49:49,674 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411550
2023-11-24 07:49:59,596 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.59 vs. limit=22.5
2023-11-24 07:50:05,638 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2750, loss[loss=0.07582, simple_loss=0.1091, pruned_loss=0.016, audio_tagging_loss=0.005278, over 15843.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09067, pruned_loss=0.01341, audio_tagging_loss=0.008812, over 3031436.77 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:50:09,741 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.955e+01 8.410e+01 9.303e+01 9.931e+01 1.664e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-24 07:50:14,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.10 vs. limit=15.0
2023-11-24 07:50:25,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2743806.6666666665, ans=0.1
2023-11-24 07:50:29,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2743873.3333333335, ans=0.035
2023-11-24 07:50:41,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2743940.0, ans=0.0
2023-11-24 07:50:43,103 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:50:51,884 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411600
2023-11-24 07:50:53,597 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:50:54,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2744006.6666666665, ans=0.2
2023-11-24 07:50:59,881 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:51:06,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2744006.6666666665, ans=0.0
2023-11-24 07:51:08,614 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2800, loss[loss=0.05822, simple_loss=0.07177, pruned_loss=0.01185, audio_tagging_loss=0.01048, over 15754.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.0902, pruned_loss=0.01326, audio_tagging_loss=0.008801, over 3032272.53 frames. ], batch size: 61, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:51:26,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2744140.0, ans=0.0
2023-11-24 07:51:27,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2744140.0, ans=0.0
2023-11-24 07:51:36,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2744206.6666666665, ans=0.125
2023-11-24 07:51:43,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2744206.6666666665, ans=0.125
2023-11-24 07:51:50,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2744273.3333333335, ans=0.0
2023-11-24 07:51:54,678 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411650
2023-11-24 07:52:09,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2744406.6666666665, ans=0.125
2023-11-24 07:52:10,212 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2850, loss[loss=0.07189, simple_loss=0.09825, pruned_loss=0.01308, audio_tagging_loss=0.009684, over 14887.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09047, pruned_loss=0.01317, audio_tagging_loss=0.008775, over 3032374.04 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:52:14,254 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.630e+01 8.559e+01 8.889e+01 9.637e+01 1.206e+02, threshold=1.778e+02, percent-clipped=0.0
2023-11-24 07:52:15,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2744406.6666666665, ans=0.0
2023-11-24 07:52:18,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2744406.6666666665, ans=0.0
2023-11-24 07:52:26,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2744473.3333333335, ans=0.125
2023-11-24 07:52:41,565 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:52:42,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2744540.0, ans=0.2
2023-11-24 07:52:47,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2744606.6666666665, ans=0.09899494936611666
2023-11-24 07:52:56,682 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411700
2023-11-24 07:53:01,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2744673.3333333335, ans=0.0
2023-11-24 07:53:10,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2744673.3333333335, ans=0.0
2023-11-24 07:53:12,542 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2900, loss[loss=0.07396, simple_loss=0.102, pruned_loss=0.01447, audio_tagging_loss=0.008506, over 14793.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.09048, pruned_loss=0.0132, audio_tagging_loss=0.008769, over 3034160.73 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:53:33,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2744806.6666666665, ans=0.0
2023-11-24 07:53:41,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.max_abs, batch_count=2744873.3333333335, ans=10.0
2023-11-24 07:53:59,242 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411750
2023-11-24 07:54:05,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2745006.6666666665, ans=0.0
2023-11-24 07:54:09,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.21 vs. limit=15.0
2023-11-24 07:54:16,225 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 2950, loss[loss=0.07135, simple_loss=0.0953, pruned_loss=0.01065, audio_tagging_loss=0.01305, over 14671.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09193, pruned_loss=0.01336, audio_tagging_loss=0.008803, over 3039420.44 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 07:54:17,105 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=18.78 vs. limit=22.5
2023-11-24 07:54:19,690 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.555e+01 8.832e+01 9.434e+01 1.012e+02 1.234e+02, threshold=1.887e+02, percent-clipped=0.0
2023-11-24 07:54:29,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2745140.0, ans=0.0
2023-11-24 07:54:43,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2745206.6666666665, ans=0.125
2023-11-24 07:55:00,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2745273.3333333335, ans=6.0
2023-11-24 07:55:02,593 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411800
2023-11-24 07:55:10,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2745340.0, ans=0.125
2023-11-24 07:55:10,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2745340.0, ans=0.0
2023-11-24 07:55:18,158 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3000, loss[loss=0.08304, simple_loss=0.1149, pruned_loss=0.01702, audio_tagging_loss=0.008577, over 14897.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09141, pruned_loss=0.01335, audio_tagging_loss=0.008922, over 3037532.91 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:55:18,161 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 07:56:00,436 INFO [train_asr.py:1253] (0/4) Epoch 35, validation: loss=0.05789, simple_loss=0.05083, pruned_loss=0.005097, audio_tagging_loss=0.02738, over 4681554.00 frames. 
2023-11-24 07:56:00,436 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 07:56:06,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2745406.6666666665, ans=0.1
2023-11-24 07:56:21,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2745473.3333333335, ans=0.125
2023-11-24 07:56:31,397 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 07:56:46,035 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411850
2023-11-24 07:57:02,433 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3050, loss[loss=0.05875, simple_loss=0.07963, pruned_loss=0.01033, audio_tagging_loss=0.008601, over 15298.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09157, pruned_loss=0.01346, audio_tagging_loss=0.008971, over 3039087.04 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:57:05,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2745740.0, ans=0.125
2023-11-24 07:57:07,182 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.364e+01 8.656e+01 9.299e+01 1.004e+02 2.054e+02, threshold=1.860e+02, percent-clipped=1.0
2023-11-24 07:57:08,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2745740.0, ans=0.0
2023-11-24 07:57:18,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2745806.6666666665, ans=0.125
2023-11-24 07:57:38,872 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 07:57:49,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411900
2023-11-24 07:58:04,305 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3100, loss[loss=0.07119, simple_loss=0.09133, pruned_loss=0.01602, audio_tagging_loss=0.009505, over 15098.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.0919, pruned_loss=0.0136, audio_tagging_loss=0.008987, over 3034635.81 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:58:07,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2746073.3333333335, ans=0.125
2023-11-24 07:58:08,533 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=14.03 vs. limit=15.0
2023-11-24 07:58:42,758 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.86 vs. limit=22.5
2023-11-24 07:58:50,692 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 411950
2023-11-24 07:58:52,386 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.27 vs. limit=15.0
2023-11-24 07:58:53,539 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.73 vs. limit=15.0
2023-11-24 07:58:56,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2746340.0, ans=0.1
2023-11-24 07:59:05,908 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3150, loss[loss=0.06577, simple_loss=0.09103, pruned_loss=0.01039, audio_tagging_loss=0.009864, over 14759.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.0921, pruned_loss=0.01356, audio_tagging_loss=0.00912, over 3032757.51 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 07:59:11,716 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.198e+01 8.533e+01 9.313e+01 1.001e+02 1.475e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 07:59:26,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2746473.3333333335, ans=0.125
2023-11-24 07:59:38,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2746540.0, ans=0.125
2023-11-24 07:59:51,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412000
2023-11-24 07:59:53,430 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-412000.pt
2023-11-24 08:00:12,756 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3200, loss[loss=0.07668, simple_loss=0.1036, pruned_loss=0.01663, audio_tagging_loss=0.008238, over 14628.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.0919, pruned_loss=0.01338, audio_tagging_loss=0.009101, over 3038719.11 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:00:13,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2746740.0, ans=0.125
2023-11-24 08:00:46,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2746873.3333333335, ans=0.0
2023-11-24 08:00:51,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2746940.0, ans=0.125
2023-11-24 08:00:58,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412050
2023-11-24 08:01:14,287 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3250, loss[loss=0.0533, simple_loss=0.06327, pruned_loss=0.006524, audio_tagging_loss=0.01514, over 14379.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09167, pruned_loss=0.01327, audio_tagging_loss=0.009221, over 3042777.22 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:01:20,018 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.106e+01 8.341e+01 8.971e+01 9.631e+01 1.385e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-24 08:01:27,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2747140.0, ans=10.0
2023-11-24 08:01:31,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=2747140.0, ans=15.0
2023-11-24 08:01:57,081 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:02:00,450 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412100
2023-11-24 08:02:01,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.36 vs. limit=6.0
2023-11-24 08:02:04,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2747340.0, ans=0.2
2023-11-24 08:02:10,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2747340.0, ans=0.0
2023-11-24 08:02:15,758 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3300, loss[loss=0.06047, simple_loss=0.0792, pruned_loss=0.0115, audio_tagging_loss=0.009369, over 14652.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09161, pruned_loss=0.01342, audio_tagging_loss=0.009256, over 3043102.87 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:02:20,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2747406.6666666665, ans=0.0
2023-11-24 08:02:23,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2747406.6666666665, ans=0.125
2023-11-24 08:02:27,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2747406.6666666665, ans=0.1
2023-11-24 08:02:27,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2747406.6666666665, ans=0.0
2023-11-24 08:03:01,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412150
2023-11-24 08:03:05,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.13 vs. limit=15.0
2023-11-24 08:03:19,330 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3350, loss[loss=0.05606, simple_loss=0.07028, pruned_loss=0.009749, audio_tagging_loss=0.01117, over 15643.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.0912, pruned_loss=0.01333, audio_tagging_loss=0.009156, over 3049902.08 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:03:25,145 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.567e+01 8.617e+01 9.240e+01 1.018e+02 1.398e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 08:03:48,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.40 vs. limit=22.5
2023-11-24 08:03:49,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2747873.3333333335, ans=0.1
2023-11-24 08:03:50,575 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=6.14 vs. limit=12.0
2023-11-24 08:04:01,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2747940.0, ans=0.125
2023-11-24 08:04:03,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2747940.0, ans=0.125
2023-11-24 08:04:04,503 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412200
2023-11-24 08:04:15,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2748006.6666666665, ans=0.125
2023-11-24 08:04:15,830 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.60 vs. limit=15.0
2023-11-24 08:04:20,966 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3400, loss[loss=0.08216, simple_loss=0.1143, pruned_loss=0.01644, audio_tagging_loss=0.008563, over 15775.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09106, pruned_loss=0.01317, audio_tagging_loss=0.009048, over 3049638.80 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:04:37,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2748140.0, ans=0.0
2023-11-24 08:04:43,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2748140.0, ans=0.125
2023-11-24 08:04:44,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2748140.0, ans=0.125
2023-11-24 08:05:05,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2748273.3333333335, ans=0.1
2023-11-24 08:05:07,313 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412250
2023-11-24 08:05:10,377 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.05 vs. limit=15.0
2023-11-24 08:05:22,633 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3450, loss[loss=0.07731, simple_loss=0.1093, pruned_loss=0.01789, audio_tagging_loss=0.004752, over 15366.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09135, pruned_loss=0.01317, audio_tagging_loss=0.008995, over 3049569.87 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:05:28,833 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.390e+01 8.590e+01 9.237e+01 1.003e+02 1.214e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 08:05:32,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2748406.6666666665, ans=0.125
2023-11-24 08:05:32,912 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.52 vs. limit=22.5
2023-11-24 08:05:33,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2748406.6666666665, ans=0.125
2023-11-24 08:06:01,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2748606.6666666665, ans=0.0
2023-11-24 08:06:04,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2748606.6666666665, ans=0.125
2023-11-24 08:06:08,555 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412300
2023-11-24 08:06:16,512 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:06:20,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2748673.3333333335, ans=0.0
2023-11-24 08:06:25,689 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3500, loss[loss=0.07489, simple_loss=0.1033, pruned_loss=0.01582, audio_tagging_loss=0.007446, over 15107.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09171, pruned_loss=0.01331, audio_tagging_loss=0.008922, over 3050258.46 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:06:26,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2748740.0, ans=0.125
2023-11-24 08:06:57,203 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:07:02,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2748940.0, ans=0.125
2023-11-24 08:07:12,224 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412350
2023-11-24 08:07:12,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2748940.0, ans=0.04949747468305833
2023-11-24 08:07:27,851 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3550, loss[loss=0.05901, simple_loss=0.08568, pruned_loss=0.009872, audio_tagging_loss=0.006302, over 14952.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09011, pruned_loss=0.013, audio_tagging_loss=0.00902, over 3040506.69 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:07:29,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2749073.3333333335, ans=0.2
2023-11-24 08:07:33,741 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.970e+01 8.276e+01 8.953e+01 9.644e+01 1.310e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 08:07:35,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2749073.3333333335, ans=0.125
2023-11-24 08:07:45,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2749140.0, ans=0.125
2023-11-24 08:08:01,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2749206.6666666665, ans=0.2
2023-11-24 08:08:01,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2749206.6666666665, ans=0.125
2023-11-24 08:08:14,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412400
2023-11-24 08:08:23,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2749340.0, ans=0.125
2023-11-24 08:08:30,066 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3600, loss[loss=0.05232, simple_loss=0.06954, pruned_loss=0.009001, audio_tagging_loss=0.00855, over 15448.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09012, pruned_loss=0.01323, audio_tagging_loss=0.008962, over 3037786.35 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:08:39,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2749406.6666666665, ans=0.125
2023-11-24 08:09:10,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2749606.6666666665, ans=10.0
2023-11-24 08:09:16,454 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412450
2023-11-24 08:09:16,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2749606.6666666665, ans=0.0
2023-11-24 08:09:33,405 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3650, loss[loss=0.05551, simple_loss=0.0811, pruned_loss=0.006684, audio_tagging_loss=0.008279, over 14117.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.09021, pruned_loss=0.01305, audio_tagging_loss=0.008843, over 3039059.94 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:09:39,300 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.657e+01 8.298e+01 8.982e+01 9.715e+01 1.238e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-24 08:09:55,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2749806.6666666665, ans=0.125
2023-11-24 08:10:18,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2749940.0, ans=0.125
2023-11-24 08:10:19,988 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412500
2023-11-24 08:10:35,331 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3700, loss[loss=0.05679, simple_loss=0.07253, pruned_loss=0.008423, audio_tagging_loss=0.01211, over 14750.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09012, pruned_loss=0.01299, audio_tagging_loss=0.008977, over 3038690.82 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:10:42,661 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.33 vs. limit=15.0
2023-11-24 08:11:17,290 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.66 vs. limit=15.0
2023-11-24 08:11:21,567 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412550
2023-11-24 08:11:29,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2750340.0, ans=0.0
2023-11-24 08:11:30,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2750340.0, ans=0.125
2023-11-24 08:11:37,818 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3750, loss[loss=0.04794, simple_loss=0.06048, pruned_loss=0.008162, audio_tagging_loss=0.009536, over 15002.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09052, pruned_loss=0.0131, audio_tagging_loss=0.008963, over 3042478.10 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:11:39,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2750406.6666666665, ans=0.0
2023-11-24 08:11:44,955 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.608e+01 8.598e+01 9.101e+01 9.921e+01 1.299e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 08:11:53,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2750473.3333333335, ans=0.125
2023-11-24 08:12:05,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2750540.0, ans=0.125
2023-11-24 08:12:20,786 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:12:24,318 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412600
2023-11-24 08:12:28,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2750673.3333333335, ans=0.125
2023-11-24 08:12:36,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2750673.3333333335, ans=0.125
2023-11-24 08:12:40,940 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3800, loss[loss=0.08602, simple_loss=0.1213, pruned_loss=0.01855, audio_tagging_loss=0.006827, over 15528.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.09043, pruned_loss=0.01307, audio_tagging_loss=0.009006, over 3047592.67 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:12:52,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2750806.6666666665, ans=0.125
2023-11-24 08:13:01,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2750806.6666666665, ans=0.125
2023-11-24 08:13:07,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2750873.3333333335, ans=0.125
2023-11-24 08:13:09,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2750873.3333333335, ans=0.125
2023-11-24 08:13:27,037 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412650
2023-11-24 08:13:31,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2751006.6666666665, ans=0.04949747468305833
2023-11-24 08:13:43,145 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3850, loss[loss=0.06927, simple_loss=0.0921, pruned_loss=0.01313, audio_tagging_loss=0.01009, over 14778.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.0912, pruned_loss=0.01332, audio_tagging_loss=0.009023, over 3051217.23 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:13:46,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2751073.3333333335, ans=0.0
2023-11-24 08:13:50,161 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.566e+01 8.505e+01 9.405e+01 9.842e+01 1.302e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-24 08:13:50,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2751073.3333333335, ans=0.125
2023-11-24 08:13:57,455 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.69 vs. limit=6.0
2023-11-24 08:14:17,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2751206.6666666665, ans=0.0
2023-11-24 08:14:25,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2751273.3333333335, ans=0.1
2023-11-24 08:14:29,314 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412700
2023-11-24 08:14:32,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2751340.0, ans=0.0
2023-11-24 08:14:45,024 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3900, loss[loss=0.06244, simple_loss=0.08347, pruned_loss=0.01026, audio_tagging_loss=0.01045, over 15569.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09127, pruned_loss=0.01319, audio_tagging_loss=0.009023, over 3045810.35 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:14:58,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2751473.3333333335, ans=0.125
2023-11-24 08:15:00,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2751473.3333333335, ans=0.2
2023-11-24 08:15:02,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2751473.3333333335, ans=0.2
2023-11-24 08:15:04,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2751473.3333333335, ans=0.1
2023-11-24 08:15:07,117 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.75 vs. limit=22.5
2023-11-24 08:15:09,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2751540.0, ans=0.1
2023-11-24 08:15:11,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2751540.0, ans=0.1
2023-11-24 08:15:13,216 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.53 vs. limit=6.0
2023-11-24 08:15:21,718 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.47 vs. limit=15.0
2023-11-24 08:15:29,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2751606.6666666665, ans=0.1
2023-11-24 08:15:30,551 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412750
2023-11-24 08:15:32,903 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.43 vs. limit=15.0
2023-11-24 08:15:47,562 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 3950, loss[loss=0.07914, simple_loss=0.1135, pruned_loss=0.01349, audio_tagging_loss=0.008923, over 15400.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09199, pruned_loss=0.01327, audio_tagging_loss=0.009051, over 3042839.71 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:15:55,233 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.568e+01 8.460e+01 9.070e+01 9.889e+01 1.315e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 08:16:06,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2751806.6666666665, ans=0.125
2023-11-24 08:16:07,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2751806.6666666665, ans=0.0
2023-11-24 08:16:24,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2751940.0, ans=0.2
2023-11-24 08:16:24,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2751940.0, ans=0.0
2023-11-24 08:16:33,310 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412800
2023-11-24 08:16:41,348 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.15 vs. limit=15.0
2023-11-24 08:16:44,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2752006.6666666665, ans=0.1
2023-11-24 08:16:50,261 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4000, loss[loss=0.05119, simple_loss=0.06207, pruned_loss=0.01107, audio_tagging_loss=0.00909, over 15296.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09218, pruned_loss=0.01336, audio_tagging_loss=0.009076, over 3048202.86 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:16:50,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2752073.3333333335, ans=0.0
2023-11-24 08:16:58,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2752073.3333333335, ans=0.125
2023-11-24 08:17:05,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.74 vs. limit=12.0
2023-11-24 08:17:05,974 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.12 vs. limit=15.0
2023-11-24 08:17:06,929 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:17:15,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2752206.6666666665, ans=0.125
2023-11-24 08:17:17,246 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.47 vs. limit=15.0
2023-11-24 08:17:18,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2752206.6666666665, ans=0.95
2023-11-24 08:17:36,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412850
2023-11-24 08:17:38,131 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.44 vs. limit=15.0
2023-11-24 08:17:46,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2752340.0, ans=0.2
2023-11-24 08:17:51,861 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4050, loss[loss=0.0744, simple_loss=0.1023, pruned_loss=0.0144, audio_tagging_loss=0.008836, over 14899.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09244, pruned_loss=0.0134, audio_tagging_loss=0.009013, over 3046937.83 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:17:54,232 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:17:58,907 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.632e+01 8.603e+01 9.245e+01 9.992e+01 1.368e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 08:18:06,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2752473.3333333335, ans=0.125
2023-11-24 08:18:16,167 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.47 vs. limit=15.0
2023-11-24 08:18:18,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2752540.0, ans=0.125
2023-11-24 08:18:20,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2752540.0, ans=0.0
2023-11-24 08:18:21,969 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.05 vs. limit=6.0
2023-11-24 08:18:25,544 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.50 vs. limit=15.0
2023-11-24 08:18:31,537 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.24 vs. limit=12.0
2023-11-24 08:18:38,175 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412900
2023-11-24 08:18:51,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2752673.3333333335, ans=0.0
2023-11-24 08:18:54,267 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4100, loss[loss=0.06083, simple_loss=0.08099, pruned_loss=0.01106, audio_tagging_loss=0.00927, over 15347.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09278, pruned_loss=0.01335, audio_tagging_loss=0.009006, over 3052326.09 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:19:03,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2752740.0, ans=0.1
2023-11-24 08:19:14,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2752806.6666666665, ans=0.125
2023-11-24 08:19:14,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2752806.6666666665, ans=0.125
2023-11-24 08:19:20,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2752873.3333333335, ans=0.1
2023-11-24 08:19:35,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2752940.0, ans=0.125
2023-11-24 08:19:41,094 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 412950
2023-11-24 08:19:56,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2753073.3333333335, ans=0.0
2023-11-24 08:19:57,712 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4150, loss[loss=0.07908, simple_loss=0.1152, pruned_loss=0.0144, audio_tagging_loss=0.007099, over 15000.00 frames. ], tot_loss[loss=0.0689, simple_loss=0.09325, pruned_loss=0.01333, audio_tagging_loss=0.008948, over 3054943.51 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:20:00,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2753073.3333333335, ans=0.125
2023-11-24 08:20:04,896 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.354e+01 9.149e+01 1.001e+02 1.406e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 08:20:09,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.93 vs. limit=10.0
2023-11-24 08:20:13,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2753140.0, ans=0.125
2023-11-24 08:20:17,679 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.03 vs. limit=22.5
2023-11-24 08:20:34,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2753273.3333333335, ans=0.125
2023-11-24 08:20:43,173 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:20:44,466 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413000
2023-11-24 08:20:46,259 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2753273.3333333335, ans=0.125
2023-11-24 08:21:00,137 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4200, loss[loss=0.06945, simple_loss=0.09543, pruned_loss=0.01341, audio_tagging_loss=0.008336, over 16021.00 frames. ], tot_loss[loss=0.06877, simple_loss=0.09314, pruned_loss=0.01327, audio_tagging_loss=0.008938, over 3054464.12 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:21:00,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2753406.6666666665, ans=0.125
2023-11-24 08:21:13,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2753473.3333333335, ans=0.0
2023-11-24 08:21:39,965 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.76 vs. limit=15.0
2023-11-24 08:21:45,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2753606.6666666665, ans=0.0
2023-11-24 08:21:46,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413050
2023-11-24 08:21:58,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2753673.3333333335, ans=0.0
2023-11-24 08:22:01,973 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4250, loss[loss=0.06286, simple_loss=0.08174, pruned_loss=0.01193, audio_tagging_loss=0.01006, over 15579.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09242, pruned_loss=0.01307, audio_tagging_loss=0.008933, over 3047255.23 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:22:11,988 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.934e+01 8.341e+01 8.897e+01 9.785e+01 1.363e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-24 08:22:24,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2753806.6666666665, ans=0.1
2023-11-24 08:22:29,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2753873.3333333335, ans=0.1
2023-11-24 08:22:40,558 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.84 vs. limit=6.0
2023-11-24 08:22:41,733 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.89 vs. limit=15.0
2023-11-24 08:22:48,166 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413100
2023-11-24 08:22:48,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2753940.0, ans=0.125
2023-11-24 08:22:52,312 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.77 vs. limit=6.0
2023-11-24 08:23:05,790 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4300, loss[loss=0.06461, simple_loss=0.09147, pruned_loss=0.01147, audio_tagging_loss=0.007407, over 15389.00 frames. ], tot_loss[loss=0.06871, simple_loss=0.09342, pruned_loss=0.01319, audio_tagging_loss=0.008807, over 3050720.94 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:23:20,907 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.52 vs. limit=15.0
2023-11-24 08:23:21,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2754140.0, ans=0.125
2023-11-24 08:23:21,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2754140.0, ans=0.125
2023-11-24 08:23:38,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2754206.6666666665, ans=0.125
2023-11-24 08:23:52,265 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413150
2023-11-24 08:24:00,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2754340.0, ans=0.125
2023-11-24 08:24:07,364 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4350, loss[loss=0.05089, simple_loss=0.06518, pruned_loss=0.008156, audio_tagging_loss=0.01014, over 15435.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09276, pruned_loss=0.01319, audio_tagging_loss=0.008836, over 3056464.30 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:24:10,016 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:24:11,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.92 vs. limit=15.0
2023-11-24 08:24:15,736 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.072e+01 8.729e+01 9.185e+01 1.017e+02 1.222e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-24 08:24:20,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2754473.3333333335, ans=0.5
2023-11-24 08:24:25,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2754473.3333333335, ans=0.125
2023-11-24 08:24:31,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2754540.0, ans=0.125
2023-11-24 08:24:44,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2754606.6666666665, ans=0.2
2023-11-24 08:24:53,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413200
2023-11-24 08:24:55,726 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.33 vs. limit=22.5
2023-11-24 08:24:57,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2754673.3333333335, ans=0.0
2023-11-24 08:25:09,267 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4400, loss[loss=0.07837, simple_loss=0.1092, pruned_loss=0.01723, audio_tagging_loss=0.006524, over 15073.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09229, pruned_loss=0.01318, audio_tagging_loss=0.008771, over 3053439.27 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:25:55,013 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413250
2023-11-24 08:26:12,571 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4450, loss[loss=0.05748, simple_loss=0.07467, pruned_loss=0.01255, audio_tagging_loss=0.007591, over 14890.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09221, pruned_loss=0.01321, audio_tagging_loss=0.00873, over 3051825.04 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:26:20,940 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.576e+01 8.416e+01 8.955e+01 9.921e+01 1.330e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 08:26:33,357 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.02 vs. limit=15.0
2023-11-24 08:26:34,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2755140.0, ans=0.1
2023-11-24 08:26:41,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2755206.6666666665, ans=0.2
2023-11-24 08:26:42,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2755206.6666666665, ans=0.125
2023-11-24 08:26:58,596 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413300
2023-11-24 08:27:00,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2755273.3333333335, ans=0.125
2023-11-24 08:27:01,182 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.70 vs. limit=22.5
2023-11-24 08:27:03,090 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.73 vs. limit=10.0
2023-11-24 08:27:14,160 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=12.0
2023-11-24 08:27:14,680 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4500, loss[loss=0.06356, simple_loss=0.09078, pruned_loss=0.009238, audio_tagging_loss=0.008929, over 15383.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09289, pruned_loss=0.01335, audio_tagging_loss=0.008641, over 3061244.95 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:27:16,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2755406.6666666665, ans=0.125
2023-11-24 08:27:25,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2755473.3333333335, ans=0.125
2023-11-24 08:27:35,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2755473.3333333335, ans=0.0
2023-11-24 08:27:46,130 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2755540.0, ans=0.2
2023-11-24 08:28:00,506 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413350
2023-11-24 08:28:11,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2755673.3333333335, ans=0.125
2023-11-24 08:28:15,970 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4550, loss[loss=0.09, simple_loss=0.127, pruned_loss=0.02053, audio_tagging_loss=0.005942, over 14588.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.09214, pruned_loss=0.01338, audio_tagging_loss=0.008663, over 3060168.64 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:28:24,740 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.054e+01 8.588e+01 9.174e+01 1.003e+02 1.201e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-24 08:28:34,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2755806.6666666665, ans=0.2
2023-11-24 08:28:45,144 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.93 vs. limit=22.5
2023-11-24 08:28:53,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2755940.0, ans=0.125
2023-11-24 08:29:00,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2755940.0, ans=0.125
2023-11-24 08:29:02,261 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.32 vs. limit=22.5
2023-11-24 08:29:02,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413400
2023-11-24 08:29:04,348 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:29:19,837 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4600, loss[loss=0.07403, simple_loss=0.09738, pruned_loss=0.01649, audio_tagging_loss=0.008851, over 14961.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09167, pruned_loss=0.01343, audio_tagging_loss=0.008798, over 3051970.40 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:29:41,462 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:29:44,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.68 vs. limit=15.0
2023-11-24 08:30:06,495 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413450
2023-11-24 08:30:08,278 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.90 vs. limit=15.0
2023-11-24 08:30:22,933 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4650, loss[loss=0.06953, simple_loss=0.09692, pruned_loss=0.01394, audio_tagging_loss=0.00713, over 15379.00 frames. ], tot_loss[loss=0.06838, simple_loss=0.09217, pruned_loss=0.01348, audio_tagging_loss=0.008818, over 3052869.02 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:30:31,177 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.223e+01 8.404e+01 8.988e+01 9.502e+01 1.265e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 08:30:32,068 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.20 vs. limit=15.0
2023-11-24 08:30:35,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2756473.3333333335, ans=0.125
2023-11-24 08:30:44,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2756473.3333333335, ans=0.125
2023-11-24 08:30:52,762 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.94 vs. limit=22.5
2023-11-24 08:30:55,233 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=2756540.0, ans=15.0
2023-11-24 08:30:59,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2756606.6666666665, ans=0.125
2023-11-24 08:31:08,786 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413500
2023-11-24 08:31:17,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2756673.3333333335, ans=0.0
2023-11-24 08:31:24,028 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4700, loss[loss=0.07486, simple_loss=0.09029, pruned_loss=0.01678, audio_tagging_loss=0.01294, over 15374.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09271, pruned_loss=0.01338, audio_tagging_loss=0.008896, over 3056907.23 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:31:24,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2756740.0, ans=0.2
2023-11-24 08:31:27,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2756740.0, ans=0.0
2023-11-24 08:31:28,387 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.04 vs. limit=15.0
2023-11-24 08:31:36,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2756806.6666666665, ans=0.0
2023-11-24 08:31:49,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2756873.3333333335, ans=0.0
2023-11-24 08:32:04,390 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:32:07,052 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.74 vs. limit=15.0
2023-11-24 08:32:08,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2756940.0, ans=0.0
2023-11-24 08:32:10,217 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413550
2023-11-24 08:32:16,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2757006.6666666665, ans=0.125
2023-11-24 08:32:26,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2757073.3333333335, ans=0.2
2023-11-24 08:32:26,991 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4750, loss[loss=0.07595, simple_loss=0.1006, pruned_loss=0.0169, audio_tagging_loss=0.008752, over 15235.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.0922, pruned_loss=0.01334, audio_tagging_loss=0.009021, over 3043306.84 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:32:38,224 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.376e+01 8.531e+01 9.142e+01 9.868e+01 1.183e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 08:33:03,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2757273.3333333335, ans=0.125
2023-11-24 08:33:13,249 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413600
2023-11-24 08:33:17,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2757340.0, ans=0.125
2023-11-24 08:33:29,353 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4800, loss[loss=0.07712, simple_loss=0.1109, pruned_loss=0.01517, audio_tagging_loss=0.006494, over 14846.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09197, pruned_loss=0.0134, audio_tagging_loss=0.009128, over 3041708.40 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:33:42,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2757473.3333333335, ans=0.125
2023-11-24 08:33:44,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2757473.3333333335, ans=0.1
2023-11-24 08:33:53,622 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.87 vs. limit=15.0
2023-11-24 08:34:03,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2757540.0, ans=0.125
2023-11-24 08:34:03,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2757540.0, ans=0.125
2023-11-24 08:34:07,143 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.41 vs. limit=15.0
2023-11-24 08:34:15,087 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.28 vs. limit=15.0
2023-11-24 08:34:15,808 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413650
2023-11-24 08:34:15,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2757606.6666666665, ans=0.125
2023-11-24 08:34:27,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2757673.3333333335, ans=0.2
2023-11-24 08:34:31,811 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4850, loss[loss=0.06213, simple_loss=0.07678, pruned_loss=0.01343, audio_tagging_loss=0.01031, over 14004.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09121, pruned_loss=0.01328, audio_tagging_loss=0.009216, over 3035168.70 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:34:37,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2757740.0, ans=0.0
2023-11-24 08:34:42,342 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.928e+01 8.773e+01 9.268e+01 1.019e+02 1.463e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 08:35:07,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2757873.3333333335, ans=0.1
2023-11-24 08:35:09,842 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.71 vs. limit=15.0
2023-11-24 08:35:12,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2757940.0, ans=0.125
2023-11-24 08:35:18,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413700
2023-11-24 08:35:28,153 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:35:34,419 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4900, loss[loss=0.06753, simple_loss=0.08455, pruned_loss=0.01679, audio_tagging_loss=0.008466, over 16595.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.09165, pruned_loss=0.01338, audio_tagging_loss=0.009194, over 3039450.99 frames. ], batch size: 66, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:36:12,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2758273.3333333335, ans=0.125
2023-11-24 08:36:20,908 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413750
2023-11-24 08:36:32,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2758340.0, ans=0.1
2023-11-24 08:36:32,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2758340.0, ans=0.0
2023-11-24 08:36:33,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2758340.0, ans=0.0
2023-11-24 08:36:37,225 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 4950, loss[loss=0.08416, simple_loss=0.1171, pruned_loss=0.01894, audio_tagging_loss=0.00666, over 15956.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09205, pruned_loss=0.01358, audio_tagging_loss=0.009032, over 3038272.41 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:36:45,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2758406.6666666665, ans=0.125
2023-11-24 08:36:47,868 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.724e+01 8.295e+01 8.995e+01 9.901e+01 1.240e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 08:36:55,133 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.92 vs. limit=10.0
2023-11-24 08:37:04,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2758540.0, ans=0.0
2023-11-24 08:37:24,181 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413800
2023-11-24 08:37:39,931 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5000, loss[loss=0.05845, simple_loss=0.07827, pruned_loss=0.0106, audio_tagging_loss=0.00872, over 16602.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09189, pruned_loss=0.01341, audio_tagging_loss=0.008955, over 3044890.46 frames. ], batch size: 62, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:38:16,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2758940.0, ans=0.1
2023-11-24 08:38:20,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2758940.0, ans=0.0
2023-11-24 08:38:26,179 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413850
2023-11-24 08:38:29,251 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.89 vs. limit=15.0
2023-11-24 08:38:42,462 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5050, loss[loss=0.07297, simple_loss=0.1034, pruned_loss=0.01475, audio_tagging_loss=0.006534, over 15286.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09206, pruned_loss=0.01335, audio_tagging_loss=0.008801, over 3039977.54 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:38:48,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2759073.3333333335, ans=0.0
2023-11-24 08:38:54,966 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.036e+01 8.375e+01 9.094e+01 9.590e+01 1.351e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 08:39:01,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2759140.0, ans=0.125
2023-11-24 08:39:07,164 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.61 vs. limit=15.0
2023-11-24 08:39:10,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2759206.6666666665, ans=0.0
2023-11-24 08:39:12,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2759206.6666666665, ans=0.125
2023-11-24 08:39:24,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2759273.3333333335, ans=0.09899494936611666
2023-11-24 08:39:28,559 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413900
2023-11-24 08:39:45,564 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5100, loss[loss=0.05481, simple_loss=0.07363, pruned_loss=0.008349, audio_tagging_loss=0.009645, over 14465.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09143, pruned_loss=0.01326, audio_tagging_loss=0.008857, over 3047770.77 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:40:03,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2759473.3333333335, ans=0.07
2023-11-24 08:40:20,014 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:40:24,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2759606.6666666665, ans=0.125
2023-11-24 08:40:30,774 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2759606.6666666665, ans=0.1
2023-11-24 08:40:31,783 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 413950
2023-11-24 08:40:40,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2759673.3333333335, ans=0.125
2023-11-24 08:40:46,917 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5150, loss[loss=0.07045, simple_loss=0.0905, pruned_loss=0.01546, audio_tagging_loss=0.009747, over 15019.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.0913, pruned_loss=0.01329, audio_tagging_loss=0.008869, over 3045199.74 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:40:52,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.02 vs. limit=22.5
2023-11-24 08:40:58,160 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.417e+01 9.000e+01 9.770e+01 1.432e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 08:41:03,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2759806.6666666665, ans=0.0
2023-11-24 08:41:15,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2759873.3333333335, ans=0.125
2023-11-24 08:41:30,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2759940.0, ans=10.0
2023-11-24 08:41:33,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414000
2023-11-24 08:41:49,200 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5200, loss[loss=0.05308, simple_loss=0.07421, pruned_loss=0.009466, audio_tagging_loss=0.006508, over 14905.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09169, pruned_loss=0.01345, audio_tagging_loss=0.008823, over 3045540.53 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:42:03,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2760140.0, ans=0.125
2023-11-24 08:42:12,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2760140.0, ans=0.125
2023-11-24 08:42:16,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2760206.6666666665, ans=15.0
2023-11-24 08:42:35,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414050
2023-11-24 08:42:37,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2760273.3333333335, ans=0.07
2023-11-24 08:42:37,896 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.70 vs. limit=22.5
2023-11-24 08:42:50,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2760340.0, ans=0.125
2023-11-24 08:42:52,960 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5250, loss[loss=0.06629, simple_loss=0.09708, pruned_loss=0.01225, audio_tagging_loss=0.005504, over 14869.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.09232, pruned_loss=0.01359, audio_tagging_loss=0.008711, over 3041450.31 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:42:59,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2760406.6666666665, ans=0.1
2023-11-24 08:43:03,535 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.341e+01 8.495e+01 9.165e+01 9.828e+01 1.218e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 08:43:05,695 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0
2023-11-24 08:43:38,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2760606.6666666665, ans=0.2
2023-11-24 08:43:39,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414100
2023-11-24 08:43:42,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2760673.3333333335, ans=0.2
2023-11-24 08:43:54,376 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5300, loss[loss=0.0631, simple_loss=0.08605, pruned_loss=0.012, audio_tagging_loss=0.008077, over 15211.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09251, pruned_loss=0.01356, audio_tagging_loss=0.008615, over 3042623.80 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:44:13,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2760806.6666666665, ans=0.1
2023-11-24 08:44:35,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2760940.0, ans=0.125
2023-11-24 08:44:38,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2760940.0, ans=0.1
2023-11-24 08:44:40,595 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414150
2023-11-24 08:44:44,738 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=20.11 vs. limit=22.5
2023-11-24 08:44:56,026 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5350, loss[loss=0.0786, simple_loss=0.1128, pruned_loss=0.01492, audio_tagging_loss=0.007263, over 15842.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09335, pruned_loss=0.01357, audio_tagging_loss=0.008669, over 3040015.34 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:44:59,798 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2761073.3333333335, ans=0.125
2023-11-24 08:45:07,828 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.034e+01 8.463e+01 9.143e+01 9.837e+01 1.383e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 08:45:10,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2761140.0, ans=0.2
2023-11-24 08:45:36,136 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=7.99 vs. limit=12.0
2023-11-24 08:45:41,487 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414200
2023-11-24 08:45:43,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2761273.3333333335, ans=0.1
2023-11-24 08:45:48,427 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.60 vs. limit=6.0
2023-11-24 08:45:58,644 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5400, loss[loss=0.05426, simple_loss=0.06929, pruned_loss=0.009873, audio_tagging_loss=0.009739, over 14303.00 frames. ], tot_loss[loss=0.06929, simple_loss=0.09371, pruned_loss=0.01371, audio_tagging_loss=0.008727, over 3040332.95 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:46:27,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2761540.0, ans=0.0
2023-11-24 08:46:44,167 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414250
2023-11-24 08:47:00,096 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5450, loss[loss=0.0716, simple_loss=0.08897, pruned_loss=0.01468, audio_tagging_loss=0.01244, over 15233.00 frames. ], tot_loss[loss=0.06926, simple_loss=0.09334, pruned_loss=0.0138, audio_tagging_loss=0.008802, over 3046883.48 frames. ], batch size: 59, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:47:11,909 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.247e+01 8.777e+01 9.198e+01 9.795e+01 1.303e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-24 08:47:19,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2761806.6666666665, ans=0.0
2023-11-24 08:47:29,052 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:47:29,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2761873.3333333335, ans=0.1
2023-11-24 08:47:46,391 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414300
2023-11-24 08:47:48,385 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.65 vs. limit=22.5
2023-11-24 08:47:49,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2762006.6666666665, ans=0.125
2023-11-24 08:47:56,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2762006.6666666665, ans=0.09899494936611666
2023-11-24 08:47:57,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2762006.6666666665, ans=0.0
2023-11-24 08:47:58,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2762006.6666666665, ans=0.2
2023-11-24 08:48:01,996 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5500, loss[loss=0.05658, simple_loss=0.07692, pruned_loss=0.008709, audio_tagging_loss=0.009412, over 14914.00 frames. ], tot_loss[loss=0.06922, simple_loss=0.09332, pruned_loss=0.0137, audio_tagging_loss=0.008865, over 3049596.39 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:48:37,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=2762206.6666666665, ans=22.5
2023-11-24 08:48:48,687 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414350
2023-11-24 08:49:05,648 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5550, loss[loss=0.07136, simple_loss=0.08676, pruned_loss=0.01589, audio_tagging_loss=0.01209, over 14839.00 frames. ], tot_loss[loss=0.06907, simple_loss=0.09284, pruned_loss=0.01365, audio_tagging_loss=0.008997, over 3057773.90 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:49:17,957 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.668e+01 8.369e+01 9.115e+01 9.992e+01 1.269e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 08:49:26,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=2762473.3333333335, ans=0.05
2023-11-24 08:49:51,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414400
2023-11-24 08:50:08,636 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5600, loss[loss=0.06629, simple_loss=0.08932, pruned_loss=0.01277, audio_tagging_loss=0.008853, over 15541.00 frames. ], tot_loss[loss=0.06895, simple_loss=0.09255, pruned_loss=0.01355, audio_tagging_loss=0.009129, over 3044413.61 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:50:10,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2762740.0, ans=0.125
2023-11-24 08:50:25,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2762806.6666666665, ans=0.1
2023-11-24 08:50:31,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2762873.3333333335, ans=0.0
2023-11-24 08:50:34,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2762873.3333333335, ans=0.07
2023-11-24 08:50:38,900 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.81 vs. limit=22.5
2023-11-24 08:50:41,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2762873.3333333335, ans=0.125
2023-11-24 08:50:53,475 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:50:54,716 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414450
2023-11-24 08:50:55,144 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.69 vs. limit=12.0
2023-11-24 08:51:08,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2763006.6666666665, ans=0.125
2023-11-24 08:51:10,080 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5650, loss[loss=0.05344, simple_loss=0.06751, pruned_loss=0.008007, audio_tagging_loss=0.01168, over 14503.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09181, pruned_loss=0.0134, audio_tagging_loss=0.009215, over 3050283.27 frames. ], batch size: 53, lr: 1.94e-03, grad_scale: 32.0
2023-11-24 08:51:20,040 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.78 vs. limit=12.0
2023-11-24 08:51:21,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2763140.0, ans=0.025
2023-11-24 08:51:22,334 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.229e+01 8.298e+01 8.960e+01 9.539e+01 1.200e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-24 08:51:22,577 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 08:51:25,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_na.min_abs, batch_count=2763140.0, ans=0.02
2023-11-24 08:51:27,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2763140.0, ans=0.1
2023-11-24 08:51:35,483 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.56 vs. limit=12.0
2023-11-24 08:51:44,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2763206.6666666665, ans=0.0
2023-11-24 08:51:50,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2763273.3333333335, ans=0.125
2023-11-24 08:51:51,821 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.52 vs. limit=15.0
2023-11-24 08:51:52,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2763273.3333333335, ans=0.1
2023-11-24 08:51:56,285 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414500
2023-11-24 08:52:12,850 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5700, loss[loss=0.06787, simple_loss=0.09539, pruned_loss=0.01266, audio_tagging_loss=0.007518, over 14784.00 frames. ], tot_loss[loss=0.06873, simple_loss=0.09235, pruned_loss=0.01348, audio_tagging_loss=0.009082, over 3047428.23 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:52:32,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten.whitening_limit, batch_count=2763473.3333333335, ans=15.0
2023-11-24 08:52:37,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2763540.0, ans=0.0
2023-11-24 08:52:58,802 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414550
2023-11-24 08:53:07,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2763673.3333333335, ans=0.2
2023-11-24 08:53:15,298 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5750, loss[loss=0.06137, simple_loss=0.08085, pruned_loss=0.01392, audio_tagging_loss=0.007026, over 14806.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.0911, pruned_loss=0.01338, audio_tagging_loss=0.009085, over 3050664.87 frames. ], batch size: 55, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:53:18,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2763740.0, ans=0.5
2023-11-24 08:53:28,331 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.460e+01 8.483e+01 9.031e+01 9.613e+01 1.136e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 08:53:45,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2763873.3333333335, ans=0.0
2023-11-24 08:53:55,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2763940.0, ans=0.125
2023-11-24 08:53:55,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2763940.0, ans=0.2
2023-11-24 08:54:01,789 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414600
2023-11-24 08:54:12,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2764006.6666666665, ans=0.125
2023-11-24 08:54:17,426 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5800, loss[loss=0.08917, simple_loss=0.126, pruned_loss=0.02, audio_tagging_loss=0.006154, over 15097.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09146, pruned_loss=0.01339, audio_tagging_loss=0.008932, over 3048295.24 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:54:45,978 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.06 vs. limit=15.0
2023-11-24 08:54:56,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2764273.3333333335, ans=0.0
2023-11-24 08:55:04,014 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414650
2023-11-24 08:55:13,185 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.31 vs. limit=15.0
2023-11-24 08:55:14,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2764340.0, ans=0.125
2023-11-24 08:55:20,530 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5850, loss[loss=0.08118, simple_loss=0.1105, pruned_loss=0.01759, audio_tagging_loss=0.008342, over 14490.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09148, pruned_loss=0.0134, audio_tagging_loss=0.008914, over 3038714.90 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:55:25,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2764406.6666666665, ans=0.2
2023-11-24 08:55:36,182 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.408e+01 8.653e+01 9.084e+01 9.959e+01 1.265e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 08:55:46,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2764540.0, ans=0.0
2023-11-24 08:55:47,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2764540.0, ans=0.125
2023-11-24 08:56:01,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2764606.6666666665, ans=0.125
2023-11-24 08:56:03,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2764606.6666666665, ans=0.125
2023-11-24 08:56:07,477 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414700
2023-11-24 08:56:07,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2764606.6666666665, ans=0.2
2023-11-24 08:56:18,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2764673.3333333335, ans=0.125
2023-11-24 08:56:21,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2764673.3333333335, ans=0.0
2023-11-24 08:56:23,851 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5900, loss[loss=0.06534, simple_loss=0.09944, pruned_loss=0.008956, audio_tagging_loss=0.006666, over 14549.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09192, pruned_loss=0.01348, audio_tagging_loss=0.008849, over 3046736.39 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:56:44,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2764806.6666666665, ans=0.0
2023-11-24 08:56:47,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2764873.3333333335, ans=0.125
2023-11-24 08:57:10,267 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414750
2023-11-24 08:57:24,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2765006.6666666665, ans=0.0
2023-11-24 08:57:26,296 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 5950, loss[loss=0.08101, simple_loss=0.1222, pruned_loss=0.01326, audio_tagging_loss=0.006634, over 15747.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09265, pruned_loss=0.01351, audio_tagging_loss=0.008709, over 3045379.20 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 08:57:40,401 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.559e+01 8.356e+01 8.865e+01 9.769e+01 1.184e+02, threshold=1.773e+02, percent-clipped=0.0
2023-11-24 08:57:42,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2765140.0, ans=0.125
2023-11-24 08:57:48,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2765140.0, ans=0.1
2023-11-24 08:58:12,298 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414800
2023-11-24 08:58:22,795 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0
2023-11-24 08:58:27,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.whiten.whitening_limit, batch_count=2765406.6666666665, ans=12.0
2023-11-24 08:58:27,956 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6000, loss[loss=0.04995, simple_loss=0.05456, pruned_loss=0.009745, audio_tagging_loss=0.01292, over 15210.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09096, pruned_loss=0.01322, audio_tagging_loss=0.008784, over 3041570.24 frames. ], batch size: 60, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 08:58:27,959 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 08:58:56,759 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.0108, 3.9284, 3.7071, 3.1790], device='cuda:0')
2023-11-24 08:59:09,821 INFO [train_asr.py:1253] (0/4) Epoch 35, validation: loss=0.05756, simple_loss=0.05084, pruned_loss=0.005093, audio_tagging_loss=0.02705, over 4681554.00 frames. 
2023-11-24 08:59:09,822 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 08:59:21,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2765473.3333333335, ans=0.1
2023-11-24 08:59:25,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2765473.3333333335, ans=0.0
2023-11-24 08:59:30,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2765473.3333333335, ans=0.1
2023-11-24 08:59:31,205 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.92 vs. limit=22.5
2023-11-24 08:59:54,891 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 08:59:56,139 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414850
2023-11-24 08:59:58,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2765673.3333333335, ans=0.1
2023-11-24 09:00:11,864 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6050, loss[loss=0.06311, simple_loss=0.08967, pruned_loss=0.009893, audio_tagging_loss=0.008379, over 15386.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09156, pruned_loss=0.01338, audio_tagging_loss=0.008795, over 3048133.71 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:00:14,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2765740.0, ans=0.1
2023-11-24 09:00:15,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2765740.0, ans=0.125
2023-11-24 09:00:16,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2765740.0, ans=0.025
2023-11-24 09:00:25,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2765806.6666666665, ans=0.125
2023-11-24 09:00:26,526 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.506e+01 8.401e+01 9.139e+01 9.786e+01 1.325e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 09:00:33,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2765806.6666666665, ans=0.2
2023-11-24 09:00:57,481 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.37 vs. limit=10.0
2023-11-24 09:00:58,023 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414900
2023-11-24 09:01:10,442 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.17 vs. limit=15.0
2023-11-24 09:01:13,969 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6100, loss[loss=0.06023, simple_loss=0.08494, pruned_loss=0.008241, audio_tagging_loss=0.009519, over 14927.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09077, pruned_loss=0.01317, audio_tagging_loss=0.008812, over 3048356.29 frames. ], batch size: 54, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:01:35,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2766140.0, ans=0.125
2023-11-24 09:01:39,245 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.45 vs. limit=15.0
2023-11-24 09:01:55,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2766273.3333333335, ans=0.125
2023-11-24 09:02:00,262 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 414950
2023-11-24 09:02:13,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2766340.0, ans=0.2
2023-11-24 09:02:16,788 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6150, loss[loss=0.08517, simple_loss=0.1174, pruned_loss=0.01723, audio_tagging_loss=0.009257, over 15733.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09086, pruned_loss=0.01321, audio_tagging_loss=0.008941, over 3036998.13 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:02:24,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2766406.6666666665, ans=0.125
2023-11-24 09:02:31,215 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.088e+01 8.378e+01 9.060e+01 9.653e+01 1.339e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 09:03:01,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2766606.6666666665, ans=0.0
2023-11-24 09:03:03,148 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415000
2023-11-24 09:03:18,782 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6200, loss[loss=0.07903, simple_loss=0.09953, pruned_loss=0.01962, audio_tagging_loss=0.009649, over 14758.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09032, pruned_loss=0.01315, audio_tagging_loss=0.009086, over 3039957.28 frames. ], batch size: 56, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 09:03:19,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2766740.0, ans=0.025
2023-11-24 09:03:22,099 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.79 vs. limit=5.0
2023-11-24 09:04:04,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415050
2023-11-24 09:04:21,356 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6250, loss[loss=0.07866, simple_loss=0.1003, pruned_loss=0.01843, audio_tagging_loss=0.01008, over 15847.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09052, pruned_loss=0.01313, audio_tagging_loss=0.009222, over 3041330.18 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 09:04:22,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2767073.3333333335, ans=0.125
2023-11-24 09:04:33,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2767140.0, ans=0.1
2023-11-24 09:04:34,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2767140.0, ans=0.125
2023-11-24 09:04:35,231 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.33 vs. limit=22.5
2023-11-24 09:04:38,116 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.219e+01 8.234e+01 8.847e+01 9.550e+01 1.216e+02, threshold=1.769e+02, percent-clipped=0.0
2023-11-24 09:04:43,346 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2767140.0, ans=0.125
2023-11-24 09:04:52,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff2.min_abs, batch_count=2767206.6666666665, ans=0.1
2023-11-24 09:05:07,453 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415100
2023-11-24 09:05:11,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2767340.0, ans=0.0
2023-11-24 09:05:18,605 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.12 vs. limit=6.0
2023-11-24 09:05:24,277 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6300, loss[loss=0.07934, simple_loss=0.1075, pruned_loss=0.01791, audio_tagging_loss=0.007657, over 16045.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09116, pruned_loss=0.01335, audio_tagging_loss=0.009212, over 3040359.35 frames. ], batch size: 58, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 09:05:31,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2767406.6666666665, ans=0.1
2023-11-24 09:05:33,125 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.61 vs. limit=22.5
2023-11-24 09:05:41,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2767473.3333333335, ans=0.125
2023-11-24 09:05:54,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2767540.0, ans=0.125
2023-11-24 09:05:58,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2767540.0, ans=0.2
2023-11-24 09:06:01,785 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:06:10,490 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415150
2023-11-24 09:06:10,599 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2767606.6666666665, ans=0.0
2023-11-24 09:06:16,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2767673.3333333335, ans=0.125
2023-11-24 09:06:16,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2767673.3333333335, ans=0.0
2023-11-24 09:06:17,036 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.02 vs. limit=15.0
2023-11-24 09:06:19,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2767673.3333333335, ans=0.07
2023-11-24 09:06:26,033 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6350, loss[loss=0.05557, simple_loss=0.07554, pruned_loss=0.01033, audio_tagging_loss=0.007467, over 16443.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09094, pruned_loss=0.01317, audio_tagging_loss=0.009191, over 3036088.78 frames. ], batch size: 64, lr: 1.94e-03, grad_scale: 8.0
2023-11-24 09:06:27,708 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.06 vs. limit=6.0
2023-11-24 09:06:35,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2767740.0, ans=0.1
2023-11-24 09:06:38,639 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.89 vs. limit=5.0
2023-11-24 09:06:39,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2767806.6666666665, ans=0.0
2023-11-24 09:06:41,155 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.597e+01 9.129e+01 9.711e+01 1.215e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 09:06:45,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2767806.6666666665, ans=0.125
2023-11-24 09:06:52,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2767873.3333333335, ans=0.1
2023-11-24 09:06:52,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2767873.3333333335, ans=0.0
2023-11-24 09:06:53,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2767873.3333333335, ans=0.125
2023-11-24 09:07:11,401 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415200
2023-11-24 09:07:13,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2767940.0, ans=0.1
2023-11-24 09:07:17,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2768006.6666666665, ans=0.0
2023-11-24 09:07:19,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2768006.6666666665, ans=0.125
2023-11-24 09:07:21,943 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.08 vs. limit=22.5
2023-11-24 09:07:27,187 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6400, loss[loss=0.06396, simple_loss=0.08033, pruned_loss=0.01494, audio_tagging_loss=0.008849, over 14724.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09073, pruned_loss=0.01319, audio_tagging_loss=0.009246, over 3038830.21 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:07:31,020 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:07:34,833 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.49 vs. limit=15.0
2023-11-24 09:07:37,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2768073.3333333335, ans=0.125
2023-11-24 09:07:50,982 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.09 vs. limit=15.0
2023-11-24 09:07:54,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2768206.6666666665, ans=0.125
2023-11-24 09:08:06,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2768273.3333333335, ans=0.04949747468305833
2023-11-24 09:08:06,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2768273.3333333335, ans=0.125
2023-11-24 09:08:12,981 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415250
2023-11-24 09:08:30,546 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6450, loss[loss=0.09383, simple_loss=0.121, pruned_loss=0.02331, audio_tagging_loss=0.01002, over 15231.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09099, pruned_loss=0.01326, audio_tagging_loss=0.00923, over 3037118.26 frames. ], batch size: 57, lr: 1.94e-03, grad_scale: 16.0
2023-11-24 09:08:35,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2768406.6666666665, ans=0.1
2023-11-24 09:08:46,587 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.512e+01 9.076e+01 9.568e+01 1.780e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 09:08:51,007 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.60 vs. limit=22.5
2023-11-24 09:08:58,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2768540.0, ans=0.09899494936611666
2023-11-24 09:09:17,360 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415300
2023-11-24 09:09:24,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2768673.3333333335, ans=0.07
2023-11-24 09:09:32,754 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6500, loss[loss=0.07727, simple_loss=0.1134, pruned_loss=0.01351, audio_tagging_loss=0.007049, over 14557.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.0914, pruned_loss=0.01323, audio_tagging_loss=0.009133, over 3040894.49 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:09:50,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2768806.6666666665, ans=0.125
2023-11-24 09:10:05,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2768873.3333333335, ans=0.0
2023-11-24 09:10:18,023 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415350
2023-11-24 09:10:21,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2769006.6666666665, ans=0.125
2023-11-24 09:10:33,293 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6550, loss[loss=0.08224, simple_loss=0.1111, pruned_loss=0.01611, audio_tagging_loss=0.01056, over 14761.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.0923, pruned_loss=0.01348, audio_tagging_loss=0.008975, over 3044346.79 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:10:40,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2769073.3333333335, ans=0.5
2023-11-24 09:10:48,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2769140.0, ans=0.0
2023-11-24 09:10:50,886 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.574e+01 8.510e+01 8.986e+01 9.947e+01 1.296e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-24 09:10:53,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2769140.0, ans=0.0
2023-11-24 09:11:19,795 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415400
2023-11-24 09:11:19,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2769273.3333333335, ans=0.0
2023-11-24 09:11:37,607 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6600, loss[loss=0.06564, simple_loss=0.09484, pruned_loss=0.01214, audio_tagging_loss=0.006074, over 15567.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09235, pruned_loss=0.01343, audio_tagging_loss=0.008876, over 3041925.59 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:11:40,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2769406.6666666665, ans=0.1
2023-11-24 09:12:04,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2769540.0, ans=0.1
2023-11-24 09:12:23,243 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415450
2023-11-24 09:12:39,127 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6650, loss[loss=0.07055, simple_loss=0.09618, pruned_loss=0.013, audio_tagging_loss=0.009457, over 16438.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09213, pruned_loss=0.01323, audio_tagging_loss=0.008774, over 3043087.55 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:12:42,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2769740.0, ans=0.125
2023-11-24 09:12:54,686 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.324e+01 8.429e+01 8.957e+01 9.773e+01 1.353e+02, threshold=1.791e+02, percent-clipped=0.0
2023-11-24 09:13:00,154 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.02 vs. limit=10.0
2023-11-24 09:13:24,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2769940.0, ans=0.05
2023-11-24 09:13:25,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415500
2023-11-24 09:13:27,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2769940.0, ans=0.2
2023-11-24 09:13:36,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2770006.6666666665, ans=0.0
2023-11-24 09:13:39,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.28 vs. limit=22.5
2023-11-24 09:13:41,135 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6700, loss[loss=0.05688, simple_loss=0.0788, pruned_loss=0.008236, audio_tagging_loss=0.009247, over 14840.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09212, pruned_loss=0.01315, audio_tagging_loss=0.008806, over 3044797.11 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:14:27,381 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415550
2023-11-24 09:14:29,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2770340.0, ans=0.125
2023-11-24 09:14:43,952 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6750, loss[loss=0.05777, simple_loss=0.08555, pruned_loss=0.008305, audio_tagging_loss=0.006693, over 15094.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09198, pruned_loss=0.01323, audio_tagging_loss=0.008799, over 3033008.98 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:15:00,422 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.302e+01 8.550e+01 9.289e+01 9.814e+01 1.264e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-24 09:15:05,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2770473.3333333335, ans=0.5
2023-11-24 09:15:30,376 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415600
2023-11-24 09:15:37,409 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:15:45,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2770673.3333333335, ans=0.0
2023-11-24 09:15:47,265 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6800, loss[loss=0.06328, simple_loss=0.08317, pruned_loss=0.0133, audio_tagging_loss=0.008393, over 14225.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09383, pruned_loss=0.01352, audio_tagging_loss=0.008684, over 3037760.51 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:15:55,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2770740.0, ans=0.1
2023-11-24 09:16:13,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2770873.3333333335, ans=0.0
2023-11-24 09:16:15,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2770873.3333333335, ans=0.125
2023-11-24 09:16:33,501 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415650
2023-11-24 09:16:48,707 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6850, loss[loss=0.06639, simple_loss=0.08925, pruned_loss=0.01202, audio_tagging_loss=0.009752, over 15194.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09322, pruned_loss=0.01339, audio_tagging_loss=0.008777, over 3036700.55 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:16:55,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2771073.3333333335, ans=0.0
2023-11-24 09:16:55,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2771073.3333333335, ans=0.0
2023-11-24 09:16:55,636 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.20 vs. limit=15.0
2023-11-24 09:17:05,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2771140.0, ans=0.125
2023-11-24 09:17:06,375 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.752e+01 8.373e+01 8.924e+01 9.619e+01 1.192e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-24 09:17:06,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2771140.0, ans=0.125
2023-11-24 09:17:21,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2771206.6666666665, ans=0.0
2023-11-24 09:17:32,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2771273.3333333335, ans=0.125
2023-11-24 09:17:34,746 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415700
2023-11-24 09:17:48,822 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0
2023-11-24 09:17:50,600 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6900, loss[loss=0.05526, simple_loss=0.0711, pruned_loss=0.009154, audio_tagging_loss=0.01055, over 14403.00 frames. ], tot_loss[loss=0.06832, simple_loss=0.09264, pruned_loss=0.01325, audio_tagging_loss=0.00876, over 3036391.90 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:18:02,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2771473.3333333335, ans=0.07
2023-11-24 09:18:03,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2771473.3333333335, ans=0.0
2023-11-24 09:18:18,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.36 vs. limit=12.0
2023-11-24 09:18:36,684 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415750
2023-11-24 09:18:37,839 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 09:18:42,699 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:18:44,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2771673.3333333335, ans=0.125
2023-11-24 09:18:45,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2771673.3333333335, ans=0.125
2023-11-24 09:18:53,284 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 6950, loss[loss=0.07456, simple_loss=0.1049, pruned_loss=0.01409, audio_tagging_loss=0.008053, over 14942.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09309, pruned_loss=0.01329, audio_tagging_loss=0.00863, over 3039294.81 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:19:10,529 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.481e+01 9.077e+01 9.564e+01 1.596e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 09:19:11,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2771806.6666666665, ans=0.2
2023-11-24 09:19:33,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2771940.0, ans=0.025
2023-11-24 09:19:34,774 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.81 vs. limit=10.0
2023-11-24 09:19:39,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415800
2023-11-24 09:19:47,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2772006.6666666665, ans=0.125
2023-11-24 09:19:55,978 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7000, loss[loss=0.07168, simple_loss=0.09435, pruned_loss=0.01647, audio_tagging_loss=0.008036, over 15118.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09263, pruned_loss=0.01333, audio_tagging_loss=0.008669, over 3033466.37 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:19:59,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2772073.3333333335, ans=0.125
2023-11-24 09:20:00,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2772073.3333333335, ans=0.05
2023-11-24 09:20:03,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2772073.3333333335, ans=0.125
2023-11-24 09:20:23,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2772206.6666666665, ans=0.5
2023-11-24 09:20:26,652 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.72 vs. limit=15.0
2023-11-24 09:20:27,531 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:20:36,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2772273.3333333335, ans=0.125
2023-11-24 09:20:42,073 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415850
2023-11-24 09:20:47,515 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.80 vs. limit=15.0
2023-11-24 09:20:57,813 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7050, loss[loss=0.07745, simple_loss=0.1052, pruned_loss=0.01705, audio_tagging_loss=0.007821, over 16483.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09131, pruned_loss=0.01312, audio_tagging_loss=0.008906, over 3035692.25 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:21:02,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2772406.6666666665, ans=0.0
2023-11-24 09:21:15,711 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.245e+01 8.341e+01 9.106e+01 9.596e+01 1.490e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 09:21:18,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=2772473.3333333335, ans=0.1
2023-11-24 09:21:24,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2772540.0, ans=0.125
2023-11-24 09:21:44,348 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415900
2023-11-24 09:22:00,649 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7100, loss[loss=0.06655, simple_loss=0.09091, pruned_loss=0.01221, audio_tagging_loss=0.008889, over 16611.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09184, pruned_loss=0.0131, audio_tagging_loss=0.008954, over 3038326.80 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:22:03,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2772740.0, ans=0.0
2023-11-24 09:22:03,704 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.25 vs. limit=22.5
2023-11-24 09:22:17,022 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.30 vs. limit=15.0
2023-11-24 09:22:27,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2772873.3333333335, ans=0.0
2023-11-24 09:22:27,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2772873.3333333335, ans=0.125
2023-11-24 09:22:46,710 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 415950
2023-11-24 09:22:52,119 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.49 vs. limit=15.0
2023-11-24 09:22:53,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2773006.6666666665, ans=0.2
2023-11-24 09:22:53,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2773006.6666666665, ans=0.2
2023-11-24 09:23:02,694 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7150, loss[loss=0.09112, simple_loss=0.1295, pruned_loss=0.01944, audio_tagging_loss=0.006948, over 15567.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09087, pruned_loss=0.01311, audio_tagging_loss=0.00902, over 3045303.90 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:23:12,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2773073.3333333335, ans=0.125
2023-11-24 09:23:19,806 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.644e+01 9.348e+01 1.022e+02 1.334e+02, threshold=1.870e+02, percent-clipped=0.0
2023-11-24 09:23:24,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2773140.0, ans=0.1
2023-11-24 09:23:27,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2773206.6666666665, ans=0.125
2023-11-24 09:23:35,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2773206.6666666665, ans=0.125
2023-11-24 09:23:42,938 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.11 vs. limit=6.0
2023-11-24 09:23:48,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416000
2023-11-24 09:23:49,539 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-416000.pt
2023-11-24 09:24:09,379 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7200, loss[loss=0.05534, simple_loss=0.0786, pruned_loss=0.007605, audio_tagging_loss=0.008439, over 14518.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09194, pruned_loss=0.01316, audio_tagging_loss=0.008995, over 3044717.35 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:24:10,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2773406.6666666665, ans=0.025
2023-11-24 09:24:50,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2773606.6666666665, ans=0.125
2023-11-24 09:24:55,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416050
2023-11-24 09:25:09,354 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.09 vs. limit=22.5
2023-11-24 09:25:12,606 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7250, loss[loss=0.05687, simple_loss=0.07282, pruned_loss=0.01067, audio_tagging_loss=0.009785, over 15248.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09168, pruned_loss=0.01314, audio_tagging_loss=0.009107, over 3052022.83 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:25:20,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2773740.0, ans=0.0
2023-11-24 09:25:28,836 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.27 vs. limit=15.0
2023-11-24 09:25:29,222 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.544e+01 8.813e+01 9.334e+01 1.034e+02 1.372e+02, threshold=1.867e+02, percent-clipped=0.0
2023-11-24 09:25:33,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2773806.6666666665, ans=10.0
2023-11-24 09:25:54,525 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.21 vs. limit=15.0
2023-11-24 09:25:58,719 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416100
2023-11-24 09:26:13,976 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7300, loss[loss=0.0569, simple_loss=0.08104, pruned_loss=0.008161, audio_tagging_loss=0.008218, over 15367.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09131, pruned_loss=0.01315, audio_tagging_loss=0.009173, over 3052992.78 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:26:14,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2774073.3333333335, ans=0.125
2023-11-24 09:26:26,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2774140.0, ans=0.1
2023-11-24 09:26:41,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2774206.6666666665, ans=0.125
2023-11-24 09:26:41,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2774206.6666666665, ans=0.0
2023-11-24 09:26:50,095 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.19 vs. limit=15.0
2023-11-24 09:26:54,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2774273.3333333335, ans=0.125
2023-11-24 09:26:56,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2774273.3333333335, ans=0.125
2023-11-24 09:26:58,192 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.04 vs. limit=15.0
2023-11-24 09:26:59,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2774273.3333333335, ans=0.125
2023-11-24 09:27:00,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416150
2023-11-24 09:27:16,241 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7350, loss[loss=0.06338, simple_loss=0.08226, pruned_loss=0.01245, audio_tagging_loss=0.009804, over 16383.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.09166, pruned_loss=0.01333, audio_tagging_loss=0.009043, over 3056070.67 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:27:18,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2774406.6666666665, ans=0.125
2023-11-24 09:27:26,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2774406.6666666665, ans=0.125
2023-11-24 09:27:34,372 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.268e+01 8.604e+01 9.108e+01 1.008e+02 1.406e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 09:27:46,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=2774540.0, ans=0.05
2023-11-24 09:27:53,733 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:28:02,574 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416200
2023-11-24 09:28:04,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2774606.6666666665, ans=0.125
2023-11-24 09:28:19,625 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7400, loss[loss=0.06938, simple_loss=0.09392, pruned_loss=0.01445, audio_tagging_loss=0.007979, over 15333.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09055, pruned_loss=0.01314, audio_tagging_loss=0.009004, over 3051137.56 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:28:25,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2774740.0, ans=0.2
2023-11-24 09:28:37,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2774806.6666666665, ans=0.2
2023-11-24 09:28:37,530 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.94 vs. limit=15.0
2023-11-24 09:29:02,055 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.68 vs. limit=10.0
2023-11-24 09:29:05,726 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416250
2023-11-24 09:29:08,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2775006.6666666665, ans=0.125
2023-11-24 09:29:21,046 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7450, loss[loss=0.07614, simple_loss=0.1092, pruned_loss=0.01324, audio_tagging_loss=0.008308, over 15245.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09111, pruned_loss=0.0132, audio_tagging_loss=0.008898, over 3053126.21 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:29:28,643 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.80 vs. limit=15.0
2023-11-24 09:29:38,157 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.622e+01 8.229e+01 9.115e+01 9.575e+01 1.366e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 09:29:39,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2775140.0, ans=0.0
2023-11-24 09:30:07,454 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416300
2023-11-24 09:30:22,796 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7500, loss[loss=0.07729, simple_loss=0.1025, pruned_loss=0.01995, audio_tagging_loss=0.006107, over 15781.00 frames. ], tot_loss[loss=0.0666, simple_loss=0.08946, pruned_loss=0.01296, audio_tagging_loss=0.008911, over 3052048.14 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:30:38,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2775473.3333333335, ans=0.125
2023-11-24 09:30:51,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2775540.0, ans=0.125
2023-11-24 09:31:03,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2775606.6666666665, ans=0.1
2023-11-24 09:31:08,210 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416350
2023-11-24 09:31:08,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2775606.6666666665, ans=0.125
2023-11-24 09:31:10,946 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:31:18,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2775673.3333333335, ans=0.0
2023-11-24 09:31:25,174 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7550, loss[loss=0.06567, simple_loss=0.08761, pruned_loss=0.01331, audio_tagging_loss=0.008555, over 15394.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09028, pruned_loss=0.01312, audio_tagging_loss=0.008842, over 3053580.95 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:31:41,459 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.230e+01 8.548e+01 9.078e+01 9.572e+01 1.867e+02, threshold=1.816e+02, percent-clipped=1.0
2023-11-24 09:32:03,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2775940.0, ans=0.0
2023-11-24 09:32:08,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2775940.0, ans=0.125
2023-11-24 09:32:11,183 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416400
2023-11-24 09:32:15,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2776006.6666666665, ans=0.2
2023-11-24 09:32:17,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2776006.6666666665, ans=0.125
2023-11-24 09:32:18,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2776006.6666666665, ans=0.0
2023-11-24 09:32:26,773 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7600, loss[loss=0.05581, simple_loss=0.07233, pruned_loss=0.01124, audio_tagging_loss=0.008407, over 14947.00 frames. ], tot_loss[loss=0.06673, simple_loss=0.0898, pruned_loss=0.01295, audio_tagging_loss=0.008881, over 3045995.44 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:32:38,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2776140.0, ans=0.1
2023-11-24 09:32:52,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2776206.6666666665, ans=0.125
2023-11-24 09:33:09,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2776273.3333333335, ans=0.125
2023-11-24 09:33:12,571 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416450
2023-11-24 09:33:24,263 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2776340.0, ans=0.035
2023-11-24 09:33:25,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2776340.0, ans=0.125
2023-11-24 09:33:27,748 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7650, loss[loss=0.06702, simple_loss=0.1013, pruned_loss=0.01153, audio_tagging_loss=0.004828, over 15279.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.08989, pruned_loss=0.0128, audio_tagging_loss=0.008882, over 3042901.42 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:33:29,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2776406.6666666665, ans=0.1
2023-11-24 09:33:30,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2776406.6666666665, ans=0.0
2023-11-24 09:33:48,912 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.062e+01 8.355e+01 9.042e+01 9.809e+01 1.325e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 09:33:49,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2776473.3333333335, ans=0.125
2023-11-24 09:33:58,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2776540.0, ans=0.125
2023-11-24 09:33:58,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2776540.0, ans=0.0
2023-11-24 09:34:07,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2776606.6666666665, ans=0.125
2023-11-24 09:34:13,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416500
2023-11-24 09:34:13,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2776606.6666666665, ans=0.125
2023-11-24 09:34:30,134 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7700, loss[loss=0.06567, simple_loss=0.08935, pruned_loss=0.01323, audio_tagging_loss=0.007756, over 14538.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.08979, pruned_loss=0.01284, audio_tagging_loss=0.008859, over 3041729.44 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:34:31,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2776740.0, ans=0.125
2023-11-24 09:35:03,555 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.00 vs. limit=15.0
2023-11-24 09:35:12,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2776940.0, ans=0.125
2023-11-24 09:35:12,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2776940.0, ans=0.0
2023-11-24 09:35:14,389 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416550
2023-11-24 09:35:14,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2776940.0, ans=0.1
2023-11-24 09:35:19,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2777006.6666666665, ans=0.125
2023-11-24 09:35:31,205 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7750, loss[loss=0.06687, simple_loss=0.08755, pruned_loss=0.01438, audio_tagging_loss=0.008714, over 15414.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.08981, pruned_loss=0.01288, audio_tagging_loss=0.008889, over 3049842.67 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:35:37,842 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.85 vs. limit=15.0
2023-11-24 09:35:49,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2777140.0, ans=0.125
2023-11-24 09:35:50,351 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.633e+01 8.304e+01 9.008e+01 9.844e+01 1.304e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-24 09:35:58,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2777206.6666666665, ans=0.0
2023-11-24 09:35:59,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2777206.6666666665, ans=0.025
2023-11-24 09:36:06,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2777206.6666666665, ans=0.2
2023-11-24 09:36:15,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2777273.3333333335, ans=0.2
2023-11-24 09:36:17,198 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416600
2023-11-24 09:36:19,391 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.45 vs. limit=15.0
2023-11-24 09:36:32,692 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.10 vs. limit=10.0
2023-11-24 09:36:33,302 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7800, loss[loss=0.06181, simple_loss=0.09037, pruned_loss=0.0105, audio_tagging_loss=0.006126, over 15172.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09114, pruned_loss=0.01305, audio_tagging_loss=0.008823, over 3047430.84 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:37:15,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2777606.6666666665, ans=0.125
2023-11-24 09:37:19,380 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416650
2023-11-24 09:37:24,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2777673.3333333335, ans=0.125
2023-11-24 09:37:24,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2777673.3333333335, ans=0.1
2023-11-24 09:37:31,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2777673.3333333335, ans=0.1
2023-11-24 09:37:35,054 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7850, loss[loss=0.06841, simple_loss=0.08561, pruned_loss=0.01589, audio_tagging_loss=0.00971, over 14736.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09101, pruned_loss=0.01311, audio_tagging_loss=0.008814, over 3049276.28 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:37:40,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2777740.0, ans=0.0
2023-11-24 09:37:52,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2777806.6666666665, ans=0.1
2023-11-24 09:37:54,939 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=11.62 vs. limit=15.0
2023-11-24 09:37:55,541 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.322e+01 8.418e+01 9.005e+01 9.795e+01 1.227e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 09:38:03,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2777873.3333333335, ans=0.125
2023-11-24 09:38:21,188 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416700
2023-11-24 09:38:22,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2777940.0, ans=0.2
2023-11-24 09:38:29,490 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.58 vs. limit=15.0
2023-11-24 09:38:30,706 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.58 vs. limit=15.0
2023-11-24 09:38:35,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2778006.6666666665, ans=0.2
2023-11-24 09:38:37,569 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7900, loss[loss=0.0475, simple_loss=0.06067, pruned_loss=0.005751, audio_tagging_loss=0.01142, over 15893.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.09005, pruned_loss=0.01289, audio_tagging_loss=0.00891, over 3044488.60 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:38:39,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2778073.3333333335, ans=0.125
2023-11-24 09:38:51,111 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.86 vs. limit=22.5
2023-11-24 09:39:01,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2778206.6666666665, ans=0.0
2023-11-24 09:39:13,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2778273.3333333335, ans=0.125
2023-11-24 09:39:23,512 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416750
2023-11-24 09:39:38,829 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 7950, loss[loss=0.05916, simple_loss=0.06982, pruned_loss=0.01354, audio_tagging_loss=0.0107, over 13822.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09004, pruned_loss=0.01284, audio_tagging_loss=0.008956, over 3043388.89 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:39:45,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2778406.6666666665, ans=0.0
2023-11-24 09:39:53,590 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 09:39:58,837 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.977e+01 8.587e+01 9.115e+01 9.680e+01 1.233e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 09:40:12,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2778540.0, ans=0.125
2023-11-24 09:40:16,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2778606.6666666665, ans=15.0
2023-11-24 09:40:25,206 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416800
2023-11-24 09:40:41,547 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8000, loss[loss=0.07231, simple_loss=0.09933, pruned_loss=0.01507, audio_tagging_loss=0.007576, over 15303.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09021, pruned_loss=0.01279, audio_tagging_loss=0.009083, over 3051788.71 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:40:57,971 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.whiten.whitening_limit, batch_count=2778806.6666666665, ans=15.0
2023-11-24 09:41:11,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2778873.3333333335, ans=0.125
2023-11-24 09:41:26,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2778940.0, ans=0.125
2023-11-24 09:41:27,984 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416850
2023-11-24 09:41:39,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2779006.6666666665, ans=0.125
2023-11-24 09:41:40,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.38 vs. limit=10.0
2023-11-24 09:41:44,425 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8050, loss[loss=0.05536, simple_loss=0.07367, pruned_loss=0.009896, audio_tagging_loss=0.008629, over 14567.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.08973, pruned_loss=0.01279, audio_tagging_loss=0.009158, over 3046599.36 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:42:05,158 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.991e+01 8.535e+01 9.115e+01 9.754e+01 2.166e+02, threshold=1.823e+02, percent-clipped=1.0
2023-11-24 09:42:27,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2779273.3333333335, ans=0.0
2023-11-24 09:42:29,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2779273.3333333335, ans=0.0
2023-11-24 09:42:30,573 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416900
2023-11-24 09:42:30,705 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:42:46,449 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8100, loss[loss=0.06952, simple_loss=0.1033, pruned_loss=0.009069, audio_tagging_loss=0.008779, over 15935.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09036, pruned_loss=0.01297, audio_tagging_loss=0.009158, over 3052234.78 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:42:47,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2779406.6666666665, ans=0.0
2023-11-24 09:43:12,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2779540.0, ans=0.2
2023-11-24 09:43:13,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.01 vs. limit=22.5
2023-11-24 09:43:15,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2779540.0, ans=0.0
2023-11-24 09:43:26,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.60 vs. limit=12.0
2023-11-24 09:43:32,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 416950
2023-11-24 09:43:47,842 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8150, loss[loss=0.07328, simple_loss=0.09396, pruned_loss=0.01744, audio_tagging_loss=0.008863, over 14681.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09188, pruned_loss=0.01326, audio_tagging_loss=0.008951, over 3055689.05 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:43:51,263 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.51 vs. limit=15.0
2023-11-24 09:43:53,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2779740.0, ans=0.125
2023-11-24 09:44:09,535 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.453e+01 8.620e+01 9.310e+01 1.002e+02 1.330e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-24 09:44:18,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2779873.3333333335, ans=0.0
2023-11-24 09:44:20,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2779873.3333333335, ans=0.1
2023-11-24 09:44:34,108 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417000
2023-11-24 09:44:50,744 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8200, loss[loss=0.04419, simple_loss=0.06167, pruned_loss=0.004138, audio_tagging_loss=0.009219, over 15783.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09141, pruned_loss=0.01314, audio_tagging_loss=0.008896, over 3056241.84 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:44:51,961 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 09:44:56,082 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.62 vs. limit=15.0
2023-11-24 09:45:04,966 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.56 vs. limit=6.0
2023-11-24 09:45:16,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2780206.6666666665, ans=0.0
2023-11-24 09:45:17,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2780206.6666666665, ans=0.125
2023-11-24 09:45:37,023 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417050
2023-11-24 09:45:52,422 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8250, loss[loss=0.07173, simple_loss=0.09931, pruned_loss=0.0148, audio_tagging_loss=0.007276, over 14277.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09156, pruned_loss=0.01325, audio_tagging_loss=0.008794, over 3054089.34 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:46:11,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2780473.3333333335, ans=0.0
2023-11-24 09:46:13,773 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.228e+01 8.658e+01 9.148e+01 9.760e+01 1.278e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 09:46:22,607 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.74 vs. limit=15.0
2023-11-24 09:46:31,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2780606.6666666665, ans=0.125
2023-11-24 09:46:38,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417100
2023-11-24 09:46:53,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2780740.0, ans=0.0
2023-11-24 09:46:54,580 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8300, loss[loss=0.08168, simple_loss=0.1197, pruned_loss=0.01419, audio_tagging_loss=0.007625, over 15194.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09104, pruned_loss=0.01322, audio_tagging_loss=0.008825, over 3054660.76 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:47:35,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.36 vs. limit=15.0
2023-11-24 09:47:41,291 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417150
2023-11-24 09:47:44,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2781006.6666666665, ans=0.1
2023-11-24 09:47:51,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2781006.6666666665, ans=0.125
2023-11-24 09:47:58,468 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8350, loss[loss=0.06178, simple_loss=0.07834, pruned_loss=0.01264, audio_tagging_loss=0.009967, over 15537.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09121, pruned_loss=0.01324, audio_tagging_loss=0.008714, over 3056147.35 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:48:05,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2781073.3333333335, ans=0.0
2023-11-24 09:48:06,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2781073.3333333335, ans=0.0
2023-11-24 09:48:11,550 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 09:48:18,476 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.379e+01 9.010e+01 9.696e+01 1.228e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-24 09:48:44,408 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417200
2023-11-24 09:48:47,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2781340.0, ans=0.0
2023-11-24 09:49:00,076 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8400, loss[loss=0.06742, simple_loss=0.08934, pruned_loss=0.01168, audio_tagging_loss=0.01108, over 14758.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09045, pruned_loss=0.01307, audio_tagging_loss=0.008768, over 3050470.79 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:49:09,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2781406.6666666665, ans=0.1
2023-11-24 09:49:17,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2781473.3333333335, ans=0.125
2023-11-24 09:49:46,325 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417250
2023-11-24 09:49:47,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2781606.6666666665, ans=0.07
2023-11-24 09:49:50,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2781673.3333333335, ans=0.0
2023-11-24 09:50:02,448 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8450, loss[loss=0.0739, simple_loss=0.0982, pruned_loss=0.01697, audio_tagging_loss=0.007832, over 15940.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09061, pruned_loss=0.01304, audio_tagging_loss=0.008833, over 3054084.78 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:50:06,693 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.10 vs. limit=15.0
2023-11-24 09:50:24,536 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.178e+01 8.593e+01 9.155e+01 9.677e+01 1.308e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 09:50:42,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2781940.0, ans=0.0
2023-11-24 09:50:47,582 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.56 vs. limit=15.0
2023-11-24 09:50:49,333 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417300
2023-11-24 09:50:58,818 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.06 vs. limit=12.0
2023-11-24 09:51:03,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2782006.6666666665, ans=0.125
2023-11-24 09:51:06,318 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8500, loss[loss=0.04349, simple_loss=0.05265, pruned_loss=0.006831, audio_tagging_loss=0.01033, over 14207.00 frames. ], tot_loss[loss=0.06618, simple_loss=0.08894, pruned_loss=0.01278, audio_tagging_loss=0.008921, over 3051007.37 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:51:10,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2782073.3333333335, ans=0.2
2023-11-24 09:51:11,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2782073.3333333335, ans=0.1
2023-11-24 09:51:18,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2782140.0, ans=0.125
2023-11-24 09:51:22,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2782140.0, ans=0.0
2023-11-24 09:51:28,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2782140.0, ans=0.125
2023-11-24 09:51:52,028 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417350
2023-11-24 09:52:07,822 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8550, loss[loss=0.06964, simple_loss=0.09467, pruned_loss=0.01427, audio_tagging_loss=0.008033, over 14412.00 frames. ], tot_loss[loss=0.0666, simple_loss=0.0899, pruned_loss=0.01282, audio_tagging_loss=0.008826, over 3049281.68 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:52:16,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2782406.6666666665, ans=0.125
2023-11-24 09:52:22,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2782473.3333333335, ans=0.1
2023-11-24 09:52:28,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2782473.3333333335, ans=0.125
2023-11-24 09:52:29,799 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.338e+01 8.685e+01 9.326e+01 1.025e+02 1.164e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 09:52:31,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2782540.0, ans=0.0
2023-11-24 09:52:36,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2782540.0, ans=0.1
2023-11-24 09:52:47,873 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.16 vs. limit=10.0
2023-11-24 09:52:54,250 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417400
2023-11-24 09:53:10,017 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8600, loss[loss=0.05789, simple_loss=0.06718, pruned_loss=0.01221, audio_tagging_loss=0.01209, over 15172.00 frames. ], tot_loss[loss=0.0662, simple_loss=0.0893, pruned_loss=0.01269, audio_tagging_loss=0.008862, over 3043853.44 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:53:19,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2782740.0, ans=0.0
2023-11-24 09:53:22,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2782806.6666666665, ans=0.2
2023-11-24 09:53:27,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.54 vs. limit=15.0
2023-11-24 09:53:35,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2782873.3333333335, ans=0.125
2023-11-24 09:53:43,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2782873.3333333335, ans=0.0
2023-11-24 09:53:56,307 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417450
2023-11-24 09:53:59,346 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.61 vs. limit=10.0
2023-11-24 09:54:12,799 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8650, loss[loss=0.06785, simple_loss=0.09734, pruned_loss=0.01134, audio_tagging_loss=0.007839, over 15077.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09103, pruned_loss=0.01296, audio_tagging_loss=0.00878, over 3043157.65 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:54:26,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2783140.0, ans=0.0
2023-11-24 09:54:35,508 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.880e+01 8.560e+01 9.076e+01 1.003e+02 1.218e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 09:54:59,566 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417500
2023-11-24 09:54:59,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2783273.3333333335, ans=0.125
2023-11-24 09:55:05,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2783340.0, ans=0.125
2023-11-24 09:55:16,158 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8700, loss[loss=0.08205, simple_loss=0.1107, pruned_loss=0.01672, audio_tagging_loss=0.009978, over 15221.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09115, pruned_loss=0.01285, audio_tagging_loss=0.008892, over 3047989.53 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:55:34,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2783473.3333333335, ans=0.0
2023-11-24 09:55:39,574 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2783540.0, ans=0.125
2023-11-24 09:55:51,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2783540.0, ans=0.0
2023-11-24 09:56:02,444 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417550
2023-11-24 09:56:13,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2783673.3333333335, ans=0.0
2023-11-24 09:56:17,915 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8750, loss[loss=0.06979, simple_loss=0.08982, pruned_loss=0.01597, audio_tagging_loss=0.008918, over 14516.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09144, pruned_loss=0.01302, audio_tagging_loss=0.009047, over 3042419.60 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 09:56:19,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2783740.0, ans=0.0
2023-11-24 09:56:19,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2783740.0, ans=0.125
2023-11-24 09:56:22,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2783740.0, ans=0.035
2023-11-24 09:56:40,386 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.442e+01 8.767e+01 9.281e+01 1.037e+02 1.287e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-24 09:56:56,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2783940.0, ans=0.125
2023-11-24 09:56:58,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2783940.0, ans=0.0
2023-11-24 09:57:03,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417600
2023-11-24 09:57:20,230 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8800, loss[loss=0.05201, simple_loss=0.06182, pruned_loss=0.009561, audio_tagging_loss=0.01154, over 14723.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09097, pruned_loss=0.01307, audio_tagging_loss=0.009251, over 3042179.76 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:57:30,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2784073.3333333335, ans=0.0
2023-11-24 09:57:37,243 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.13 vs. limit=22.5
2023-11-24 09:57:39,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2784140.0, ans=0.0
2023-11-24 09:57:43,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2784206.6666666665, ans=0.125
2023-11-24 09:57:47,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2784206.6666666665, ans=0.125
2023-11-24 09:57:57,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2784273.3333333335, ans=0.125
2023-11-24 09:58:05,991 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417650
2023-11-24 09:58:06,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2784273.3333333335, ans=0.0
2023-11-24 09:58:15,641 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.27 vs. limit=6.0
2023-11-24 09:58:21,982 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8850, loss[loss=0.07465, simple_loss=0.1108, pruned_loss=0.01281, audio_tagging_loss=0.006433, over 15448.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09205, pruned_loss=0.0133, audio_tagging_loss=0.009234, over 3048134.69 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:58:25,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2784406.6666666665, ans=0.125
2023-11-24 09:58:25,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2784406.6666666665, ans=0.0
2023-11-24 09:58:32,834 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 09:58:34,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2784473.3333333335, ans=0.2
2023-11-24 09:58:34,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2784473.3333333335, ans=0.0
2023-11-24 09:58:42,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2784473.3333333335, ans=0.125
2023-11-24 09:58:43,357 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.018e+01 8.550e+01 9.128e+01 9.722e+01 1.464e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 09:58:53,551 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.67 vs. limit=10.0
2023-11-24 09:59:07,710 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417700
2023-11-24 09:59:15,549 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.14 vs. limit=22.5
2023-11-24 09:59:17,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2784673.3333333335, ans=0.125
2023-11-24 09:59:23,018 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8900, loss[loss=0.0892, simple_loss=0.1235, pruned_loss=0.01873, audio_tagging_loss=0.008745, over 16749.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.0916, pruned_loss=0.01331, audio_tagging_loss=0.009105, over 3044949.20 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 09:59:32,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2784740.0, ans=0.125
2023-11-24 09:59:44,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2784806.6666666665, ans=0.125
2023-11-24 09:59:51,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2784873.3333333335, ans=0.125
2023-11-24 09:59:55,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2784873.3333333335, ans=0.125
2023-11-24 10:00:00,397 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:00:02,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2784940.0, ans=0.125
2023-11-24 10:00:07,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2784940.0, ans=0.1
2023-11-24 10:00:08,507 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417750
2023-11-24 10:00:08,744 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:00:12,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2785006.6666666665, ans=0.0
2023-11-24 10:00:15,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2785006.6666666665, ans=0.5
2023-11-24 10:00:16,247 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.76 vs. limit=22.5
2023-11-24 10:00:21,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2785006.6666666665, ans=0.125
2023-11-24 10:00:24,343 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 8950, loss[loss=0.06752, simple_loss=0.08434, pruned_loss=0.01604, audio_tagging_loss=0.009301, over 14666.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09116, pruned_loss=0.01323, audio_tagging_loss=0.009018, over 3038416.90 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:00:45,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2785140.0, ans=0.125
2023-11-24 10:00:47,811 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.675e+01 8.510e+01 9.195e+01 1.012e+02 1.504e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 10:00:56,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.21 vs. limit=15.0
2023-11-24 10:01:01,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2785273.3333333335, ans=0.0
2023-11-24 10:01:09,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417800
2023-11-24 10:01:20,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2785340.0, ans=0.1
2023-11-24 10:01:24,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2785340.0, ans=0.1
2023-11-24 10:01:26,224 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9000, loss[loss=0.05731, simple_loss=0.07733, pruned_loss=0.008623, audio_tagging_loss=0.01002, over 15872.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.0914, pruned_loss=0.01336, audio_tagging_loss=0.008978, over 3040741.61 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:01:26,227 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 10:01:55,964 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.6529, 2.9932, 2.7920, 2.7239, 3.3658, 3.3614, 3.2258, 3.5594],
       device='cuda:0')
2023-11-24 10:02:04,060 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.5901, 3.7065, 3.9266, 3.4770], device='cuda:0')
2023-11-24 10:02:08,540 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4658, 3.2959, 3.7074, 3.5289], device='cuda:0')
2023-11-24 10:02:10,234 INFO [train_asr.py:1253] (0/4) Epoch 35, validation: loss=0.05875, simple_loss=0.05079, pruned_loss=0.005122, audio_tagging_loss=0.02823, over 4681554.00 frames. 
2023-11-24 10:02:10,235 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 10:02:10,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2785406.6666666665, ans=0.0
2023-11-24 10:02:11,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2785406.6666666665, ans=0.1
2023-11-24 10:02:16,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2785406.6666666665, ans=0.125
2023-11-24 10:02:38,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2785540.0, ans=0.125
2023-11-24 10:02:48,555 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.16 vs. limit=22.5
2023-11-24 10:02:51,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2785606.6666666665, ans=0.125
2023-11-24 10:02:56,309 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417850
2023-11-24 10:03:00,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2785673.3333333335, ans=0.1
2023-11-24 10:03:06,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2785673.3333333335, ans=0.0
2023-11-24 10:03:12,422 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9050, loss[loss=0.04745, simple_loss=0.06407, pruned_loss=0.006316, audio_tagging_loss=0.009103, over 15651.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09149, pruned_loss=0.01341, audio_tagging_loss=0.008945, over 3038148.12 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:03:36,291 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.575e+01 9.161e+01 9.743e+01 1.218e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 10:03:53,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2785940.0, ans=0.0
2023-11-24 10:03:58,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417900
2023-11-24 10:04:14,929 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9100, loss[loss=0.07676, simple_loss=0.09987, pruned_loss=0.0164, audio_tagging_loss=0.01042, over 14898.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.0914, pruned_loss=0.0134, audio_tagging_loss=0.008908, over 3041163.91 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:04:23,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2786073.3333333335, ans=0.125
2023-11-24 10:04:29,744 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.63 vs. limit=15.0
2023-11-24 10:04:36,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2786140.0, ans=0.125
2023-11-24 10:04:45,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2786206.6666666665, ans=0.0
2023-11-24 10:04:49,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2786206.6666666665, ans=0.0
2023-11-24 10:04:55,714 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:04:59,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2786273.3333333335, ans=0.0
2023-11-24 10:05:00,728 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 417950
2023-11-24 10:05:00,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2786273.3333333335, ans=0.0
2023-11-24 10:05:10,595 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=22.5
2023-11-24 10:05:15,919 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9150, loss[loss=0.0726, simple_loss=0.1029, pruned_loss=0.01324, audio_tagging_loss=0.007935, over 13951.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09142, pruned_loss=0.01341, audio_tagging_loss=0.008851, over 3043960.39 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:05:32,776 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.65 vs. limit=22.5
2023-11-24 10:05:39,648 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.159e+01 8.404e+01 9.029e+01 9.911e+01 1.244e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 10:05:55,239 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.57 vs. limit=15.0
2023-11-24 10:05:57,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2786606.6666666665, ans=0.0
2023-11-24 10:06:00,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2786606.6666666665, ans=0.0
2023-11-24 10:06:00,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2786606.6666666665, ans=0.2
2023-11-24 10:06:01,931 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418000
2023-11-24 10:06:18,312 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9200, loss[loss=0.06046, simple_loss=0.07004, pruned_loss=0.01645, audio_tagging_loss=0.008992, over 14422.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09144, pruned_loss=0.01354, audio_tagging_loss=0.008817, over 3045418.74 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:06:25,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2786740.0, ans=0.0
2023-11-24 10:06:28,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2786740.0, ans=0.125
2023-11-24 10:06:37,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2786806.6666666665, ans=0.09899494936611666
2023-11-24 10:07:04,199 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418050
2023-11-24 10:07:15,128 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.44 vs. limit=15.0
2023-11-24 10:07:20,425 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9250, loss[loss=0.0722, simple_loss=0.1054, pruned_loss=0.01242, audio_tagging_loss=0.0071, over 14664.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09139, pruned_loss=0.01355, audio_tagging_loss=0.008817, over 3046195.75 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:07:37,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2787140.0, ans=0.0
2023-11-24 10:07:43,439 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.222e+01 8.443e+01 8.965e+01 9.843e+01 1.266e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 10:08:02,088 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.73 vs. limit=15.0
2023-11-24 10:08:05,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2787273.3333333335, ans=0.125
2023-11-24 10:08:06,272 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418100
2023-11-24 10:08:13,389 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.94 vs. limit=15.0
2023-11-24 10:08:22,355 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9300, loss[loss=0.05885, simple_loss=0.08218, pruned_loss=0.009951, audio_tagging_loss=0.007802, over 14954.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09029, pruned_loss=0.01322, audio_tagging_loss=0.008956, over 3046566.58 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:08:36,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2787473.3333333335, ans=0.125
2023-11-24 10:08:54,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2787540.0, ans=0.1
2023-11-24 10:09:03,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2787606.6666666665, ans=0.2
2023-11-24 10:09:06,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2787606.6666666665, ans=0.0
2023-11-24 10:09:08,356 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418150
2023-11-24 10:09:15,464 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2787673.3333333335, ans=0.025
2023-11-24 10:09:22,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2787740.0, ans=0.07
2023-11-24 10:09:23,909 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9350, loss[loss=0.04847, simple_loss=0.059, pruned_loss=0.009132, audio_tagging_loss=0.009836, over 13980.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09037, pruned_loss=0.01324, audio_tagging_loss=0.008933, over 3044817.43 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:09:48,172 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.653e+01 8.452e+01 8.975e+01 9.526e+01 2.573e+02, threshold=1.795e+02, percent-clipped=1.0
2023-11-24 10:09:51,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2787873.3333333335, ans=0.125
2023-11-24 10:10:09,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418200
2023-11-24 10:10:19,164 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.08 vs. limit=15.0
2023-11-24 10:10:22,056 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.13 vs. limit=22.5
2023-11-24 10:10:26,244 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9400, loss[loss=0.06866, simple_loss=0.09321, pruned_loss=0.01096, audio_tagging_loss=0.01109, over 16481.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09079, pruned_loss=0.0132, audio_tagging_loss=0.009028, over 3052152.69 frames. ], batch size: 64, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:10:26,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2788073.3333333335, ans=0.125
2023-11-24 10:10:28,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2788073.3333333335, ans=0.125
2023-11-24 10:10:32,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2788073.3333333335, ans=0.1
2023-11-24 10:10:39,617 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.01 vs. limit=6.0
2023-11-24 10:10:41,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2788140.0, ans=0.09899494936611666
2023-11-24 10:10:47,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2788140.0, ans=0.125
2023-11-24 10:10:53,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2788206.6666666665, ans=0.125
2023-11-24 10:11:12,722 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418250
2023-11-24 10:11:25,992 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:11:27,482 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.88 vs. limit=22.5
2023-11-24 10:11:28,920 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9450, loss[loss=0.07357, simple_loss=0.1039, pruned_loss=0.01517, audio_tagging_loss=0.006431, over 15811.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09121, pruned_loss=0.01334, audio_tagging_loss=0.009028, over 3052337.93 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:11:43,851 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:11:54,121 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.573e+01 9.142e+01 1.013e+02 1.307e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 10:12:08,485 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.03 vs. limit=22.5
2023-11-24 10:12:14,912 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418300
2023-11-24 10:12:15,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2788606.6666666665, ans=0.125
2023-11-24 10:12:20,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2788673.3333333335, ans=0.2
2023-11-24 10:12:22,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2788673.3333333335, ans=0.0
2023-11-24 10:12:24,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2788673.3333333335, ans=0.0
2023-11-24 10:12:30,833 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9500, loss[loss=0.08736, simple_loss=0.1263, pruned_loss=0.01817, audio_tagging_loss=0.006044, over 15889.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09108, pruned_loss=0.01329, audio_tagging_loss=0.009054, over 3047490.41 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:12:34,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2788740.0, ans=0.125
2023-11-24 10:12:35,484 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.97 vs. limit=15.0
2023-11-24 10:12:58,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2788873.3333333335, ans=0.0
2023-11-24 10:13:00,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2788873.3333333335, ans=0.125
2023-11-24 10:13:14,967 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:13:15,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2788940.0, ans=0.125
2023-11-24 10:13:17,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418350
2023-11-24 10:13:34,256 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9550, loss[loss=0.05058, simple_loss=0.06657, pruned_loss=0.006441, audio_tagging_loss=0.01085, over 15076.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09165, pruned_loss=0.01346, audio_tagging_loss=0.009129, over 3051502.93 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:13:40,381 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2789073.3333333335, ans=0.125
2023-11-24 10:13:47,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2789140.0, ans=0.125
2023-11-24 10:13:58,336 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.515e+01 8.574e+01 9.293e+01 1.001e+02 1.205e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 10:14:11,476 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.07 vs. limit=15.0
2023-11-24 10:14:13,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2789273.3333333335, ans=0.05
2023-11-24 10:14:20,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418400
2023-11-24 10:14:22,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2789273.3333333335, ans=0.2
2023-11-24 10:14:36,069 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9600, loss[loss=0.05751, simple_loss=0.07758, pruned_loss=0.008982, audio_tagging_loss=0.009737, over 15479.00 frames. ], tot_loss[loss=0.06845, simple_loss=0.09193, pruned_loss=0.01336, audio_tagging_loss=0.009126, over 3049350.01 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:14:40,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2789406.6666666665, ans=0.0
2023-11-24 10:14:55,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2789473.3333333335, ans=0.1
2023-11-24 10:15:16,690 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2789606.6666666665, ans=0.125
2023-11-24 10:15:17,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2789606.6666666665, ans=0.1
2023-11-24 10:15:22,514 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418450
2023-11-24 10:15:26,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2789673.3333333335, ans=0.0
2023-11-24 10:15:38,344 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9650, loss[loss=0.07788, simple_loss=0.1043, pruned_loss=0.01899, audio_tagging_loss=0.006733, over 14958.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09092, pruned_loss=0.0131, audio_tagging_loss=0.00915, over 3043465.04 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:16:04,132 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.403e+01 8.395e+01 8.995e+01 9.967e+01 1.274e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 10:16:15,576 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.30 vs. limit=15.0
2023-11-24 10:16:19,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2789940.0, ans=0.1
2023-11-24 10:16:20,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.08 vs. limit=6.0
2023-11-24 10:16:21,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2789940.0, ans=0.0
2023-11-24 10:16:25,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418500
2023-11-24 10:16:31,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2790006.6666666665, ans=0.2
2023-11-24 10:16:38,625 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.94 vs. limit=15.0
2023-11-24 10:16:42,595 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9700, loss[loss=0.06882, simple_loss=0.08938, pruned_loss=0.01721, audio_tagging_loss=0.006928, over 14685.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09097, pruned_loss=0.01327, audio_tagging_loss=0.009011, over 3044164.21 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:16:44,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2790073.3333333335, ans=0.0
2023-11-24 10:16:51,261 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.40 vs. limit=15.0
2023-11-24 10:17:15,261 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2790206.6666666665, ans=10.0
2023-11-24 10:17:19,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2790273.3333333335, ans=0.0
2023-11-24 10:17:20,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2790273.3333333335, ans=0.125
2023-11-24 10:17:21,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2790273.3333333335, ans=0.0
2023-11-24 10:17:23,310 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.44 vs. limit=15.0
2023-11-24 10:17:27,067 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.05 vs. limit=10.0
2023-11-24 10:17:27,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418550
2023-11-24 10:17:28,327 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.55 vs. limit=15.0
2023-11-24 10:17:29,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2790273.3333333335, ans=0.125
2023-11-24 10:17:36,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2790340.0, ans=0.025
2023-11-24 10:17:40,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2790340.0, ans=0.0
2023-11-24 10:17:43,649 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9750, loss[loss=0.08548, simple_loss=0.1176, pruned_loss=0.01781, audio_tagging_loss=0.008864, over 17134.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09127, pruned_loss=0.01328, audio_tagging_loss=0.008835, over 3048927.54 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:18:06,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2790473.3333333335, ans=0.125
2023-11-24 10:18:08,323 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.283e+01 8.345e+01 8.930e+01 9.592e+01 1.228e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 10:18:10,396 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.42 vs. limit=15.0
2023-11-24 10:18:11,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2790540.0, ans=0.125
2023-11-24 10:18:18,693 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.50 vs. limit=15.0
2023-11-24 10:18:26,750 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.89 vs. limit=15.0
2023-11-24 10:18:29,691 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418600
2023-11-24 10:18:36,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2790673.3333333335, ans=0.125
2023-11-24 10:18:36,198 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2790673.3333333335, ans=0.0
2023-11-24 10:18:45,259 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9800, loss[loss=0.07075, simple_loss=0.09169, pruned_loss=0.01557, audio_tagging_loss=0.009325, over 15605.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09159, pruned_loss=0.01339, audio_tagging_loss=0.008807, over 3047589.87 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:18:48,811 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.53 vs. limit=15.0
2023-11-24 10:18:52,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2790740.0, ans=0.1
2023-11-24 10:19:02,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2790806.6666666665, ans=0.125
2023-11-24 10:19:20,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2790873.3333333335, ans=0.125
2023-11-24 10:19:31,366 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418650
2023-11-24 10:19:40,634 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:19:48,363 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9850, loss[loss=0.05466, simple_loss=0.07286, pruned_loss=0.008336, audio_tagging_loss=0.009892, over 15969.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09149, pruned_loss=0.0134, audio_tagging_loss=0.008776, over 3052922.93 frames. ], batch size: 61, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:20:03,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2791140.0, ans=0.1
2023-11-24 10:20:12,659 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.248e+01 8.621e+01 9.085e+01 9.764e+01 1.357e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 10:20:18,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2791206.6666666665, ans=0.1
2023-11-24 10:20:34,234 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418700
2023-11-24 10:20:50,730 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9900, loss[loss=0.06021, simple_loss=0.08671, pruned_loss=0.009775, audio_tagging_loss=0.007085, over 15658.00 frames. ], tot_loss[loss=0.06833, simple_loss=0.09218, pruned_loss=0.01346, audio_tagging_loss=0.00878, over 3054938.02 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:21:03,213 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.32 vs. limit=22.5
2023-11-24 10:21:09,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2791473.3333333335, ans=0.125
2023-11-24 10:21:10,283 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=8.43 vs. limit=12.0
2023-11-24 10:21:13,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2791540.0, ans=0.0
2023-11-24 10:21:15,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2791540.0, ans=0.0
2023-11-24 10:21:36,769 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418750
2023-11-24 10:21:41,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2791673.3333333335, ans=0.0
2023-11-24 10:21:45,536 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.27 vs. limit=6.0
2023-11-24 10:21:46,794 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.95 vs. limit=15.0
2023-11-24 10:21:52,135 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 9950, loss[loss=0.05932, simple_loss=0.07547, pruned_loss=0.01166, audio_tagging_loss=0.009925, over 15448.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09213, pruned_loss=0.0133, audio_tagging_loss=0.008784, over 3062169.76 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:22:18,402 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.114e+01 8.444e+01 9.082e+01 9.757e+01 1.187e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 10:22:23,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2791873.3333333335, ans=0.0
2023-11-24 10:22:28,510 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2791873.3333333335, ans=0.1
2023-11-24 10:22:39,134 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418800
2023-11-24 10:22:55,824 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10000, loss[loss=0.05026, simple_loss=0.06842, pruned_loss=0.008222, audio_tagging_loss=0.007824, over 14848.00 frames. ], tot_loss[loss=0.06801, simple_loss=0.09188, pruned_loss=0.01333, audio_tagging_loss=0.008737, over 3053890.21 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:23:02,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2792073.3333333335, ans=0.0
2023-11-24 10:23:28,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2792206.6666666665, ans=0.2
2023-11-24 10:23:41,124 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2792273.3333333335, ans=0.125
2023-11-24 10:23:42,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418850
2023-11-24 10:23:48,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2792340.0, ans=0.0
2023-11-24 10:23:59,887 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10050, loss[loss=0.06408, simple_loss=0.08256, pruned_loss=0.01239, audio_tagging_loss=0.0104, over 14868.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.0916, pruned_loss=0.01333, audio_tagging_loss=0.008792, over 3052765.00 frames. ], batch size: 56, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:24:06,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.41 vs. limit=15.0
2023-11-24 10:24:08,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2792406.6666666665, ans=0.0
2023-11-24 10:24:18,084 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:24:24,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2792540.0, ans=0.125
2023-11-24 10:24:25,781 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.144e+01 8.576e+01 9.106e+01 9.929e+01 1.520e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 10:24:35,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2792540.0, ans=0.0
2023-11-24 10:24:37,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2792606.6666666665, ans=0.125
2023-11-24 10:24:40,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2792606.6666666665, ans=0.125
2023-11-24 10:24:42,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2792606.6666666665, ans=0.1
2023-11-24 10:24:47,104 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418900
2023-11-24 10:25:02,631 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10100, loss[loss=0.09468, simple_loss=0.1332, pruned_loss=0.02017, audio_tagging_loss=0.007914, over 15149.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.09244, pruned_loss=0.01345, audio_tagging_loss=0.008839, over 3053015.15 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:25:16,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2792806.6666666665, ans=0.2
2023-11-24 10:25:17,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2792806.6666666665, ans=0.04949747468305833
2023-11-24 10:25:20,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2792806.6666666665, ans=0.0
2023-11-24 10:25:23,665 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.99 vs. limit=10.0
2023-11-24 10:25:36,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2792873.3333333335, ans=0.1
2023-11-24 10:25:48,997 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 418950
2023-11-24 10:25:51,289 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:25:58,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2793006.6666666665, ans=0.0
2023-11-24 10:26:01,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2793006.6666666665, ans=0.125
2023-11-24 10:26:05,135 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10150, loss[loss=0.06614, simple_loss=0.08321, pruned_loss=0.01206, audio_tagging_loss=0.01247, over 15956.00 frames. ], tot_loss[loss=0.06854, simple_loss=0.09253, pruned_loss=0.0134, audio_tagging_loss=0.008874, over 3046126.42 frames. ], batch size: 59, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:26:14,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2793073.3333333335, ans=0.1
2023-11-24 10:26:26,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2793140.0, ans=0.0
2023-11-24 10:26:31,172 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.136e+01 8.585e+01 9.069e+01 9.759e+01 1.192e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 10:26:33,579 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:26:47,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2793273.3333333335, ans=0.125
2023-11-24 10:26:50,939 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419000
2023-11-24 10:27:08,904 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10200, loss[loss=0.0727, simple_loss=0.1031, pruned_loss=0.01141, audio_tagging_loss=0.009721, over 16699.00 frames. ], tot_loss[loss=0.06888, simple_loss=0.09317, pruned_loss=0.01339, audio_tagging_loss=0.008904, over 3047062.00 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:27:21,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.57 vs. limit=22.5
2023-11-24 10:27:26,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2793473.3333333335, ans=0.125
2023-11-24 10:27:26,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2793473.3333333335, ans=10.0
2023-11-24 10:27:30,995 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:27:52,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0
2023-11-24 10:27:55,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419050
2023-11-24 10:27:59,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2793673.3333333335, ans=0.125
2023-11-24 10:28:04,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2793673.3333333335, ans=0.125
2023-11-24 10:28:10,834 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10250, loss[loss=0.07748, simple_loss=0.106, pruned_loss=0.01766, audio_tagging_loss=0.006811, over 15926.00 frames. ], tot_loss[loss=0.06893, simple_loss=0.09315, pruned_loss=0.01346, audio_tagging_loss=0.00889, over 3048481.20 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:28:15,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2793740.0, ans=0.125
2023-11-24 10:28:25,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2793806.6666666665, ans=0.0
2023-11-24 10:28:37,728 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.049e+01 8.437e+01 9.083e+01 9.683e+01 1.798e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 10:28:50,812 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.18 vs. limit=22.5
2023-11-24 10:28:54,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2793940.0, ans=0.125
2023-11-24 10:28:57,503 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419100
2023-11-24 10:29:08,378 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2794006.6666666665, ans=0.125
2023-11-24 10:29:12,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2794073.3333333335, ans=0.2
2023-11-24 10:29:13,539 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10300, loss[loss=0.04386, simple_loss=0.05107, pruned_loss=0.005312, audio_tagging_loss=0.01301, over 14127.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09255, pruned_loss=0.01331, audio_tagging_loss=0.009048, over 3043278.11 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:29:22,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2794073.3333333335, ans=15.0
2023-11-24 10:29:34,024 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.88 vs. limit=22.5
2023-11-24 10:29:37,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2794140.0, ans=0.125
2023-11-24 10:30:00,353 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419150
2023-11-24 10:30:02,239 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.32 vs. limit=10.0
2023-11-24 10:30:16,905 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10350, loss[loss=0.06244, simple_loss=0.07995, pruned_loss=0.01307, audio_tagging_loss=0.009396, over 15812.00 frames. ], tot_loss[loss=0.06846, simple_loss=0.09203, pruned_loss=0.01326, audio_tagging_loss=0.009183, over 3042530.87 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:30:34,344 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:30:37,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2794473.3333333335, ans=0.125
2023-11-24 10:30:42,293 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.663e+01 8.615e+01 9.196e+01 1.008e+02 1.353e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 10:31:03,027 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419200
2023-11-24 10:31:13,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2794673.3333333335, ans=0.125
2023-11-24 10:31:19,400 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10400, loss[loss=0.09138, simple_loss=0.12, pruned_loss=0.02358, audio_tagging_loss=0.007817, over 16240.00 frames. ], tot_loss[loss=0.06864, simple_loss=0.09202, pruned_loss=0.0134, audio_tagging_loss=0.009227, over 3046596.53 frames. ], batch size: 58, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:31:20,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2794740.0, ans=0.1
2023-11-24 10:31:38,332 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=10.00 vs. limit=15.0
2023-11-24 10:31:50,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2794873.3333333335, ans=0.0
2023-11-24 10:31:55,007 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.28 vs. limit=15.0
2023-11-24 10:32:02,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2794940.0, ans=0.2
2023-11-24 10:32:05,598 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419250
2023-11-24 10:32:07,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2794940.0, ans=6.0
2023-11-24 10:32:21,633 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10450, loss[loss=0.07247, simple_loss=0.1021, pruned_loss=0.01425, audio_tagging_loss=0.007153, over 16590.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09152, pruned_loss=0.01334, audio_tagging_loss=0.009247, over 3043095.79 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:32:48,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2795206.6666666665, ans=0.0
2023-11-24 10:32:49,181 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.247e+01 8.462e+01 9.061e+01 9.658e+01 1.208e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 10:33:07,168 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419300
2023-11-24 10:33:16,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2795340.0, ans=0.0
2023-11-24 10:33:23,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2795406.6666666665, ans=0.1
2023-11-24 10:33:24,657 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10500, loss[loss=0.06583, simple_loss=0.09421, pruned_loss=0.009241, audio_tagging_loss=0.009479, over 16203.00 frames. ], tot_loss[loss=0.06882, simple_loss=0.09252, pruned_loss=0.01351, audio_tagging_loss=0.009054, over 3048181.90 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:33:35,984 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.84 vs. limit=6.0
2023-11-24 10:33:59,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2795540.0, ans=0.0
2023-11-24 10:34:04,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2795606.6666666665, ans=0.125
2023-11-24 10:34:08,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2795606.6666666665, ans=0.125
2023-11-24 10:34:11,678 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419350
2023-11-24 10:34:16,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2795673.3333333335, ans=0.2
2023-11-24 10:34:27,464 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10550, loss[loss=0.06963, simple_loss=0.09399, pruned_loss=0.01634, audio_tagging_loss=0.006299, over 14421.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09138, pruned_loss=0.01317, audio_tagging_loss=0.009023, over 3044855.61 frames. ], batch size: 54, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:34:33,626 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:34:51,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.05 vs. limit=15.0
2023-11-24 10:34:54,571 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.441e+01 8.781e+01 9.299e+01 1.006e+02 1.266e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-24 10:34:57,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2795873.3333333335, ans=0.09899494936611666
2023-11-24 10:34:58,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.72 vs. limit=15.0
2023-11-24 10:34:59,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2795873.3333333335, ans=0.125
2023-11-24 10:35:13,057 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419400
2023-11-24 10:35:13,264 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2795940.0, ans=0.125
2023-11-24 10:35:21,620 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.60 vs. limit=15.0
2023-11-24 10:35:25,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=15.0
2023-11-24 10:35:27,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2796006.6666666665, ans=0.2
2023-11-24 10:35:29,252 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10600, loss[loss=0.06218, simple_loss=0.09088, pruned_loss=0.01012, audio_tagging_loss=0.006616, over 16011.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.0912, pruned_loss=0.01311, audio_tagging_loss=0.008859, over 3041671.36 frames. ], batch size: 60, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:35:29,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2796073.3333333335, ans=0.0
2023-11-24 10:35:54,286 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=21.06 vs. limit=22.5
2023-11-24 10:36:00,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2796206.6666666665, ans=0.125
2023-11-24 10:36:02,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2796206.6666666665, ans=0.0
2023-11-24 10:36:03,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2796206.6666666665, ans=0.07
2023-11-24 10:36:11,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.57 vs. limit=15.0
2023-11-24 10:36:15,417 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419450
2023-11-24 10:36:18,021 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:36:18,415 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.71 vs. limit=6.0
2023-11-24 10:36:21,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.43 vs. limit=6.0
2023-11-24 10:36:30,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2796406.6666666665, ans=0.125
2023-11-24 10:36:32,459 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10650, loss[loss=0.04442, simple_loss=0.05345, pruned_loss=0.004896, audio_tagging_loss=0.0128, over 15058.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09086, pruned_loss=0.01316, audio_tagging_loss=0.008816, over 3049810.39 frames. ], batch size: 62, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:36:59,536 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.866e+01 8.535e+01 9.205e+01 9.977e+01 1.279e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 10:37:18,841 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419500
2023-11-24 10:37:24,559 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.05 vs. limit=15.0
2023-11-24 10:37:34,639 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10700, loss[loss=0.04632, simple_loss=0.06726, pruned_loss=0.004574, audio_tagging_loss=0.008114, over 14965.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09094, pruned_loss=0.01316, audio_tagging_loss=0.008784, over 3049104.41 frames. ], batch size: 57, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:37:36,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2796740.0, ans=0.1
2023-11-24 10:37:42,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2796740.0, ans=0.0
2023-11-24 10:37:59,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2796873.3333333335, ans=0.125
2023-11-24 10:38:02,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2796873.3333333335, ans=0.1
2023-11-24 10:38:11,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2796940.0, ans=0.2
2023-11-24 10:38:18,074 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.94 vs. limit=6.0
2023-11-24 10:38:21,149 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419550
2023-11-24 10:38:21,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2796940.0, ans=0.0
2023-11-24 10:38:26,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2797006.6666666665, ans=0.125
2023-11-24 10:38:37,181 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10750, loss[loss=0.09335, simple_loss=0.1272, pruned_loss=0.02369, audio_tagging_loss=0.006075, over 15734.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09085, pruned_loss=0.01308, audio_tagging_loss=0.008805, over 3050825.18 frames. ], batch size: 55, lr: 1.93e-03, grad_scale: 16.0
2023-11-24 10:38:39,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2797073.3333333335, ans=0.0
2023-11-24 10:38:41,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2797073.3333333335, ans=0.1
2023-11-24 10:38:48,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2797140.0, ans=0.125
2023-11-24 10:38:57,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2797140.0, ans=0.125
2023-11-24 10:39:05,289 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.988e+01 8.423e+01 9.138e+01 9.778e+01 1.311e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 10:39:05,924 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=4.96 vs. limit=15.0
2023-11-24 10:39:24,019 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419600
2023-11-24 10:39:34,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2797340.0, ans=0.0
2023-11-24 10:39:40,623 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10800, loss[loss=0.05301, simple_loss=0.05978, pruned_loss=0.0132, audio_tagging_loss=0.009925, over 14014.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09125, pruned_loss=0.01318, audio_tagging_loss=0.008785, over 3052386.87 frames. ], batch size: 53, lr: 1.93e-03, grad_scale: 32.0
2023-11-24 10:40:06,436 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:40:10,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2797540.0, ans=0.2
2023-11-24 10:40:26,294 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419650
2023-11-24 10:40:29,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2797673.3333333335, ans=0.125
2023-11-24 10:40:32,817 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=3.73 vs. limit=6.0
2023-11-24 10:40:38,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer_ff2.min_abs, batch_count=2797673.3333333335, ans=0.1
2023-11-24 10:40:42,946 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10850, loss[loss=0.05645, simple_loss=0.08317, pruned_loss=0.006616, audio_tagging_loss=0.00825, over 15121.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09115, pruned_loss=0.01311, audio_tagging_loss=0.008777, over 3052931.10 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:41:00,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2797806.6666666665, ans=0.1
2023-11-24 10:41:09,947 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.161e+01 8.591e+01 9.237e+01 1.014e+02 1.325e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 10:41:20,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2797940.0, ans=0.0
2023-11-24 10:41:29,160 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419700
2023-11-24 10:41:39,726 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:41:44,409 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10900, loss[loss=0.06595, simple_loss=0.09071, pruned_loss=0.01208, audio_tagging_loss=0.008509, over 15058.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.0908, pruned_loss=0.01322, audio_tagging_loss=0.008821, over 3048370.76 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:41:56,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2798140.0, ans=0.1
2023-11-24 10:42:06,091 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.64 vs. limit=15.0
2023-11-24 10:42:25,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2798273.3333333335, ans=0.1
2023-11-24 10:42:30,844 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419750
2023-11-24 10:42:37,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.94 vs. limit=15.0
2023-11-24 10:42:40,417 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 10:42:47,348 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 10950, loss[loss=0.05333, simple_loss=0.07166, pruned_loss=0.008817, audio_tagging_loss=0.008686, over 14991.00 frames. ], tot_loss[loss=0.06767, simple_loss=0.09117, pruned_loss=0.01327, audio_tagging_loss=0.008813, over 3048417.56 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:43:08,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2798473.3333333335, ans=0.0
2023-11-24 10:43:14,656 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.263e+01 9.239e+01 9.893e+01 2.160e+02, threshold=1.848e+02, percent-clipped=1.0
2023-11-24 10:43:19,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.22 vs. limit=6.0
2023-11-24 10:43:29,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.72 vs. limit=15.0
2023-11-24 10:43:33,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419800
2023-11-24 10:43:34,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2798606.6666666665, ans=0.125
2023-11-24 10:43:46,681 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.38 vs. limit=6.0
2023-11-24 10:43:50,673 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11000, loss[loss=0.05764, simple_loss=0.08129, pruned_loss=0.009671, audio_tagging_loss=0.007326, over 15010.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09116, pruned_loss=0.0132, audio_tagging_loss=0.008813, over 3041737.75 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:43:58,917 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:44:09,212 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.45 vs. limit=15.0
2023-11-24 10:44:21,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2798873.3333333335, ans=0.07
2023-11-24 10:44:33,149 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.15 vs. limit=15.0
2023-11-24 10:44:37,157 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419850
2023-11-24 10:44:38,837 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.64 vs. limit=15.0
2023-11-24 10:44:43,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2799006.6666666665, ans=0.2
2023-11-24 10:44:51,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2799073.3333333335, ans=0.125
2023-11-24 10:44:52,419 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11050, loss[loss=0.0832, simple_loss=0.1157, pruned_loss=0.01913, audio_tagging_loss=0.006194, over 15419.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.0916, pruned_loss=0.01335, audio_tagging_loss=0.008822, over 3048434.43 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:45:21,957 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.396e+01 8.466e+01 9.230e+01 9.960e+01 1.939e+02, threshold=1.846e+02, percent-clipped=1.0
2023-11-24 10:45:26,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.29 vs. limit=12.0
2023-11-24 10:45:36,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2799273.3333333335, ans=0.0
2023-11-24 10:45:38,882 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419900
2023-11-24 10:45:50,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2799340.0, ans=0.125
2023-11-24 10:45:55,579 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11100, loss[loss=0.06099, simple_loss=0.07215, pruned_loss=0.01232, audio_tagging_loss=0.01259, over 15280.00 frames. ], tot_loss[loss=0.06865, simple_loss=0.09238, pruned_loss=0.01351, audio_tagging_loss=0.008957, over 3050958.60 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:46:38,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2799606.6666666665, ans=0.125
2023-11-24 10:46:41,794 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 419950
2023-11-24 10:46:58,127 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11150, loss[loss=0.06571, simple_loss=0.09275, pruned_loss=0.01007, audio_tagging_loss=0.009268, over 15241.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09186, pruned_loss=0.01324, audio_tagging_loss=0.009103, over 3055290.43 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:47:03,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2799740.0, ans=0.1
2023-11-24 10:47:10,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2799806.6666666665, ans=0.125
2023-11-24 10:47:11,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2799806.6666666665, ans=0.125
2023-11-24 10:47:26,048 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.463e+01 9.073e+01 9.645e+01 1.170e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 10:47:26,436 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2799873.3333333335, ans=0.125
2023-11-24 10:47:30,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2799873.3333333335, ans=0.125
2023-11-24 10:47:38,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2799940.0, ans=0.0
2023-11-24 10:47:44,400 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420000
2023-11-24 10:47:45,876 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-420000.pt
2023-11-24 10:48:03,620 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11200, loss[loss=0.06913, simple_loss=0.08515, pruned_loss=0.01388, audio_tagging_loss=0.01267, over 15542.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.09153, pruned_loss=0.01325, audio_tagging_loss=0.00912, over 3048469.41 frames. ], batch size: 60, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:48:19,898 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2800140.0, ans=0.125
2023-11-24 10:48:22,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2800140.0, ans=0.125
2023-11-24 10:48:24,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2800140.0, ans=0.2
2023-11-24 10:48:33,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2800206.6666666665, ans=0.125
2023-11-24 10:48:50,452 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420050
2023-11-24 10:49:06,597 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11250, loss[loss=0.07607, simple_loss=0.1081, pruned_loss=0.01291, audio_tagging_loss=0.00911, over 15138.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09029, pruned_loss=0.01315, audio_tagging_loss=0.009178, over 3039737.36 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:49:06,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2800406.6666666665, ans=0.125
2023-11-24 10:49:13,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2800406.6666666665, ans=0.0
2023-11-24 10:49:35,712 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.322e+01 8.479e+01 9.057e+01 9.752e+01 1.909e+02, threshold=1.811e+02, percent-clipped=1.0
2023-11-24 10:49:36,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.51 vs. limit=15.0
2023-11-24 10:49:43,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2800606.6666666665, ans=0.125
2023-11-24 10:49:47,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2800606.6666666665, ans=0.1
2023-11-24 10:49:52,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420100
2023-11-24 10:49:54,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2800606.6666666665, ans=0.125
2023-11-24 10:50:06,353 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2800673.3333333335, ans=0.2
2023-11-24 10:50:09,696 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11300, loss[loss=0.06302, simple_loss=0.08528, pruned_loss=0.01146, audio_tagging_loss=0.008924, over 15131.00 frames. ], tot_loss[loss=0.06785, simple_loss=0.09106, pruned_loss=0.01321, audio_tagging_loss=0.009103, over 3034993.69 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:50:38,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2800873.3333333335, ans=0.125
2023-11-24 10:50:55,476 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420150
2023-11-24 10:51:10,849 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11350, loss[loss=0.06875, simple_loss=0.09782, pruned_loss=0.01266, audio_tagging_loss=0.007179, over 15026.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09126, pruned_loss=0.01308, audio_tagging_loss=0.008893, over 3043348.67 frames. ], batch size: 54, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:51:18,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2801073.3333333335, ans=0.2
2023-11-24 10:51:40,024 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.605e+01 8.545e+01 9.244e+01 9.787e+01 1.195e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 10:51:49,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2801273.3333333335, ans=0.125
2023-11-24 10:51:56,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420200
2023-11-24 10:52:13,254 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11400, loss[loss=0.07199, simple_loss=0.09212, pruned_loss=0.01645, audio_tagging_loss=0.009477, over 14030.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.09071, pruned_loss=0.013, audio_tagging_loss=0.008852, over 3040585.40 frames. ], batch size: 54, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:52:39,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2801540.0, ans=0.1
2023-11-24 10:52:43,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2801540.0, ans=0.125
2023-11-24 10:52:45,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2801540.0, ans=0.125
2023-11-24 10:52:52,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2801606.6666666665, ans=0.125
2023-11-24 10:52:54,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2801606.6666666665, ans=0.2
2023-11-24 10:52:58,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2801606.6666666665, ans=0.0
2023-11-24 10:53:00,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420250
2023-11-24 10:53:00,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2801606.6666666665, ans=0.2
2023-11-24 10:53:17,795 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11450, loss[loss=0.05744, simple_loss=0.07848, pruned_loss=0.009546, audio_tagging_loss=0.008653, over 15444.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09044, pruned_loss=0.01306, audio_tagging_loss=0.008885, over 3046852.88 frames. ], batch size: 58, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:53:37,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2801806.6666666665, ans=0.0
2023-11-24 10:53:46,378 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.482e+01 8.489e+01 9.216e+01 1.000e+02 1.338e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 10:53:50,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer_na.min_abs, batch_count=2801873.3333333335, ans=0.02
2023-11-24 10:54:03,645 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420300
2023-11-24 10:54:19,460 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11500, loss[loss=0.07774, simple_loss=0.1044, pruned_loss=0.01648, audio_tagging_loss=0.009084, over 14837.00 frames. ], tot_loss[loss=0.06622, simple_loss=0.08905, pruned_loss=0.01279, audio_tagging_loss=0.008906, over 3048029.05 frames. ], batch size: 55, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:54:31,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2802140.0, ans=0.125
2023-11-24 10:54:53,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2802206.6666666665, ans=0.09899494936611666
2023-11-24 10:54:54,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2802206.6666666665, ans=0.0
2023-11-24 10:54:57,509 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.58 vs. limit=15.0
2023-11-24 10:55:05,373 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420350
2023-11-24 10:55:20,741 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11550, loss[loss=0.08616, simple_loss=0.1218, pruned_loss=0.02099, audio_tagging_loss=0.004256, over 15442.00 frames. ], tot_loss[loss=0.06632, simple_loss=0.08916, pruned_loss=0.01281, audio_tagging_loss=0.008928, over 3050314.58 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 16.0
2023-11-24 10:55:21,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2802406.6666666665, ans=0.1
2023-11-24 10:55:32,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2802473.3333333335, ans=0.1
2023-11-24 10:55:45,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2802540.0, ans=0.05
2023-11-24 10:55:50,992 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.205e+01 8.625e+01 9.353e+01 9.944e+01 1.426e+02, threshold=1.871e+02, percent-clipped=0.0
2023-11-24 10:55:56,968 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 10:56:03,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2802606.6666666665, ans=0.1
2023-11-24 10:56:05,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2802606.6666666665, ans=0.0
2023-11-24 10:56:06,461 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420400
2023-11-24 10:56:22,932 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11600, loss[loss=0.06493, simple_loss=0.09004, pruned_loss=0.01163, audio_tagging_loss=0.008283, over 15138.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.08958, pruned_loss=0.01282, audio_tagging_loss=0.008839, over 3050389.17 frames. ], batch size: 56, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:56:23,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2802740.0, ans=0.2
2023-11-24 10:56:23,106 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2802740.0, ans=0.035
2023-11-24 10:56:58,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2802940.0, ans=0.1
2023-11-24 10:57:03,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2802940.0, ans=0.2
2023-11-24 10:57:03,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2802940.0, ans=0.125
2023-11-24 10:57:08,653 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420450
2023-11-24 10:57:24,767 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11650, loss[loss=0.05517, simple_loss=0.07621, pruned_loss=0.008227, audio_tagging_loss=0.00884, over 16414.00 frames. ], tot_loss[loss=0.06627, simple_loss=0.08898, pruned_loss=0.01287, audio_tagging_loss=0.008904, over 3050938.75 frames. ], batch size: 62, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:57:26,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2803073.3333333335, ans=10.0
2023-11-24 10:57:29,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2803073.3333333335, ans=0.0
2023-11-24 10:57:32,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2803073.3333333335, ans=0.125
2023-11-24 10:57:35,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2803140.0, ans=0.125
2023-11-24 10:57:38,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2803140.0, ans=0.0
2023-11-24 10:57:54,138 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.212e+01 8.520e+01 9.268e+01 1.018e+02 1.416e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 10:58:10,506 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420500
2023-11-24 10:58:18,187 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.71 vs. limit=15.0
2023-11-24 10:58:18,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.06 vs. limit=12.0
2023-11-24 10:58:25,806 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11700, loss[loss=0.07548, simple_loss=0.1072, pruned_loss=0.01379, audio_tagging_loss=0.008081, over 15054.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.08993, pruned_loss=0.01304, audio_tagging_loss=0.009018, over 3048621.16 frames. ], batch size: 54, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:58:34,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2803406.6666666665, ans=0.1
2023-11-24 10:58:35,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2803406.6666666665, ans=0.125
2023-11-24 10:58:41,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2803473.3333333335, ans=0.125
2023-11-24 10:58:48,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2803473.3333333335, ans=0.1
2023-11-24 10:59:11,891 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420550
2023-11-24 10:59:12,530 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.62 vs. limit=15.0
2023-11-24 10:59:14,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2803673.3333333335, ans=0.125
2023-11-24 10:59:19,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2803673.3333333335, ans=0.0
2023-11-24 10:59:28,516 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11750, loss[loss=0.0817, simple_loss=0.1095, pruned_loss=0.01881, audio_tagging_loss=0.008122, over 16835.00 frames. ], tot_loss[loss=0.06663, simple_loss=0.08944, pruned_loss=0.01289, audio_tagging_loss=0.009018, over 3056679.41 frames. ], batch size: 63, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 10:59:33,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2803740.0, ans=0.125
2023-11-24 10:59:33,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2803740.0, ans=0.0
2023-11-24 10:59:35,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2803740.0, ans=0.125
2023-11-24 10:59:40,924 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.71 vs. limit=15.0
2023-11-24 10:59:42,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2803806.6666666665, ans=0.2
2023-11-24 10:59:46,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2803806.6666666665, ans=0.025
2023-11-24 10:59:48,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2803806.6666666665, ans=0.2
2023-11-24 10:59:57,408 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.255e+01 8.962e+01 9.569e+01 1.238e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-24 11:00:01,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2803873.3333333335, ans=0.0
2023-11-24 11:00:13,387 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420600
2023-11-24 11:00:22,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff3.min_abs, batch_count=2804006.6666666665, ans=0.2
2023-11-24 11:00:29,789 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11800, loss[loss=0.04171, simple_loss=0.0516, pruned_loss=0.00543, audio_tagging_loss=0.01048, over 13914.00 frames. ], tot_loss[loss=0.06679, simple_loss=0.08986, pruned_loss=0.01287, audio_tagging_loss=0.00899, over 3053066.55 frames. ], batch size: 54, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:00:38,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=2804073.3333333335, ans=15.0
2023-11-24 11:01:15,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2804273.3333333335, ans=0.0
2023-11-24 11:01:16,289 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420650
2023-11-24 11:01:20,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2804340.0, ans=0.125
2023-11-24 11:01:32,366 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11850, loss[loss=0.05225, simple_loss=0.06349, pruned_loss=0.008639, audio_tagging_loss=0.01186, over 15994.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09092, pruned_loss=0.01308, audio_tagging_loss=0.009059, over 3045728.66 frames. ], batch size: 61, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:01:41,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2804406.6666666665, ans=0.1
2023-11-24 11:01:52,367 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.52 vs. limit=15.0
2023-11-24 11:02:02,504 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.352e+01 9.070e+01 9.952e+01 1.461e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 11:02:14,269 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.98 vs. limit=22.5
2023-11-24 11:02:18,522 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420700
2023-11-24 11:02:34,071 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.61 vs. limit=10.0
2023-11-24 11:02:34,402 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11900, loss[loss=0.08955, simple_loss=0.1194, pruned_loss=0.02126, audio_tagging_loss=0.008578, over 13921.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09128, pruned_loss=0.01317, audio_tagging_loss=0.009125, over 3044386.02 frames. ], batch size: 50, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:02:37,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2804740.0, ans=0.125
2023-11-24 11:02:49,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2804806.6666666665, ans=0.125
2023-11-24 11:03:02,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2804873.3333333335, ans=0.035
2023-11-24 11:03:14,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2804940.0, ans=0.1
2023-11-24 11:03:20,549 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420750
2023-11-24 11:03:31,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2805006.6666666665, ans=0.1
2023-11-24 11:03:37,212 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 11950, loss[loss=0.06857, simple_loss=0.09152, pruned_loss=0.01231, audio_tagging_loss=0.0105, over 14683.00 frames. ], tot_loss[loss=0.06859, simple_loss=0.09231, pruned_loss=0.01333, audio_tagging_loss=0.009104, over 3044628.02 frames. ], batch size: 57, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:03:55,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2805140.0, ans=0.125
2023-11-24 11:03:55,079 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2805140.0, ans=0.1
2023-11-24 11:04:07,052 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.174e+01 8.367e+01 9.062e+01 9.589e+01 1.237e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 11:04:22,826 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420800
2023-11-24 11:04:37,832 INFO [train_asr.py:1221] (0/4) Epoch 35, batch 12000, loss[loss=0.06151, simple_loss=0.07619, pruned_loss=0.01164, audio_tagging_loss=0.01178, over 13671.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09184, pruned_loss=0.01319, audio_tagging_loss=0.009249, over 3038590.51 frames. ], batch size: 53, lr: 1.92e-03, grad_scale: 32.0
2023-11-24 11:04:37,835 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 11:05:19,860 INFO [train_asr.py:1253] (0/4) Epoch 35, validation: loss=0.0585, simple_loss=0.05078, pruned_loss=0.005085, audio_tagging_loss=0.02803, over 4681554.00 frames. 
2023-11-24 11:05:19,861 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 11:05:32,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2805473.3333333335, ans=0.1
2023-11-24 11:05:45,937 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-35.pt
2023-11-24 11:06:25,324 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 0, loss[loss=0.06661, simple_loss=0.07072, pruned_loss=0.007578, audio_tagging_loss=0.02368, over 15023.00 frames. ], tot_loss[loss=0.06661, simple_loss=0.07072, pruned_loss=0.007578, audio_tagging_loss=0.02368, over 15023.00 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:06:25,327 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 11:06:44,463 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.0430, 3.7569, 3.3147, 3.6793], device='cuda:0')
2023-11-24 11:07:04,361 INFO [train_asr.py:1253] (0/4) Epoch 36, validation: loss=0.05761, simple_loss=0.0508, pruned_loss=0.005078, audio_tagging_loss=0.02713, over 4681554.00 frames. 
2023-11-24 11:07:04,362 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 11:07:08,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2805553.3333333335, ans=0.125
2023-11-24 11:07:14,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2805553.3333333335, ans=0.125
2023-11-24 11:07:21,177 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:07:21,549 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.43 vs. limit=22.5
2023-11-24 11:07:22,263 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420850
2023-11-24 11:07:24,313 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.13 vs. limit=15.0
2023-11-24 11:07:38,398 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.61 vs. limit=15.0
2023-11-24 11:07:58,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2805820.0, ans=0.05
2023-11-24 11:08:00,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2805820.0, ans=0.0
2023-11-24 11:08:06,409 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 50, loss[loss=0.08157, simple_loss=0.1069, pruned_loss=0.01285, audio_tagging_loss=0.01526, over 15572.00 frames. ], tot_loss[loss=0.07714, simple_loss=0.09318, pruned_loss=0.01368, audio_tagging_loss=0.01687, over 683220.03 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:08:06,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2805886.6666666665, ans=0.0
2023-11-24 11:08:07,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2805886.6666666665, ans=0.1
2023-11-24 11:08:09,899 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.471e+01 9.077e+01 9.826e+01 1.067e+02 1.319e+02, threshold=1.965e+02, percent-clipped=0.0
2023-11-24 11:08:16,011 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2805886.6666666665, ans=0.025
2023-11-24 11:08:21,742 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.71 vs. limit=22.5
2023-11-24 11:08:22,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2805953.3333333335, ans=0.0
2023-11-24 11:08:25,427 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420900
2023-11-24 11:08:39,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2806020.0, ans=0.0
2023-11-24 11:08:42,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2806020.0, ans=0.125
2023-11-24 11:08:42,764 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.77 vs. limit=8.0
2023-11-24 11:08:43,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2806086.6666666665, ans=0.125
2023-11-24 11:09:08,030 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 100, loss[loss=0.05994, simple_loss=0.06472, pruned_loss=0.0094, audio_tagging_loss=0.01818, over 15384.00 frames. ], tot_loss[loss=0.07504, simple_loss=0.09082, pruned_loss=0.01316, audio_tagging_loss=0.01647, over 1207027.36 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:09:08,637 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.43 vs. limit=15.0
2023-11-24 11:09:09,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2806220.0, ans=0.125
2023-11-24 11:09:12,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2806220.0, ans=0.125
2023-11-24 11:09:12,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2806220.0, ans=0.0
2023-11-24 11:09:27,617 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 420950
2023-11-24 11:09:47,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2806420.0, ans=0.125
2023-11-24 11:10:07,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2806486.6666666665, ans=0.125
2023-11-24 11:10:11,660 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 150, loss[loss=0.06893, simple_loss=0.09146, pruned_loss=0.01219, audio_tagging_loss=0.01101, over 14548.00 frames. ], tot_loss[loss=0.07326, simple_loss=0.09048, pruned_loss=0.01314, audio_tagging_loss=0.01488, over 1615162.11 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:10:15,198 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 8.014e+01 9.024e+01 9.585e+01 1.046e+02 2.115e+02, threshold=1.917e+02, percent-clipped=1.0
2023-11-24 11:10:29,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421000
2023-11-24 11:10:52,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2806753.3333333335, ans=0.125
2023-11-24 11:11:01,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2806820.0, ans=0.125
2023-11-24 11:11:13,395 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 200, loss[loss=0.07328, simple_loss=0.09289, pruned_loss=0.01612, audio_tagging_loss=0.01072, over 15969.00 frames. ], tot_loss[loss=0.07184, simple_loss=0.09159, pruned_loss=0.01305, audio_tagging_loss=0.013, over 1935213.05 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:11:20,092 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.28 vs. limit=15.0
2023-11-24 11:11:30,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2806953.3333333335, ans=0.0
2023-11-24 11:11:31,261 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421050
2023-11-24 11:11:58,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2807086.6666666665, ans=0.0
2023-11-24 11:12:02,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.31 vs. limit=15.0
2023-11-24 11:12:14,923 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 250, loss[loss=0.05993, simple_loss=0.08269, pruned_loss=0.01052, audio_tagging_loss=0.008068, over 15879.00 frames. ], tot_loss[loss=0.07097, simple_loss=0.09194, pruned_loss=0.01328, audio_tagging_loss=0.01172, over 2186339.82 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:12:17,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2807220.0, ans=0.125
2023-11-24 11:12:18,500 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.960e+01 8.652e+01 9.389e+01 9.985e+01 1.276e+02, threshold=1.878e+02, percent-clipped=0.0
2023-11-24 11:12:34,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421100
2023-11-24 11:13:00,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2807420.0, ans=10.0
2023-11-24 11:13:02,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2807420.0, ans=0.125
2023-11-24 11:13:16,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2807553.3333333335, ans=0.0
2023-11-24 11:13:18,276 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 300, loss[loss=0.06232, simple_loss=0.07972, pruned_loss=0.01435, audio_tagging_loss=0.00811, over 14884.00 frames. ], tot_loss[loss=0.07101, simple_loss=0.09321, pruned_loss=0.0136, audio_tagging_loss=0.0108, over 2374464.30 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:13:29,313 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.02 vs. limit=12.0
2023-11-24 11:13:36,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421150
2023-11-24 11:13:38,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2807620.0, ans=0.125
2023-11-24 11:13:41,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2807686.6666666665, ans=0.0
2023-11-24 11:14:06,415 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.36 vs. limit=15.0
2023-11-24 11:14:19,895 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 350, loss[loss=0.06301, simple_loss=0.08769, pruned_loss=0.01268, audio_tagging_loss=0.006487, over 14460.00 frames. ], tot_loss[loss=0.06994, simple_loss=0.09262, pruned_loss=0.01335, audio_tagging_loss=0.01028, over 2523338.31 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:14:23,375 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.480e+01 9.093e+01 9.744e+01 1.336e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 11:14:28,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2807886.6666666665, ans=0.04949747468305833
2023-11-24 11:14:38,011 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421200
2023-11-24 11:14:46,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2808020.0, ans=0.125
2023-11-24 11:14:53,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2808020.0, ans=0.0
2023-11-24 11:15:04,424 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.73 vs. limit=15.0
2023-11-24 11:15:21,403 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 400, loss[loss=0.05663, simple_loss=0.06993, pruned_loss=0.01189, audio_tagging_loss=0.009777, over 15699.00 frames. ], tot_loss[loss=0.06945, simple_loss=0.09248, pruned_loss=0.01321, audio_tagging_loss=0.009998, over 2636891.65 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:15:21,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2808220.0, ans=0.125
2023-11-24 11:15:22,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2808220.0, ans=0.125
2023-11-24 11:15:27,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2808220.0, ans=0.125
2023-11-24 11:15:41,651 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421250
2023-11-24 11:15:47,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2808353.3333333335, ans=0.2
2023-11-24 11:16:03,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2808420.0, ans=0.1
2023-11-24 11:16:23,734 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 450, loss[loss=0.05055, simple_loss=0.06481, pruned_loss=0.007684, audio_tagging_loss=0.01046, over 15189.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09181, pruned_loss=0.01318, audio_tagging_loss=0.009712, over 2730320.03 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:16:27,924 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.965e+01 8.299e+01 9.052e+01 9.986e+01 1.410e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 11:16:39,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2808620.0, ans=0.0
2023-11-24 11:16:39,413 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2808620.0, ans=0.1
2023-11-24 11:16:42,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421300
2023-11-24 11:17:02,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2808753.3333333335, ans=0.125
2023-11-24 11:17:21,398 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.54 vs. limit=15.0
2023-11-24 11:17:26,701 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 500, loss[loss=0.05756, simple_loss=0.08475, pruned_loss=0.01016, audio_tagging_loss=0.005028, over 14643.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09045, pruned_loss=0.01286, audio_tagging_loss=0.00957, over 2799335.73 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:17:39,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten.whitening_limit, batch_count=2808953.3333333335, ans=15.0
2023-11-24 11:17:43,993 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.58 vs. limit=15.0
2023-11-24 11:17:44,583 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421350
2023-11-24 11:17:48,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2808953.3333333335, ans=0.0
2023-11-24 11:18:10,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2809086.6666666665, ans=0.0
2023-11-24 11:18:21,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2809153.3333333335, ans=0.125
2023-11-24 11:18:24,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2809153.3333333335, ans=0.125
2023-11-24 11:18:28,542 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 550, loss[loss=0.05664, simple_loss=0.07105, pruned_loss=0.008005, audio_tagging_loss=0.0131, over 15153.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.08985, pruned_loss=0.01267, audio_tagging_loss=0.0095, over 2854432.02 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:18:32,037 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.126e+01 8.386e+01 8.873e+01 9.796e+01 1.246e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-24 11:18:32,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2809220.0, ans=0.125
2023-11-24 11:18:47,309 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421400
2023-11-24 11:18:53,148 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:18:59,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2809353.3333333335, ans=0.125
2023-11-24 11:19:17,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2809486.6666666665, ans=0.0
2023-11-24 11:19:27,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2809486.6666666665, ans=0.2
2023-11-24 11:19:31,271 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 600, loss[loss=0.05159, simple_loss=0.06331, pruned_loss=0.01224, audio_tagging_loss=0.007693, over 14698.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.08926, pruned_loss=0.01265, audio_tagging_loss=0.009398, over 2888263.76 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:19:44,928 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.01 vs. limit=15.0
2023-11-24 11:19:50,539 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421450
2023-11-24 11:20:00,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2809686.6666666665, ans=0.125
2023-11-24 11:20:03,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2809686.6666666665, ans=0.125
2023-11-24 11:20:12,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2809753.3333333335, ans=0.2
2023-11-24 11:20:33,874 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 650, loss[loss=0.06176, simple_loss=0.08488, pruned_loss=0.01071, audio_tagging_loss=0.008605, over 14469.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.0898, pruned_loss=0.01286, audio_tagging_loss=0.009347, over 2928808.33 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:20:34,265 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:20:37,358 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.046e+01 8.541e+01 9.260e+01 1.007e+02 1.291e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 11:20:41,939 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.96 vs. limit=22.5
2023-11-24 11:20:49,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2809953.3333333335, ans=0.1
2023-11-24 11:20:51,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421500
2023-11-24 11:20:51,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2809953.3333333335, ans=0.1
2023-11-24 11:21:01,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2810020.0, ans=0.0
2023-11-24 11:21:03,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2810020.0, ans=0.0
2023-11-24 11:21:10,039 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.36 vs. limit=22.5
2023-11-24 11:21:18,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2810086.6666666665, ans=0.125
2023-11-24 11:21:30,147 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.19 vs. limit=15.0
2023-11-24 11:21:34,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2810220.0, ans=0.1
2023-11-24 11:21:35,326 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 700, loss[loss=0.08959, simple_loss=0.1195, pruned_loss=0.02102, audio_tagging_loss=0.008814, over 15141.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.08958, pruned_loss=0.01279, audio_tagging_loss=0.00925, over 2957544.02 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:21:54,221 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421550
2023-11-24 11:21:59,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2810353.3333333335, ans=0.05
2023-11-24 11:22:11,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2810420.0, ans=0.125
2023-11-24 11:22:25,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2810486.6666666665, ans=0.125
2023-11-24 11:22:37,459 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 750, loss[loss=0.07059, simple_loss=0.1002, pruned_loss=0.01416, audio_tagging_loss=0.006355, over 14849.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09151, pruned_loss=0.01306, audio_tagging_loss=0.009134, over 2982831.33 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:22:38,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2810553.3333333335, ans=0.1
2023-11-24 11:22:41,365 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.250e+01 8.522e+01 9.060e+01 9.732e+01 1.237e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 11:22:43,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2810553.3333333335, ans=6.0
2023-11-24 11:22:52,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2810620.0, ans=0.1
2023-11-24 11:22:56,244 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421600
2023-11-24 11:23:18,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2810753.3333333335, ans=0.0
2023-11-24 11:23:40,411 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 800, loss[loss=0.07698, simple_loss=0.1012, pruned_loss=0.01742, audio_tagging_loss=0.008941, over 16109.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09151, pruned_loss=0.01292, audio_tagging_loss=0.009158, over 2989985.54 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:23:42,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2810886.6666666665, ans=0.125
2023-11-24 11:23:56,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2810953.3333333335, ans=0.125
2023-11-24 11:23:58,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421650
2023-11-24 11:24:17,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2811086.6666666665, ans=0.2
2023-11-24 11:24:18,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2811086.6666666665, ans=0.2
2023-11-24 11:24:32,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2811153.3333333335, ans=6.0
2023-11-24 11:24:40,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2811153.3333333335, ans=0.015
2023-11-24 11:24:42,435 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 850, loss[loss=0.07861, simple_loss=0.09788, pruned_loss=0.01832, audio_tagging_loss=0.01135, over 15223.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09169, pruned_loss=0.01285, audio_tagging_loss=0.009142, over 3015479.73 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:24:47,195 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.597e+01 8.758e+01 9.246e+01 1.015e+02 2.108e+02, threshold=1.849e+02, percent-clipped=1.0
2023-11-24 11:24:53,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2811286.6666666665, ans=0.0
2023-11-24 11:25:01,330 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421700
2023-11-24 11:25:20,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2811420.0, ans=0.125
2023-11-24 11:25:23,442 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.37 vs. limit=15.0
2023-11-24 11:25:35,597 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.74 vs. limit=15.0
2023-11-24 11:25:45,247 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 900, loss[loss=0.06825, simple_loss=0.09308, pruned_loss=0.01196, audio_tagging_loss=0.009747, over 14800.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09117, pruned_loss=0.01286, audio_tagging_loss=0.00919, over 3022692.99 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:25:45,760 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.90 vs. limit=6.0
2023-11-24 11:25:54,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2811553.3333333335, ans=0.125
2023-11-24 11:26:04,191 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421750
2023-11-24 11:26:09,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2811686.6666666665, ans=0.125
2023-11-24 11:26:14,296 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.33 vs. limit=15.0
2023-11-24 11:26:19,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2811686.6666666665, ans=0.1
2023-11-24 11:26:23,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2811753.3333333335, ans=0.0
2023-11-24 11:26:47,796 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 950, loss[loss=0.06839, simple_loss=0.09766, pruned_loss=0.01125, audio_tagging_loss=0.008314, over 16418.00 frames. ], tot_loss[loss=0.06828, simple_loss=0.09209, pruned_loss=0.01318, audio_tagging_loss=0.009054, over 3032876.39 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:26:52,397 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.341e+01 8.504e+01 9.122e+01 9.832e+01 1.245e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 11:26:55,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2811886.6666666665, ans=0.0
2023-11-24 11:26:58,692 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2811953.3333333335, ans=0.2
2023-11-24 11:27:06,041 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421800
2023-11-24 11:27:15,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.95 vs. limit=15.0
2023-11-24 11:27:17,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2812020.0, ans=0.125
2023-11-24 11:27:17,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2812020.0, ans=0.125
2023-11-24 11:27:18,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2812020.0, ans=0.125
2023-11-24 11:27:24,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2812086.6666666665, ans=0.0
2023-11-24 11:27:40,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2812153.3333333335, ans=0.2
2023-11-24 11:27:46,643 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.22 vs. limit=22.5
2023-11-24 11:27:49,570 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1000, loss[loss=0.06578, simple_loss=0.08361, pruned_loss=0.01511, audio_tagging_loss=0.008865, over 15927.00 frames. ], tot_loss[loss=0.06808, simple_loss=0.09178, pruned_loss=0.01321, audio_tagging_loss=0.008978, over 3032111.42 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:27:55,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2812220.0, ans=0.0
2023-11-24 11:27:59,105 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.06 vs. limit=12.0
2023-11-24 11:28:04,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2812286.6666666665, ans=0.0
2023-11-24 11:28:08,655 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421850
2023-11-24 11:28:15,028 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:28:20,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2812353.3333333335, ans=0.125
2023-11-24 11:28:51,786 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1050, loss[loss=0.05283, simple_loss=0.07257, pruned_loss=0.009437, audio_tagging_loss=0.007107, over 14806.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09122, pruned_loss=0.01322, audio_tagging_loss=0.008874, over 3033498.86 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:28:57,210 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.996e+01 8.446e+01 9.134e+01 9.863e+01 1.540e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 11:29:03,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.52 vs. limit=22.5
2023-11-24 11:29:07,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2812620.0, ans=0.125
2023-11-24 11:29:08,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2812620.0, ans=0.0
2023-11-24 11:29:11,235 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.45 vs. limit=15.0
2023-11-24 11:29:11,719 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421900
2023-11-24 11:29:13,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.33 vs. limit=15.0
2023-11-24 11:29:28,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2812753.3333333335, ans=0.1
2023-11-24 11:29:40,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2812753.3333333335, ans=0.0
2023-11-24 11:29:43,421 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.72 vs. limit=22.5
2023-11-24 11:29:45,529 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.70 vs. limit=12.0
2023-11-24 11:29:55,479 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1100, loss[loss=0.07505, simple_loss=0.09641, pruned_loss=0.01594, audio_tagging_loss=0.0109, over 14673.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.09006, pruned_loss=0.01292, audio_tagging_loss=0.008834, over 3027349.37 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:29:57,915 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:29:59,242 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2812886.6666666665, ans=0.125
2023-11-24 11:30:11,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2812953.3333333335, ans=0.125
2023-11-24 11:30:13,222 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 421950
2023-11-24 11:30:16,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.19 vs. limit=15.0
2023-11-24 11:30:18,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2813020.0, ans=0.125
2023-11-24 11:30:56,389 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1150, loss[loss=0.07928, simple_loss=0.1147, pruned_loss=0.01465, audio_tagging_loss=0.00726, over 15119.00 frames. ], tot_loss[loss=0.06664, simple_loss=0.08987, pruned_loss=0.01285, audio_tagging_loss=0.008853, over 3035476.79 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:30:59,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2813220.0, ans=0.125
2023-11-24 11:30:59,315 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=14.95 vs. limit=22.5
2023-11-24 11:31:01,047 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.827e+01 8.442e+01 9.003e+01 9.685e+01 1.147e+02, threshold=1.801e+02, percent-clipped=0.0
2023-11-24 11:31:07,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2813286.6666666665, ans=0.125
2023-11-24 11:31:15,478 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422000
2023-11-24 11:31:17,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2813286.6666666665, ans=0.0
2023-11-24 11:31:22,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2813353.3333333335, ans=0.2
2023-11-24 11:31:34,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2813420.0, ans=0.1
2023-11-24 11:31:41,926 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.18 vs. limit=22.5
2023-11-24 11:31:45,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2813486.6666666665, ans=0.125
2023-11-24 11:31:58,433 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1200, loss[loss=0.05641, simple_loss=0.0796, pruned_loss=0.009657, audio_tagging_loss=0.006956, over 14943.00 frames. ], tot_loss[loss=0.06633, simple_loss=0.08953, pruned_loss=0.01276, audio_tagging_loss=0.008803, over 3025142.02 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:32:00,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2813553.3333333335, ans=0.125
2023-11-24 11:32:11,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2813620.0, ans=0.0
2023-11-24 11:32:17,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422050
2023-11-24 11:32:27,748 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.87 vs. limit=15.0
2023-11-24 11:32:38,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2813753.3333333335, ans=0.0
2023-11-24 11:33:00,660 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1250, loss[loss=0.04484, simple_loss=0.05664, pruned_loss=0.008151, audio_tagging_loss=0.008372, over 15605.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.09016, pruned_loss=0.01293, audio_tagging_loss=0.008677, over 3036770.57 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:33:01,213 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.09 vs. limit=22.5
2023-11-24 11:33:07,043 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.529e+01 8.565e+01 9.272e+01 1.023e+02 1.182e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 11:33:12,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2813953.3333333335, ans=0.09899494936611666
2023-11-24 11:33:18,922 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422100
2023-11-24 11:33:28,997 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.14 vs. limit=6.0
2023-11-24 11:33:41,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2814086.6666666665, ans=0.0
2023-11-24 11:33:53,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2814153.3333333335, ans=0.0
2023-11-24 11:34:01,808 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1300, loss[loss=0.06167, simple_loss=0.09141, pruned_loss=0.01055, audio_tagging_loss=0.005416, over 14871.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09063, pruned_loss=0.01298, audio_tagging_loss=0.008637, over 3033058.71 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:34:19,681 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422150
2023-11-24 11:34:52,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2814486.6666666665, ans=0.125
2023-11-24 11:35:04,217 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1350, loss[loss=0.06827, simple_loss=0.0917, pruned_loss=0.01459, audio_tagging_loss=0.00783, over 15223.00 frames. ], tot_loss[loss=0.06652, simple_loss=0.08998, pruned_loss=0.01283, audio_tagging_loss=0.008696, over 3034818.38 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:35:10,807 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.712e+01 8.582e+01 9.240e+01 9.907e+01 1.176e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 11:35:24,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422200
2023-11-24 11:35:24,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2814620.0, ans=0.1
2023-11-24 11:35:26,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2814620.0, ans=0.125
2023-11-24 11:35:45,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2814753.3333333335, ans=0.2
2023-11-24 11:35:49,785 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:36:08,393 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1400, loss[loss=0.08846, simple_loss=0.1281, pruned_loss=0.017, audio_tagging_loss=0.007407, over 15207.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09093, pruned_loss=0.01293, audio_tagging_loss=0.008763, over 3044287.06 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:36:12,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2814886.6666666665, ans=0.125
2023-11-24 11:36:17,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2814886.6666666665, ans=0.125
2023-11-24 11:36:21,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2814953.3333333335, ans=0.2
2023-11-24 11:36:25,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2814953.3333333335, ans=0.0
2023-11-24 11:36:25,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2814953.3333333335, ans=0.0
2023-11-24 11:36:26,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422250
2023-11-24 11:37:03,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2815153.3333333335, ans=0.0
2023-11-24 11:37:10,654 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1450, loss[loss=0.0786, simple_loss=0.1002, pruned_loss=0.02139, audio_tagging_loss=0.007108, over 15294.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09186, pruned_loss=0.01329, audio_tagging_loss=0.008806, over 3045406.83 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:37:15,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2815220.0, ans=0.125
2023-11-24 11:37:16,388 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.014e+01 8.536e+01 9.281e+01 1.016e+02 1.384e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-24 11:37:16,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2815220.0, ans=0.0
2023-11-24 11:37:28,220 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422300
2023-11-24 11:37:31,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2815286.6666666665, ans=10.0
2023-11-24 11:37:41,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2815353.3333333335, ans=10.0
2023-11-24 11:37:53,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2815420.0, ans=0.2
2023-11-24 11:38:11,603 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1500, loss[loss=0.0769, simple_loss=0.1037, pruned_loss=0.01446, audio_tagging_loss=0.01061, over 16041.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09234, pruned_loss=0.01343, audio_tagging_loss=0.008912, over 3048070.89 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:38:15,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2815553.3333333335, ans=0.125
2023-11-24 11:38:19,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.69 vs. limit=15.0
2023-11-24 11:38:20,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2815553.3333333335, ans=0.2
2023-11-24 11:38:31,139 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422350
2023-11-24 11:38:37,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2815686.6666666665, ans=0.125
2023-11-24 11:38:52,395 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.43 vs. limit=6.0
2023-11-24 11:39:02,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2815820.0, ans=0.2
2023-11-24 11:39:11,779 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.82 vs. limit=15.0
2023-11-24 11:39:13,900 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1550, loss[loss=0.07898, simple_loss=0.119, pruned_loss=0.01491, audio_tagging_loss=0.00459, over 16332.00 frames. ], tot_loss[loss=0.06884, simple_loss=0.09293, pruned_loss=0.01348, audio_tagging_loss=0.008894, over 3048038.48 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:39:20,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2815886.6666666665, ans=0.125
2023-11-24 11:39:20,781 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.077e+01 8.494e+01 9.231e+01 9.838e+01 1.696e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 11:39:32,715 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422400
2023-11-24 11:39:38,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2816020.0, ans=0.125
2023-11-24 11:39:52,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2816086.6666666665, ans=0.09899494936611666
2023-11-24 11:39:59,596 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.99 vs. limit=15.0
2023-11-24 11:40:00,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2816086.6666666665, ans=0.035
2023-11-24 11:40:13,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2816153.3333333335, ans=0.0
2023-11-24 11:40:14,362 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.08 vs. limit=15.0
2023-11-24 11:40:17,315 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1600, loss[loss=0.07448, simple_loss=0.09883, pruned_loss=0.01387, audio_tagging_loss=0.01119, over 15684.00 frames. ], tot_loss[loss=0.06851, simple_loss=0.0922, pruned_loss=0.0134, audio_tagging_loss=0.009011, over 3044974.77 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:40:24,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2816220.0, ans=0.125
2023-11-24 11:40:25,053 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.94 vs. limit=15.0
2023-11-24 11:40:35,178 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422450
2023-11-24 11:40:40,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.79 vs. limit=22.5
2023-11-24 11:40:58,332 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.53 vs. limit=15.0
2023-11-24 11:41:13,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2816486.6666666665, ans=0.125
2023-11-24 11:41:16,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2816486.6666666665, ans=0.125
2023-11-24 11:41:18,987 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1650, loss[loss=0.05738, simple_loss=0.07599, pruned_loss=0.009344, audio_tagging_loss=0.01004, over 14267.00 frames. ], tot_loss[loss=0.06802, simple_loss=0.09132, pruned_loss=0.01325, audio_tagging_loss=0.009114, over 3049126.05 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:41:25,966 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.033e+01 8.620e+01 9.407e+01 1.041e+02 1.351e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-24 11:41:37,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422500
2023-11-24 11:41:45,653 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-24 11:42:06,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2816753.3333333335, ans=0.1
2023-11-24 11:42:15,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2816820.0, ans=0.0
2023-11-24 11:42:18,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2816820.0, ans=0.2
2023-11-24 11:42:20,967 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1700, loss[loss=0.06685, simple_loss=0.0926, pruned_loss=0.01291, audio_tagging_loss=0.007647, over 15930.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09081, pruned_loss=0.01322, audio_tagging_loss=0.009212, over 3052326.46 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:42:31,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2816886.6666666665, ans=0.0
2023-11-24 11:42:38,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2816953.3333333335, ans=0.2
2023-11-24 11:42:40,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422550
2023-11-24 11:42:40,705 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2816953.3333333335, ans=0.0
2023-11-24 11:42:50,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.28 vs. limit=15.0
2023-11-24 11:42:52,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2817020.0, ans=0.1
2023-11-24 11:42:58,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2817086.6666666665, ans=0.125
2023-11-24 11:43:07,606 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2817086.6666666665, ans=0.125
2023-11-24 11:43:15,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2817153.3333333335, ans=0.125
2023-11-24 11:43:22,236 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2817153.3333333335, ans=0.125
2023-11-24 11:43:24,392 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1750, loss[loss=0.08101, simple_loss=0.1197, pruned_loss=0.01403, audio_tagging_loss=0.007103, over 15561.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09094, pruned_loss=0.0132, audio_tagging_loss=0.009129, over 3050910.35 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:43:24,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2817220.0, ans=0.2
2023-11-24 11:43:26,305 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.60 vs. limit=15.0
2023-11-24 11:43:27,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2817220.0, ans=0.125
2023-11-24 11:43:31,547 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.461e+01 8.743e+01 9.239e+01 9.812e+01 1.237e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 11:43:42,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422600
2023-11-24 11:43:51,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2817353.3333333335, ans=0.95
2023-11-24 11:43:58,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff2.min_abs, batch_count=2817353.3333333335, ans=0.1
2023-11-24 11:44:26,827 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1800, loss[loss=0.04867, simple_loss=0.06427, pruned_loss=0.008181, audio_tagging_loss=0.008351, over 16164.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.0901, pruned_loss=0.01309, audio_tagging_loss=0.009, over 3042853.88 frames. ], batch size: 63, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:44:34,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2817553.3333333335, ans=0.0
2023-11-24 11:44:35,668 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.57 vs. limit=15.0
2023-11-24 11:44:45,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422650
2023-11-24 11:44:54,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2817686.6666666665, ans=0.035
2023-11-24 11:45:13,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2817753.3333333335, ans=0.125
2023-11-24 11:45:28,832 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1850, loss[loss=0.06566, simple_loss=0.09393, pruned_loss=0.01288, audio_tagging_loss=0.005817, over 14983.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09061, pruned_loss=0.01319, audio_tagging_loss=0.008934, over 3035324.77 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:45:36,368 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.377e+01 8.376e+01 8.960e+01 9.835e+01 1.450e+02, threshold=1.792e+02, percent-clipped=0.0
2023-11-24 11:45:48,725 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422700
2023-11-24 11:45:58,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2818020.0, ans=0.0
2023-11-24 11:46:04,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2818020.0, ans=0.125
2023-11-24 11:46:07,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2818086.6666666665, ans=10.0
2023-11-24 11:46:24,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2818153.3333333335, ans=0.04949747468305833
2023-11-24 11:46:29,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2818153.3333333335, ans=0.125
2023-11-24 11:46:31,729 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1900, loss[loss=0.0628, simple_loss=0.08395, pruned_loss=0.01202, audio_tagging_loss=0.008804, over 14908.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09065, pruned_loss=0.01311, audio_tagging_loss=0.008864, over 3036985.67 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 11:46:33,551 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=8.20 vs. limit=15.0
2023-11-24 11:46:50,153 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422750
2023-11-24 11:46:51,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2818286.6666666665, ans=0.2
2023-11-24 11:47:00,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2818353.3333333335, ans=0.0
2023-11-24 11:47:19,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-24 11:47:26,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2818486.6666666665, ans=0.125
2023-11-24 11:47:30,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2818486.6666666665, ans=0.125
2023-11-24 11:47:31,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2818486.6666666665, ans=0.0
2023-11-24 11:47:33,689 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 1950, loss[loss=0.08932, simple_loss=0.1279, pruned_loss=0.01937, audio_tagging_loss=0.006001, over 16431.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09129, pruned_loss=0.01331, audio_tagging_loss=0.008867, over 3039864.81 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 11:47:41,766 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.102e+01 8.719e+01 9.311e+01 9.932e+01 1.598e+02, threshold=1.862e+02, percent-clipped=0.0
2023-11-24 11:47:47,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.38 vs. limit=15.0
2023-11-24 11:47:51,894 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422800
2023-11-24 11:47:54,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2818620.0, ans=0.1
2023-11-24 11:48:02,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2818686.6666666665, ans=0.0
2023-11-24 11:48:12,091 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2818753.3333333335, ans=0.0
2023-11-24 11:48:13,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2818753.3333333335, ans=0.125
2023-11-24 11:48:16,115 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:48:17,310 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:48:17,639 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.10 vs. limit=22.5
2023-11-24 11:48:18,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2818753.3333333335, ans=0.2
2023-11-24 11:48:20,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2818753.3333333335, ans=0.125
2023-11-24 11:48:34,869 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2000, loss[loss=0.0562, simple_loss=0.07028, pruned_loss=0.01264, audio_tagging_loss=0.008426, over 16210.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09043, pruned_loss=0.01326, audio_tagging_loss=0.008909, over 3042394.70 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:48:54,368 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422850
2023-11-24 11:49:05,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2819020.0, ans=0.0
2023-11-24 11:49:13,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2819086.6666666665, ans=0.0
2023-11-24 11:49:22,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2819086.6666666665, ans=0.125
2023-11-24 11:49:38,125 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2050, loss[loss=0.06669, simple_loss=0.09198, pruned_loss=0.01296, audio_tagging_loss=0.007734, over 14702.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.08998, pruned_loss=0.013, audio_tagging_loss=0.008882, over 3040882.03 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:49:43,404 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=15.0
2023-11-24 11:49:45,997 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.00 vs. limit=15.0
2023-11-24 11:49:46,321 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.671e+01 8.908e+01 9.390e+01 1.017e+02 1.320e+02, threshold=1.878e+02, percent-clipped=0.0
2023-11-24 11:49:48,131 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.38 vs. limit=10.0
2023-11-24 11:49:49,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2819286.6666666665, ans=0.1
2023-11-24 11:49:56,525 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422900
2023-11-24 11:50:27,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2819486.6666666665, ans=0.2
2023-11-24 11:50:40,033 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2100, loss[loss=0.05673, simple_loss=0.07893, pruned_loss=0.009818, audio_tagging_loss=0.007441, over 15474.00 frames. ], tot_loss[loss=0.06648, simple_loss=0.08967, pruned_loss=0.01287, audio_tagging_loss=0.008777, over 3043030.97 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:50:40,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.09 vs. limit=10.0
2023-11-24 11:50:44,918 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.17 vs. limit=15.0
2023-11-24 11:50:52,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2819620.0, ans=0.125
2023-11-24 11:50:58,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 422950
2023-11-24 11:51:18,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2819753.3333333335, ans=0.125
2023-11-24 11:51:18,262 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 11:51:18,654 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2819753.3333333335, ans=22.5
2023-11-24 11:51:21,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2819753.3333333335, ans=0.125
2023-11-24 11:51:23,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2819753.3333333335, ans=0.125
2023-11-24 11:51:25,674 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.58 vs. limit=15.0
2023-11-24 11:51:34,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2819820.0, ans=0.125
2023-11-24 11:51:42,233 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2150, loss[loss=0.06524, simple_loss=0.09267, pruned_loss=0.01121, audio_tagging_loss=0.007692, over 16585.00 frames. ], tot_loss[loss=0.0666, simple_loss=0.08995, pruned_loss=0.0129, audio_tagging_loss=0.008726, over 3044016.43 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:51:47,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2819886.6666666665, ans=0.125
2023-11-24 11:51:51,145 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.463e+01 9.382e+01 1.013e+02 1.297e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 11:51:58,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2819953.3333333335, ans=0.125
2023-11-24 11:51:58,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2819953.3333333335, ans=0.2
2023-11-24 11:52:00,251 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.76 vs. limit=22.5
2023-11-24 11:52:01,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423000
2023-11-24 11:52:07,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2820020.0, ans=0.0
2023-11-24 11:52:18,813 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:52:37,288 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.30 vs. limit=15.0
2023-11-24 11:52:45,857 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2200, loss[loss=0.05613, simple_loss=0.07622, pruned_loss=0.0094, audio_tagging_loss=0.008619, over 15062.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09002, pruned_loss=0.01306, audio_tagging_loss=0.008862, over 3044534.25 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:52:47,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2820220.0, ans=0.125
2023-11-24 11:53:03,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423050
2023-11-24 11:53:12,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2820353.3333333335, ans=0.09899494936611666
2023-11-24 11:53:40,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2820486.6666666665, ans=0.95
2023-11-24 11:53:47,547 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2250, loss[loss=0.07447, simple_loss=0.09916, pruned_loss=0.01552, audio_tagging_loss=0.00937, over 13885.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.09176, pruned_loss=0.01333, audio_tagging_loss=0.008825, over 3050797.61 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:53:55,792 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.635e+01 8.644e+01 9.254e+01 1.001e+02 1.291e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 11:53:58,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=2820553.3333333335, ans=0.125
2023-11-24 11:54:00,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2820620.0, ans=0.5
2023-11-24 11:54:01,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2820620.0, ans=0.09899494936611666
2023-11-24 11:54:06,252 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423100
2023-11-24 11:54:49,834 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2300, loss[loss=0.06978, simple_loss=0.09134, pruned_loss=0.01493, audio_tagging_loss=0.009178, over 15924.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09119, pruned_loss=0.01326, audio_tagging_loss=0.008917, over 3045290.83 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:54:55,495 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.39 vs. limit=15.0
2023-11-24 11:55:09,880 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423150
2023-11-24 11:55:20,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2821020.0, ans=0.125
2023-11-24 11:55:20,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2821020.0, ans=0.2
2023-11-24 11:55:31,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2821086.6666666665, ans=0.125
2023-11-24 11:55:38,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2821086.6666666665, ans=0.2
2023-11-24 11:55:45,416 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 11:55:52,933 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2350, loss[loss=0.07974, simple_loss=0.1085, pruned_loss=0.01769, audio_tagging_loss=0.007818, over 15289.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09125, pruned_loss=0.01321, audio_tagging_loss=0.008944, over 3045489.26 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 11:55:55,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2821220.0, ans=0.125
2023-11-24 11:55:55,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2821220.0, ans=0.0
2023-11-24 11:56:01,694 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.049e+01 8.333e+01 8.924e+01 9.646e+01 1.536e+02, threshold=1.785e+02, percent-clipped=0.0
2023-11-24 11:56:04,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2821286.6666666665, ans=0.0
2023-11-24 11:56:08,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2821286.6666666665, ans=0.1
2023-11-24 11:56:11,491 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423200
2023-11-24 11:56:20,511 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.08 vs. limit=22.5
2023-11-24 11:56:26,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2821353.3333333335, ans=0.1
2023-11-24 11:56:32,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2821420.0, ans=0.125
2023-11-24 11:56:40,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2821420.0, ans=0.2
2023-11-24 11:56:55,517 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2400, loss[loss=0.04701, simple_loss=0.06197, pruned_loss=0.003768, audio_tagging_loss=0.01226, over 15678.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09145, pruned_loss=0.01314, audio_tagging_loss=0.009074, over 3042552.40 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:57:13,522 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423250
2023-11-24 11:57:27,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2821686.6666666665, ans=0.0
2023-11-24 11:57:27,461 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.75 vs. limit=22.5
2023-11-24 11:57:57,325 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2450, loss[loss=0.07064, simple_loss=0.08808, pruned_loss=0.01599, audio_tagging_loss=0.01061, over 16657.00 frames. ], tot_loss[loss=0.06796, simple_loss=0.09126, pruned_loss=0.01318, audio_tagging_loss=0.009151, over 3037937.72 frames. ], batch size: 63, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:58:06,727 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.416e+01 8.359e+01 8.901e+01 9.611e+01 1.267e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-24 11:58:16,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423300
2023-11-24 11:58:33,709 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.66 vs. limit=10.0
2023-11-24 11:59:00,581 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2500, loss[loss=0.0653, simple_loss=0.08391, pruned_loss=0.01369, audio_tagging_loss=0.009658, over 14889.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09119, pruned_loss=0.01309, audio_tagging_loss=0.009162, over 3039351.24 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 11:59:18,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423350
2023-11-24 11:59:33,964 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.26 vs. limit=22.5
2023-11-24 11:59:37,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2822420.0, ans=0.125
2023-11-24 11:59:57,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.min_positive, batch_count=2822486.6666666665, ans=0.05
2023-11-24 12:00:02,396 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2550, loss[loss=0.06917, simple_loss=0.09, pruned_loss=0.01313, audio_tagging_loss=0.01104, over 14639.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09084, pruned_loss=0.01292, audio_tagging_loss=0.009083, over 3041413.24 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:00:06,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2822553.3333333335, ans=0.0
2023-11-24 12:00:10,607 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.159e+01 8.497e+01 9.131e+01 9.713e+01 1.199e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 12:00:15,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2822620.0, ans=0.125
2023-11-24 12:00:20,163 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423400
2023-11-24 12:00:47,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2822753.3333333335, ans=0.125
2023-11-24 12:00:47,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2822753.3333333335, ans=0.0
2023-11-24 12:00:52,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2822820.0, ans=0.0
2023-11-24 12:01:04,230 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2600, loss[loss=0.09559, simple_loss=0.1361, pruned_loss=0.01829, audio_tagging_loss=0.00926, over 15369.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09031, pruned_loss=0.01287, audio_tagging_loss=0.009068, over 3046665.43 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:01:09,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2822886.6666666665, ans=0.125
2023-11-24 12:01:22,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2822953.3333333335, ans=0.0
2023-11-24 12:01:23,844 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423450
2023-11-24 12:02:05,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2823153.3333333335, ans=0.0
2023-11-24 12:02:07,938 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2650, loss[loss=0.06449, simple_loss=0.0932, pruned_loss=0.01027, audio_tagging_loss=0.007613, over 15362.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09049, pruned_loss=0.01292, audio_tagging_loss=0.008866, over 3043945.68 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:02:12,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2823220.0, ans=0.125
2023-11-24 12:02:18,456 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.615e+01 8.376e+01 8.976e+01 9.750e+01 1.206e+02, threshold=1.795e+02, percent-clipped=0.0
2023-11-24 12:02:20,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2823286.6666666665, ans=0.1
2023-11-24 12:02:27,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423500
2023-11-24 12:02:51,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2823420.0, ans=0.025
2023-11-24 12:02:59,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2823486.6666666665, ans=10.0
2023-11-24 12:03:00,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2823486.6666666665, ans=0.0
2023-11-24 12:03:05,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2823486.6666666665, ans=0.1
2023-11-24 12:03:09,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2823553.3333333335, ans=0.125
2023-11-24 12:03:10,477 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2700, loss[loss=0.07661, simple_loss=0.1065, pruned_loss=0.01584, audio_tagging_loss=0.007503, over 14978.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09158, pruned_loss=0.01302, audio_tagging_loss=0.008829, over 3050499.75 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:03:20,429 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.26 vs. limit=15.0
2023-11-24 12:03:21,407 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2823620.0, ans=0.2
2023-11-24 12:03:24,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2823620.0, ans=0.2
2023-11-24 12:03:28,368 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423550
2023-11-24 12:03:36,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2823686.6666666665, ans=0.0
2023-11-24 12:04:11,632 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2750, loss[loss=0.051, simple_loss=0.06778, pruned_loss=0.01002, audio_tagging_loss=0.007089, over 15148.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09105, pruned_loss=0.01301, audio_tagging_loss=0.008805, over 3046112.43 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:04:12,395 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.61 vs. limit=15.0
2023-11-24 12:04:20,993 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.189e+01 8.328e+01 8.931e+01 9.724e+01 1.117e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 12:04:30,131 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423600
2023-11-24 12:04:49,067 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.84 vs. limit=22.5
2023-11-24 12:04:56,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2824086.6666666665, ans=0.125
2023-11-24 12:04:59,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2824086.6666666665, ans=0.0
2023-11-24 12:05:03,808 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:05:13,999 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2800, loss[loss=0.08139, simple_loss=0.1186, pruned_loss=0.01417, audio_tagging_loss=0.007943, over 15901.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09116, pruned_loss=0.01308, audio_tagging_loss=0.008833, over 3043845.20 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:05:26,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2824286.6666666665, ans=0.125
2023-11-24 12:05:33,644 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423650
2023-11-24 12:05:37,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2824286.6666666665, ans=0.035
2023-11-24 12:06:15,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2824486.6666666665, ans=0.125
2023-11-24 12:06:17,164 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2850, loss[loss=0.07088, simple_loss=0.09303, pruned_loss=0.0139, audio_tagging_loss=0.01047, over 15944.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09037, pruned_loss=0.01302, audio_tagging_loss=0.00898, over 3038022.43 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:06:17,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2824553.3333333335, ans=0.125
2023-11-24 12:06:21,028 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:06:26,591 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.601e+01 9.263e+01 1.001e+02 1.378e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 12:06:32,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2824620.0, ans=0.0
2023-11-24 12:06:35,042 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423700
2023-11-24 12:06:42,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2824686.6666666665, ans=0.125
2023-11-24 12:06:43,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2824686.6666666665, ans=0.125
2023-11-24 12:06:52,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2824753.3333333335, ans=10.0
2023-11-24 12:07:09,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2824820.0, ans=0.125
2023-11-24 12:07:10,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2824820.0, ans=0.125
2023-11-24 12:07:12,241 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.57 vs. limit=22.5
2023-11-24 12:07:12,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.55 vs. limit=15.0
2023-11-24 12:07:18,755 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2900, loss[loss=0.06745, simple_loss=0.09329, pruned_loss=0.009323, audio_tagging_loss=0.01148, over 14773.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.0905, pruned_loss=0.01303, audio_tagging_loss=0.009043, over 3046440.86 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:07:33,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2824953.3333333335, ans=0.2
2023-11-24 12:07:37,119 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423750
2023-11-24 12:07:38,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2824953.3333333335, ans=0.125
2023-11-24 12:07:39,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=2824953.3333333335, ans=0.125
2023-11-24 12:08:02,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2825086.6666666665, ans=0.125
2023-11-24 12:08:02,453 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.92 vs. limit=15.0
2023-11-24 12:08:05,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2825086.6666666665, ans=0.1
2023-11-24 12:08:16,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2825153.3333333335, ans=0.125
2023-11-24 12:08:21,117 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 2950, loss[loss=0.06733, simple_loss=0.08828, pruned_loss=0.01511, audio_tagging_loss=0.008081, over 14954.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.092, pruned_loss=0.01329, audio_tagging_loss=0.008932, over 3050232.38 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:08:31,481 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.715e+01 8.518e+01 9.259e+01 1.017e+02 1.246e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 12:08:31,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2825220.0, ans=0.125
2023-11-24 12:08:41,109 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423800
2023-11-24 12:08:46,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2825353.3333333335, ans=0.2
2023-11-24 12:09:02,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2825420.0, ans=0.125
2023-11-24 12:09:15,806 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.15 vs. limit=15.0
2023-11-24 12:09:24,619 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3000, loss[loss=0.06799, simple_loss=0.08361, pruned_loss=0.01445, audio_tagging_loss=0.01174, over 13856.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09129, pruned_loss=0.01329, audio_tagging_loss=0.00899, over 3040282.17 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:09:24,622 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 12:10:03,083 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3533, 5.0117, 4.6668, 5.1700], device='cuda:0')
2023-11-24 12:10:06,936 INFO [train_asr.py:1253] (0/4) Epoch 36, validation: loss=0.05726, simple_loss=0.05083, pruned_loss=0.005098, audio_tagging_loss=0.02675, over 4681554.00 frames. 
2023-11-24 12:10:06,936 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 12:10:26,566 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423850
2023-11-24 12:10:35,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2825686.6666666665, ans=0.1
2023-11-24 12:10:37,224 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:10:39,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2825686.6666666665, ans=0.125
2023-11-24 12:10:41,851 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.16 vs. limit=15.0
2023-11-24 12:10:51,143 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2825753.3333333335, ans=0.0
2023-11-24 12:10:51,506 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.53 vs. limit=12.0
2023-11-24 12:11:10,536 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3050, loss[loss=0.07192, simple_loss=0.09839, pruned_loss=0.01493, audio_tagging_loss=0.007799, over 15724.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09111, pruned_loss=0.01322, audio_tagging_loss=0.009048, over 3043837.50 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:11:19,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2825886.6666666665, ans=0.1
2023-11-24 12:11:20,549 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.127e+01 8.750e+01 9.306e+01 1.016e+02 1.344e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-24 12:11:30,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423900
2023-11-24 12:11:32,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2825953.3333333335, ans=0.2
2023-11-24 12:11:46,844 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:12:01,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2826153.3333333335, ans=0.125
2023-11-24 12:12:02,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2826153.3333333335, ans=0.125
2023-11-24 12:12:09,190 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=14.44 vs. limit=15.0
2023-11-24 12:12:13,627 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3100, loss[loss=0.05531, simple_loss=0.0742, pruned_loss=0.0105, audio_tagging_loss=0.007701, over 14812.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09141, pruned_loss=0.0132, audio_tagging_loss=0.009004, over 3044970.89 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:12:16,338 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:12:26,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2826286.6666666665, ans=0.125
2023-11-24 12:12:32,258 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 423950
2023-11-24 12:13:03,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2826486.6666666665, ans=0.0
2023-11-24 12:13:16,553 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3150, loss[loss=0.07093, simple_loss=0.09339, pruned_loss=0.01591, audio_tagging_loss=0.008329, over 14525.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09126, pruned_loss=0.01315, audio_tagging_loss=0.009042, over 3042884.35 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:13:25,948 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.764e+01 8.600e+01 9.080e+01 9.861e+01 1.236e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 12:13:34,973 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424000
2023-11-24 12:13:37,007 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-424000.pt
2023-11-24 12:13:39,996 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:13:58,747 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:14:21,318 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3200, loss[loss=0.07845, simple_loss=0.1118, pruned_loss=0.01314, audio_tagging_loss=0.009408, over 16458.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09164, pruned_loss=0.01322, audio_tagging_loss=0.009135, over 3044007.74 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:14:41,511 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424050
2023-11-24 12:14:49,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2827020.0, ans=0.125
2023-11-24 12:14:50,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2827020.0, ans=0.125
2023-11-24 12:14:53,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2827020.0, ans=0.125
2023-11-24 12:15:06,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2827086.6666666665, ans=0.125
2023-11-24 12:15:06,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.56 vs. limit=15.0
2023-11-24 12:15:13,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2827153.3333333335, ans=0.125
2023-11-24 12:15:17,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2827153.3333333335, ans=0.0
2023-11-24 12:15:20,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2827153.3333333335, ans=0.09899494936611666
2023-11-24 12:15:25,383 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3250, loss[loss=0.0624, simple_loss=0.07982, pruned_loss=0.01317, audio_tagging_loss=0.009318, over 14806.00 frames. ], tot_loss[loss=0.06847, simple_loss=0.0919, pruned_loss=0.01327, audio_tagging_loss=0.009242, over 3047897.21 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:15:34,960 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.932e+01 8.544e+01 9.313e+01 9.948e+01 1.268e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 12:15:39,724 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.29 vs. limit=15.0
2023-11-24 12:15:42,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2827286.6666666665, ans=0.125
2023-11-24 12:15:43,939 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424100
2023-11-24 12:15:59,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=2827353.3333333335, ans=6.0
2023-11-24 12:16:21,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2827486.6666666665, ans=0.0
2023-11-24 12:16:23,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2827486.6666666665, ans=0.125
2023-11-24 12:16:27,554 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3300, loss[loss=0.07636, simple_loss=0.09678, pruned_loss=0.01698, audio_tagging_loss=0.011, over 16079.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09174, pruned_loss=0.01322, audio_tagging_loss=0.009344, over 3047667.24 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:16:31,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2827553.3333333335, ans=0.1
2023-11-24 12:16:42,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2827620.0, ans=0.125
2023-11-24 12:16:45,710 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424150
2023-11-24 12:17:13,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2827753.3333333335, ans=0.95
2023-11-24 12:17:24,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2827820.0, ans=0.125
2023-11-24 12:17:25,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2827820.0, ans=0.07
2023-11-24 12:17:29,998 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3350, loss[loss=0.08025, simple_loss=0.1161, pruned_loss=0.01196, audio_tagging_loss=0.01026, over 16025.00 frames. ], tot_loss[loss=0.06848, simple_loss=0.09192, pruned_loss=0.0133, audio_tagging_loss=0.009215, over 3051386.31 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:17:40,622 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.571e+01 9.014e+01 9.710e+01 1.291e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 12:17:49,607 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424200
2023-11-24 12:17:50,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.30 vs. limit=15.0
2023-11-24 12:18:00,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2828020.0, ans=0.1
2023-11-24 12:18:07,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2828086.6666666665, ans=0.0
2023-11-24 12:18:31,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2828153.3333333335, ans=0.2
2023-11-24 12:18:33,256 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3400, loss[loss=0.06819, simple_loss=0.09715, pruned_loss=0.01007, audio_tagging_loss=0.009545, over 15063.00 frames. ], tot_loss[loss=0.06841, simple_loss=0.09212, pruned_loss=0.01335, audio_tagging_loss=0.008999, over 3052845.36 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 32.0
2023-11-24 12:18:40,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2828220.0, ans=0.125
2023-11-24 12:18:40,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2828220.0, ans=0.09899494936611666
2023-11-24 12:18:51,914 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424250
2023-11-24 12:19:07,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2828353.3333333335, ans=0.125
2023-11-24 12:19:14,157 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:19:31,411 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2828486.6666666665, ans=0.125
2023-11-24 12:19:35,831 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3450, loss[loss=0.06914, simple_loss=0.09332, pruned_loss=0.01229, audio_tagging_loss=0.01019, over 14718.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09161, pruned_loss=0.01333, audio_tagging_loss=0.008986, over 3049218.45 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:19:47,009 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.469e+01 9.145e+01 9.844e+01 1.227e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 12:19:54,203 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424300
2023-11-24 12:20:06,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2828686.6666666665, ans=0.0
2023-11-24 12:20:36,530 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.61 vs. limit=15.0
2023-11-24 12:20:38,417 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3500, loss[loss=0.049, simple_loss=0.06264, pruned_loss=0.01252, audio_tagging_loss=0.005157, over 14343.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09216, pruned_loss=0.01348, audio_tagging_loss=0.008871, over 3051484.23 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:20:48,183 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.56 vs. limit=15.0
2023-11-24 12:20:54,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2828953.3333333335, ans=0.125
2023-11-24 12:20:58,013 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424350
2023-11-24 12:21:07,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2829020.0, ans=0.0
2023-11-24 12:21:11,081 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:21:16,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2829086.6666666665, ans=0.1
2023-11-24 12:21:23,920 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2829086.6666666665, ans=0.0
2023-11-24 12:21:25,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2829086.6666666665, ans=0.025
2023-11-24 12:21:27,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2829153.3333333335, ans=0.125
2023-11-24 12:21:40,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2829220.0, ans=0.125
2023-11-24 12:21:41,496 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3550, loss[loss=0.0728, simple_loss=0.1043, pruned_loss=0.01411, audio_tagging_loss=0.006549, over 14364.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09153, pruned_loss=0.01333, audio_tagging_loss=0.008948, over 3044005.14 frames. ], batch size: 53, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:21:43,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2829220.0, ans=0.125
2023-11-24 12:21:52,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2829220.0, ans=0.1
2023-11-24 12:21:52,757 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.784e+01 8.449e+01 8.981e+01 9.607e+01 1.173e+02, threshold=1.796e+02, percent-clipped=0.0
2023-11-24 12:22:00,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424400
2023-11-24 12:22:19,355 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.36 vs. limit=15.0
2023-11-24 12:22:28,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=2829420.0, ans=10.0
2023-11-24 12:22:44,316 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3600, loss[loss=0.06318, simple_loss=0.08351, pruned_loss=0.01156, audio_tagging_loss=0.009858, over 16342.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.08992, pruned_loss=0.01303, audio_tagging_loss=0.008912, over 3041935.55 frames. ], batch size: 61, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:22:50,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2829553.3333333335, ans=0.2
2023-11-24 12:22:56,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2829620.0, ans=0.0
2023-11-24 12:23:03,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424450
2023-11-24 12:23:46,405 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3650, loss[loss=0.05485, simple_loss=0.06854, pruned_loss=0.007532, audio_tagging_loss=0.01304, over 16518.00 frames. ], tot_loss[loss=0.0666, simple_loss=0.08951, pruned_loss=0.01288, audio_tagging_loss=0.008959, over 3041319.56 frames. ], batch size: 66, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:23:46,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.08 vs. limit=15.0
2023-11-24 12:23:51,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2829886.6666666665, ans=0.0
2023-11-24 12:23:59,993 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.262e+01 8.304e+01 8.721e+01 9.501e+01 1.428e+02, threshold=1.744e+02, percent-clipped=0.0
2023-11-24 12:24:06,043 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424500
2023-11-24 12:24:11,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2830020.0, ans=0.05
2023-11-24 12:24:17,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2830020.0, ans=0.0
2023-11-24 12:24:18,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2830020.0, ans=0.5
2023-11-24 12:24:19,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2830020.0, ans=0.0
2023-11-24 12:24:23,748 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.34 vs. limit=15.0
2023-11-24 12:24:25,168 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.99 vs. limit=6.0
2023-11-24 12:24:28,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2830086.6666666665, ans=0.1
2023-11-24 12:24:31,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2830086.6666666665, ans=0.0
2023-11-24 12:24:39,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=2830153.3333333335, ans=0.0
2023-11-24 12:24:40,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2830153.3333333335, ans=0.2
2023-11-24 12:24:43,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2830153.3333333335, ans=0.1
2023-11-24 12:24:49,725 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3700, loss[loss=0.04929, simple_loss=0.07041, pruned_loss=0.005718, audio_tagging_loss=0.008372, over 14636.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.0907, pruned_loss=0.01314, audio_tagging_loss=0.008834, over 3045729.27 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:24:50,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2830220.0, ans=0.125
2023-11-24 12:24:55,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2830220.0, ans=0.1
2023-11-24 12:25:01,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2830286.6666666665, ans=0.125
2023-11-24 12:25:01,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2830286.6666666665, ans=0.1
2023-11-24 12:25:08,374 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424550
2023-11-24 12:25:28,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2830420.0, ans=0.125
2023-11-24 12:25:31,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2830420.0, ans=0.0
2023-11-24 12:25:51,806 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3750, loss[loss=0.07122, simple_loss=0.09943, pruned_loss=0.01268, audio_tagging_loss=0.008833, over 16362.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09096, pruned_loss=0.0132, audio_tagging_loss=0.008885, over 3039443.64 frames. ], batch size: 62, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:26:03,765 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.375e+01 8.897e+01 9.293e+01 9.941e+01 1.332e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 12:26:09,863 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424600
2023-11-24 12:26:11,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2830620.0, ans=0.1
2023-11-24 12:26:19,784 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.39 vs. limit=15.0
2023-11-24 12:26:35,386 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:26:53,222 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3800, loss[loss=0.0591, simple_loss=0.07909, pruned_loss=0.008811, audio_tagging_loss=0.01074, over 15458.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09151, pruned_loss=0.01339, audio_tagging_loss=0.008901, over 3037227.40 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:27:04,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2830953.3333333335, ans=0.125
2023-11-24 12:27:12,751 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424650
2023-11-24 12:27:12,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2830953.3333333335, ans=0.125
2023-11-24 12:27:21,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2831020.0, ans=0.1
2023-11-24 12:27:37,998 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=9.56 vs. limit=12.0
2023-11-24 12:27:45,175 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.13 vs. limit=15.0
2023-11-24 12:27:52,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2831153.3333333335, ans=0.0
2023-11-24 12:27:56,867 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3850, loss[loss=0.06749, simple_loss=0.09031, pruned_loss=0.01091, audio_tagging_loss=0.01143, over 14837.00 frames. ], tot_loss[loss=0.06823, simple_loss=0.09193, pruned_loss=0.0134, audio_tagging_loss=0.008873, over 3044073.46 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:28:09,277 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.222e+01 8.465e+01 9.054e+01 9.529e+01 1.182e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-24 12:28:09,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2831286.6666666665, ans=0.1
2023-11-24 12:28:12,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2831286.6666666665, ans=0.125
2023-11-24 12:28:13,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2831286.6666666665, ans=0.125
2023-11-24 12:28:13,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2831286.6666666665, ans=0.125
2023-11-24 12:28:15,427 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424700
2023-11-24 12:28:15,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2831286.6666666665, ans=0.2
2023-11-24 12:28:15,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2831286.6666666665, ans=0.125
2023-11-24 12:28:15,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2831286.6666666665, ans=0.125
2023-11-24 12:28:30,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2831353.3333333335, ans=0.125
2023-11-24 12:28:48,934 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:28:56,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2831486.6666666665, ans=0.125
2023-11-24 12:28:58,701 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3900, loss[loss=0.05989, simple_loss=0.07937, pruned_loss=0.01029, audio_tagging_loss=0.009917, over 15796.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09185, pruned_loss=0.01336, audio_tagging_loss=0.009027, over 3041814.25 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:29:04,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2831553.3333333335, ans=0.0
2023-11-24 12:29:16,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424750
2023-11-24 12:29:46,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2831753.3333333335, ans=0.0
2023-11-24 12:29:49,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2831820.0, ans=0.125
2023-11-24 12:29:52,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2831820.0, ans=0.0
2023-11-24 12:30:00,523 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 3950, loss[loss=0.0728, simple_loss=0.08931, pruned_loss=0.01559, audio_tagging_loss=0.01255, over 14899.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09113, pruned_loss=0.0132, audio_tagging_loss=0.009147, over 3043255.53 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:30:03,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2831886.6666666665, ans=0.125
2023-11-24 12:30:14,457 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.446e+01 8.514e+01 9.052e+01 9.990e+01 1.654e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 12:30:16,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2831953.3333333335, ans=0.125
2023-11-24 12:30:18,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2831953.3333333335, ans=0.2
2023-11-24 12:30:20,004 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424800
2023-11-24 12:30:50,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=12.34 vs. limit=15.0
2023-11-24 12:31:02,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2832220.0, ans=0.1
2023-11-24 12:31:03,781 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4000, loss[loss=0.08116, simple_loss=0.111, pruned_loss=0.01682, audio_tagging_loss=0.008833, over 14807.00 frames. ], tot_loss[loss=0.06843, simple_loss=0.09193, pruned_loss=0.01329, audio_tagging_loss=0.009172, over 3045039.74 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:31:09,796 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.05 vs. limit=15.0
2023-11-24 12:31:10,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2832220.0, ans=0.0
2023-11-24 12:31:23,624 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424850
2023-11-24 12:31:50,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2832420.0, ans=0.125
2023-11-24 12:32:07,618 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4050, loss[loss=0.07484, simple_loss=0.09767, pruned_loss=0.01658, audio_tagging_loss=0.009425, over 15627.00 frames. ], tot_loss[loss=0.06891, simple_loss=0.09249, pruned_loss=0.01351, audio_tagging_loss=0.009157, over 3039891.57 frames. ], batch size: 60, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:32:11,198 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:32:19,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2832620.0, ans=0.0
2023-11-24 12:32:21,750 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.652e+01 8.871e+01 9.409e+01 1.005e+02 1.323e+02, threshold=1.882e+02, percent-clipped=0.0
2023-11-24 12:32:22,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2832620.0, ans=0.0
2023-11-24 12:32:25,522 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424900
2023-11-24 12:32:27,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2832620.0, ans=0.125
2023-11-24 12:32:34,486 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.37 vs. limit=22.5
2023-11-24 12:32:48,700 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.74 vs. limit=15.0
2023-11-24 12:32:55,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2832753.3333333335, ans=0.125
2023-11-24 12:32:57,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2832820.0, ans=0.0
2023-11-24 12:33:08,513 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2832886.6666666665, ans=0.0
2023-11-24 12:33:09,320 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4100, loss[loss=0.06808, simple_loss=0.09384, pruned_loss=0.01541, audio_tagging_loss=0.005754, over 15200.00 frames. ], tot_loss[loss=0.06856, simple_loss=0.09219, pruned_loss=0.01333, audio_tagging_loss=0.009137, over 3039818.90 frames. ], batch size: 57, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:33:09,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2832886.6666666665, ans=0.125
2023-11-24 12:33:28,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 424950
2023-11-24 12:33:40,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff2.min_abs, batch_count=2833020.0, ans=0.1
2023-11-24 12:34:10,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2833220.0, ans=0.0
2023-11-24 12:34:12,401 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4150, loss[loss=0.0737, simple_loss=0.1096, pruned_loss=0.01182, audio_tagging_loss=0.007082, over 16061.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09152, pruned_loss=0.01317, audio_tagging_loss=0.009005, over 3041274.03 frames. ], batch size: 59, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:34:20,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2833220.0, ans=0.1
2023-11-24 12:34:27,228 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.028e+01 8.532e+01 9.065e+01 9.755e+01 1.245e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-24 12:34:31,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425000
2023-11-24 12:34:45,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2833353.3333333335, ans=0.1
2023-11-24 12:34:57,040 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:35:00,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2833420.0, ans=0.2
2023-11-24 12:35:05,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2833486.6666666665, ans=0.125
2023-11-24 12:35:14,961 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4200, loss[loss=0.05371, simple_loss=0.0702, pruned_loss=0.009039, audio_tagging_loss=0.00957, over 13979.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09117, pruned_loss=0.01313, audio_tagging_loss=0.008965, over 3039054.40 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:35:34,445 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425050
2023-11-24 12:35:39,884 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.73 vs. limit=15.0
2023-11-24 12:35:59,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2833753.3333333335, ans=0.1
2023-11-24 12:36:18,760 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4250, loss[loss=0.05584, simple_loss=0.06763, pruned_loss=0.009874, audio_tagging_loss=0.01215, over 14566.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09144, pruned_loss=0.0131, audio_tagging_loss=0.008932, over 3037853.79 frames. ], batch size: 58, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:36:22,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff3.min_abs, batch_count=2833886.6666666665, ans=0.2
2023-11-24 12:36:23,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2833886.6666666665, ans=0.04949747468305833
2023-11-24 12:36:30,254 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.05 vs. limit=6.0
2023-11-24 12:36:32,903 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.335e+01 8.749e+01 9.313e+01 9.960e+01 2.008e+02, threshold=1.863e+02, percent-clipped=1.0
2023-11-24 12:36:36,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2833953.3333333335, ans=0.2
2023-11-24 12:36:37,223 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425100
2023-11-24 12:36:59,907 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.32 vs. limit=15.0
2023-11-24 12:37:20,017 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:37:20,831 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4300, loss[loss=0.05116, simple_loss=0.06175, pruned_loss=0.00878, audio_tagging_loss=0.0115, over 13994.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09126, pruned_loss=0.01323, audio_tagging_loss=0.008862, over 3038750.77 frames. ], batch size: 54, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:37:30,504 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2834220.0, ans=0.125
2023-11-24 12:37:40,043 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425150
2023-11-24 12:37:46,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2834353.3333333335, ans=0.125
2023-11-24 12:37:57,536 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.05 vs. limit=15.0
2023-11-24 12:38:00,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2834420.0, ans=0.2
2023-11-24 12:38:02,848 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2834420.0, ans=0.125
2023-11-24 12:38:04,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.87 vs. limit=15.0
2023-11-24 12:38:24,208 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4350, loss[loss=0.06095, simple_loss=0.07274, pruned_loss=0.01004, audio_tagging_loss=0.01454, over 14636.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09191, pruned_loss=0.01341, audio_tagging_loss=0.008761, over 3037795.46 frames. ], batch size: 56, lr: 1.89e-03, grad_scale: 8.0
2023-11-24 12:38:39,492 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.655e+01 9.322e+01 1.008e+02 1.169e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 12:38:42,570 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.41 vs. limit=10.0
2023-11-24 12:38:43,158 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425200
2023-11-24 12:38:59,267 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:39:02,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2834753.3333333335, ans=0.2
2023-11-24 12:39:27,400 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4400, loss[loss=0.08406, simple_loss=0.1237, pruned_loss=0.01523, audio_tagging_loss=0.006991, over 15047.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09122, pruned_loss=0.01312, audio_tagging_loss=0.008776, over 3033270.29 frames. ], batch size: 55, lr: 1.89e-03, grad_scale: 16.0
2023-11-24 12:39:45,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425250
2023-11-24 12:39:58,146 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.80 vs. limit=22.5
2023-11-24 12:40:18,225 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.99 vs. limit=15.0
2023-11-24 12:40:29,274 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4450, loss[loss=0.08154, simple_loss=0.1141, pruned_loss=0.01766, audio_tagging_loss=0.006807, over 14862.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09116, pruned_loss=0.01312, audio_tagging_loss=0.008741, over 3037354.76 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:40:44,625 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.474e+01 8.586e+01 9.357e+01 9.969e+01 1.625e+02, threshold=1.871e+02, percent-clipped=0.0
2023-11-24 12:40:44,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2835286.6666666665, ans=0.0
2023-11-24 12:40:48,580 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425300
2023-11-24 12:41:03,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2835353.3333333335, ans=0.125
2023-11-24 12:41:32,472 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4500, loss[loss=0.06459, simple_loss=0.09175, pruned_loss=0.009727, audio_tagging_loss=0.008988, over 14424.00 frames. ], tot_loss[loss=0.06813, simple_loss=0.0921, pruned_loss=0.01335, audio_tagging_loss=0.008729, over 3036132.03 frames. ], batch size: 53, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:41:51,040 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425350
2023-11-24 12:42:07,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2835686.6666666665, ans=0.125
2023-11-24 12:42:12,726 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:42:35,578 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4550, loss[loss=0.07403, simple_loss=0.09983, pruned_loss=0.01659, audio_tagging_loss=0.007534, over 15183.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09149, pruned_loss=0.01326, audio_tagging_loss=0.008718, over 3033734.33 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:42:39,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2835886.6666666665, ans=0.0
2023-11-24 12:42:47,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2835953.3333333335, ans=0.0
2023-11-24 12:42:48,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2835953.3333333335, ans=0.2
2023-11-24 12:42:50,431 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.548e+01 8.325e+01 9.085e+01 9.707e+01 1.236e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 12:42:54,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425400
2023-11-24 12:43:23,424 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 12:43:26,052 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2836153.3333333335, ans=0.2
2023-11-24 12:43:38,252 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4600, loss[loss=0.06764, simple_loss=0.08755, pruned_loss=0.01431, audio_tagging_loss=0.009553, over 14530.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09151, pruned_loss=0.01328, audio_tagging_loss=0.008789, over 3033404.50 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:43:57,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425450
2023-11-24 12:44:08,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_na.min_abs, batch_count=2836353.3333333335, ans=0.02
2023-11-24 12:44:13,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2836353.3333333335, ans=0.125
2023-11-24 12:44:24,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2836420.0, ans=0.125
2023-11-24 12:44:25,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.54 vs. limit=15.0
2023-11-24 12:44:41,248 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4650, loss[loss=0.07139, simple_loss=0.09443, pruned_loss=0.01496, audio_tagging_loss=0.009218, over 16482.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09101, pruned_loss=0.01321, audio_tagging_loss=0.008883, over 3033369.17 frames. ], batch size: 64, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:44:55,936 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.381e+01 8.503e+01 9.255e+01 1.001e+02 1.285e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 12:44:59,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425500
2023-11-24 12:45:03,454 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:45:06,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2836686.6666666665, ans=0.125
2023-11-24 12:45:20,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2836753.3333333335, ans=0.0
2023-11-24 12:45:25,141 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.69 vs. limit=15.0
2023-11-24 12:45:28,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2836753.3333333335, ans=0.125
2023-11-24 12:45:31,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2836820.0, ans=0.125
2023-11-24 12:45:38,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2836820.0, ans=0.07
2023-11-24 12:45:41,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2836820.0, ans=0.125
2023-11-24 12:45:43,859 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4700, loss[loss=0.07783, simple_loss=0.1055, pruned_loss=0.01635, audio_tagging_loss=0.008712, over 15120.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09133, pruned_loss=0.01335, audio_tagging_loss=0.00899, over 3034282.22 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:45:49,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2836886.6666666665, ans=0.125
2023-11-24 12:46:02,025 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425550
2023-11-24 12:46:06,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2836953.3333333335, ans=0.0
2023-11-24 12:46:07,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2837020.0, ans=0.1
2023-11-24 12:46:16,250 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.71 vs. limit=12.0
2023-11-24 12:46:21,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2837086.6666666665, ans=0.0
2023-11-24 12:46:38,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2837153.3333333335, ans=0.95
2023-11-24 12:46:45,792 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4750, loss[loss=0.05858, simple_loss=0.08601, pruned_loss=0.007483, audio_tagging_loss=0.008089, over 16305.00 frames. ], tot_loss[loss=0.06839, simple_loss=0.09201, pruned_loss=0.01341, audio_tagging_loss=0.008977, over 3037333.95 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:46:47,428 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.21 vs. limit=22.5
2023-11-24 12:47:01,268 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.647e+01 9.446e+01 1.032e+02 1.298e+02, threshold=1.889e+02, percent-clipped=0.0
2023-11-24 12:47:05,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425600
2023-11-24 12:47:23,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2837420.0, ans=0.1
2023-11-24 12:47:24,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2837420.0, ans=0.125
2023-11-24 12:47:34,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2837420.0, ans=0.125
2023-11-24 12:47:35,952 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.83 vs. limit=15.0
2023-11-24 12:47:49,891 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4800, loss[loss=0.06638, simple_loss=0.09106, pruned_loss=0.01152, audio_tagging_loss=0.009331, over 14502.00 frames. ], tot_loss[loss=0.06901, simple_loss=0.0929, pruned_loss=0.01353, audio_tagging_loss=0.009037, over 3043996.07 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:47:52,263 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.35 vs. limit=15.0
2023-11-24 12:47:59,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2837553.3333333335, ans=0.1
2023-11-24 12:48:04,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2837620.0, ans=0.0
2023-11-24 12:48:09,541 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425650
2023-11-24 12:48:16,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2837686.6666666665, ans=0.125
2023-11-24 12:48:53,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2837886.6666666665, ans=0.125
2023-11-24 12:48:54,401 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4850, loss[loss=0.05506, simple_loss=0.07854, pruned_loss=0.008241, audio_tagging_loss=0.007545, over 13910.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09249, pruned_loss=0.01333, audio_tagging_loss=0.009121, over 3040679.77 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:48:56,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2837886.6666666665, ans=0.125
2023-11-24 12:48:57,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2837886.6666666665, ans=0.125
2023-11-24 12:48:58,677 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.29 vs. limit=15.0
2023-11-24 12:49:05,731 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.18 vs. limit=10.0
2023-11-24 12:49:08,624 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.265e+01 8.570e+01 9.281e+01 9.821e+01 1.175e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-24 12:49:12,366 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425700
2023-11-24 12:49:18,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2838020.0, ans=0.125
2023-11-24 12:49:26,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2838020.0, ans=0.2
2023-11-24 12:49:30,417 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:49:48,379 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.60 vs. limit=15.0
2023-11-24 12:49:55,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2838220.0, ans=0.1
2023-11-24 12:49:55,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2838220.0, ans=0.125
2023-11-24 12:49:56,102 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4900, loss[loss=0.08495, simple_loss=0.1192, pruned_loss=0.01655, audio_tagging_loss=0.008789, over 14581.00 frames. ], tot_loss[loss=0.06863, simple_loss=0.09217, pruned_loss=0.01339, audio_tagging_loss=0.009144, over 3039725.10 frames. ], batch size: 52, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:50:15,257 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425750
2023-11-24 12:50:22,243 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.26 vs. limit=15.0
2023-11-24 12:50:34,288 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2838420.0, ans=0.2
2023-11-24 12:50:50,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2838486.6666666665, ans=0.0
2023-11-24 12:50:58,322 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 4950, loss[loss=0.05744, simple_loss=0.07019, pruned_loss=0.01313, audio_tagging_loss=0.009209, over 14734.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09148, pruned_loss=0.01336, audio_tagging_loss=0.009049, over 3035187.81 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:50:58,806 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 12:51:03,151 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.95 vs. limit=22.5
2023-11-24 12:51:14,270 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.977e+01 8.587e+01 9.264e+01 9.833e+01 1.255e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 12:51:17,940 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425800
2023-11-24 12:52:02,006 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5000, loss[loss=0.07401, simple_loss=0.1006, pruned_loss=0.01703, audio_tagging_loss=0.006672, over 15564.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09187, pruned_loss=0.0132, audio_tagging_loss=0.008899, over 3038036.17 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:52:14,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.36 vs. limit=15.0
2023-11-24 12:52:19,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425850
2023-11-24 12:52:38,084 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.96 vs. limit=6.0
2023-11-24 12:52:43,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2839086.6666666665, ans=0.04949747468305833
2023-11-24 12:52:46,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2839086.6666666665, ans=0.125
2023-11-24 12:52:51,108 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.92 vs. limit=22.5
2023-11-24 12:52:57,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2839153.3333333335, ans=0.125
2023-11-24 12:53:03,499 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5050, loss[loss=0.06495, simple_loss=0.08875, pruned_loss=0.01342, audio_tagging_loss=0.007157, over 15130.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09223, pruned_loss=0.01327, audio_tagging_loss=0.008766, over 3037630.82 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:53:17,862 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.090e+01 8.266e+01 8.922e+01 9.681e+01 1.367e+02, threshold=1.784e+02, percent-clipped=0.0
2023-11-24 12:53:20,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2839286.6666666665, ans=0.0
2023-11-24 12:53:22,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425900
2023-11-24 12:53:22,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2839286.6666666665, ans=0.1
2023-11-24 12:53:25,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2839286.6666666665, ans=0.1
2023-11-24 12:53:25,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2839286.6666666665, ans=0.1
2023-11-24 12:53:56,407 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.66 vs. limit=10.0
2023-11-24 12:54:06,742 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5100, loss[loss=0.06447, simple_loss=0.09177, pruned_loss=0.01022, audio_tagging_loss=0.00837, over 15922.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09223, pruned_loss=0.01326, audio_tagging_loss=0.0088, over 3034690.17 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:54:26,941 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 425950
2023-11-24 12:54:51,951 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.59 vs. limit=15.0
2023-11-24 12:55:11,170 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5150, loss[loss=0.06374, simple_loss=0.07869, pruned_loss=0.01435, audio_tagging_loss=0.01005, over 15034.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09185, pruned_loss=0.01332, audio_tagging_loss=0.008854, over 3035160.85 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:55:26,160 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.149e+01 8.369e+01 9.000e+01 9.837e+01 1.217e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 12:55:29,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426000
2023-11-24 12:55:49,461 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.55 vs. limit=12.0
2023-11-24 12:56:01,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2840153.3333333335, ans=0.0
2023-11-24 12:56:01,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2840153.3333333335, ans=0.1
2023-11-24 12:56:14,386 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5200, loss[loss=0.05753, simple_loss=0.08598, pruned_loss=0.007605, audio_tagging_loss=0.00694, over 15013.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09242, pruned_loss=0.01336, audio_tagging_loss=0.00885, over 3039963.75 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 12:56:24,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2840220.0, ans=0.1
2023-11-24 12:56:25,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2840286.6666666665, ans=0.0
2023-11-24 12:56:32,697 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426050
2023-11-24 12:56:45,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2840353.3333333335, ans=0.1
2023-11-24 12:56:50,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2840420.0, ans=0.125
2023-11-24 12:56:52,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2840420.0, ans=0.2
2023-11-24 12:56:54,698 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=9.72 vs. limit=22.5
2023-11-24 12:56:55,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2840420.0, ans=0.0
2023-11-24 12:56:55,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2840420.0, ans=0.1
2023-11-24 12:57:15,465 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5250, loss[loss=0.06431, simple_loss=0.07973, pruned_loss=0.01409, audio_tagging_loss=0.01036, over 14945.00 frames. ], tot_loss[loss=0.06885, simple_loss=0.09291, pruned_loss=0.01356, audio_tagging_loss=0.008838, over 3043163.42 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:57:19,287 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.33 vs. limit=6.0
2023-11-24 12:57:32,385 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.236e+01 8.528e+01 9.139e+01 1.004e+02 1.210e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 12:57:35,501 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426100
2023-11-24 12:57:39,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2840620.0, ans=0.125
2023-11-24 12:58:19,037 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5300, loss[loss=0.04995, simple_loss=0.06562, pruned_loss=0.007683, audio_tagging_loss=0.009456, over 14662.00 frames. ], tot_loss[loss=0.06883, simple_loss=0.09295, pruned_loss=0.01358, audio_tagging_loss=0.008774, over 3036300.64 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:58:35,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2840953.3333333335, ans=0.0
2023-11-24 12:58:38,035 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426150
2023-11-24 12:58:46,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2841020.0, ans=0.0
2023-11-24 12:58:53,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2841020.0, ans=0.1
2023-11-24 12:59:01,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2841086.6666666665, ans=0.125
2023-11-24 12:59:02,461 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.93 vs. limit=15.0
2023-11-24 12:59:22,042 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5350, loss[loss=0.04934, simple_loss=0.0699, pruned_loss=0.004921, audio_tagging_loss=0.009469, over 14494.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.0922, pruned_loss=0.01338, audio_tagging_loss=0.00874, over 3028986.08 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 12:59:25,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2841220.0, ans=0.1
2023-11-24 12:59:35,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2841286.6666666665, ans=0.0
2023-11-24 12:59:37,775 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.311e+01 8.528e+01 9.137e+01 9.887e+01 1.205e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 12:59:40,348 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426200
2023-11-24 12:59:49,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2841353.3333333335, ans=0.1
2023-11-24 13:00:20,012 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:00:24,517 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5400, loss[loss=0.06867, simple_loss=0.08624, pruned_loss=0.01621, audio_tagging_loss=0.009344, over 14746.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09121, pruned_loss=0.01314, audio_tagging_loss=0.008843, over 3032220.78 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:00:33,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2841553.3333333335, ans=0.1
2023-11-24 13:00:40,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2841620.0, ans=0.05
2023-11-24 13:00:40,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2841620.0, ans=10.0
2023-11-24 13:00:42,972 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.90 vs. limit=22.5
2023-11-24 13:00:43,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426250
2023-11-24 13:01:02,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2841753.3333333335, ans=0.09899494936611666
2023-11-24 13:01:12,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2841753.3333333335, ans=0.1
2023-11-24 13:01:27,629 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5450, loss[loss=0.06239, simple_loss=0.08727, pruned_loss=0.01179, audio_tagging_loss=0.006961, over 14837.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09057, pruned_loss=0.01306, audio_tagging_loss=0.008975, over 3030874.62 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:01:31,807 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.30 vs. limit=15.0
2023-11-24 13:01:36,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2841886.6666666665, ans=0.1
2023-11-24 13:01:43,550 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.576e+01 9.179e+01 9.817e+01 1.405e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 13:01:46,721 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426300
2023-11-24 13:02:01,824 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.18 vs. limit=15.0
2023-11-24 13:02:09,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.23 vs. limit=6.0
2023-11-24 13:02:30,305 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5500, loss[loss=0.05916, simple_loss=0.07964, pruned_loss=0.009591, audio_tagging_loss=0.009745, over 15082.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09032, pruned_loss=0.01304, audio_tagging_loss=0.00903, over 3030377.13 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:02:38,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2842220.0, ans=0.0
2023-11-24 13:02:48,921 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426350
2023-11-24 13:02:52,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2842286.6666666665, ans=0.125
2023-11-24 13:03:03,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2842353.3333333335, ans=0.2
2023-11-24 13:03:26,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=12.81 vs. limit=15.0
2023-11-24 13:03:32,886 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5550, loss[loss=0.05843, simple_loss=0.07831, pruned_loss=0.009077, audio_tagging_loss=0.0102, over 14487.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09112, pruned_loss=0.01324, audio_tagging_loss=0.009028, over 3023814.35 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 13:03:37,081 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.03 vs. limit=15.0
2023-11-24 13:03:50,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2842620.0, ans=0.125
2023-11-24 13:03:50,770 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.849e+01 8.724e+01 9.337e+01 1.024e+02 1.345e+02, threshold=1.867e+02, percent-clipped=0.0
2023-11-24 13:03:52,054 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426400
2023-11-24 13:04:03,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2842686.6666666665, ans=0.0
2023-11-24 13:04:10,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.39 vs. limit=8.0
2023-11-24 13:04:18,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2842753.3333333335, ans=0.1
2023-11-24 13:04:32,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.49 vs. limit=15.0
2023-11-24 13:04:36,533 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5600, loss[loss=0.07763, simple_loss=0.1052, pruned_loss=0.01347, audio_tagging_loss=0.01158, over 16032.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09207, pruned_loss=0.01344, audio_tagging_loss=0.009018, over 3029963.93 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:04:49,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2842953.3333333335, ans=0.125
2023-11-24 13:04:55,198 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426450
2023-11-24 13:04:58,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2842953.3333333335, ans=0.125
2023-11-24 13:05:10,950 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.71 vs. limit=15.0
2023-11-24 13:05:16,912 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.29 vs. limit=6.0
2023-11-24 13:05:20,379 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:05:23,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2843086.6666666665, ans=0.1
2023-11-24 13:05:26,507 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.47 vs. limit=15.0
2023-11-24 13:05:27,469 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.91 vs. limit=15.0
2023-11-24 13:05:39,338 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5650, loss[loss=0.07041, simple_loss=0.1001, pruned_loss=0.01277, audio_tagging_loss=0.007564, over 14179.00 frames. ], tot_loss[loss=0.0684, simple_loss=0.0919, pruned_loss=0.01342, audio_tagging_loss=0.00903, over 3039664.44 frames. ], batch size: 53, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:05:40,009 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=6.66 vs. limit=15.0
2023-11-24 13:05:41,891 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:05:43,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2843220.0, ans=0.125
2023-11-24 13:05:49,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2843220.0, ans=0.125
2023-11-24 13:05:56,550 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.631e+01 8.507e+01 8.997e+01 9.661e+01 1.393e+02, threshold=1.799e+02, percent-clipped=0.0
2023-11-24 13:05:57,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426500
2023-11-24 13:06:03,193 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.07 vs. limit=15.0
2023-11-24 13:06:13,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2843353.3333333335, ans=0.125
2023-11-24 13:06:26,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2843420.0, ans=0.2
2023-11-24 13:06:28,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2843486.6666666665, ans=0.125
2023-11-24 13:06:39,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2843486.6666666665, ans=0.1
2023-11-24 13:06:41,761 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5700, loss[loss=0.04297, simple_loss=0.05008, pruned_loss=0.009251, audio_tagging_loss=0.008674, over 14501.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09166, pruned_loss=0.01332, audio_tagging_loss=0.009121, over 3042687.48 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:06:48,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2843553.3333333335, ans=0.04949747468305833
2023-11-24 13:06:59,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2843620.0, ans=0.125
2023-11-24 13:07:01,124 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426550
2023-11-24 13:07:07,317 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2843686.6666666665, ans=0.0
2023-11-24 13:07:10,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2843686.6666666665, ans=10.0
2023-11-24 13:07:23,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2843753.3333333335, ans=0.2
2023-11-24 13:07:37,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2843820.0, ans=0.0
2023-11-24 13:07:44,425 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5750, loss[loss=0.0785, simple_loss=0.1094, pruned_loss=0.01788, audio_tagging_loss=0.005919, over 15834.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09121, pruned_loss=0.01324, audio_tagging_loss=0.009055, over 3043505.34 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:08:02,008 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.294e+01 8.384e+01 8.873e+01 9.530e+01 1.243e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-24 13:08:03,351 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426600
2023-11-24 13:08:06,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2843953.3333333335, ans=0.1
2023-11-24 13:08:06,590 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.22 vs. limit=15.0
2023-11-24 13:08:27,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2844086.6666666665, ans=0.125
2023-11-24 13:08:35,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2844153.3333333335, ans=0.0
2023-11-24 13:08:40,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2844153.3333333335, ans=0.025
2023-11-24 13:08:48,088 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5800, loss[loss=0.05096, simple_loss=0.07459, pruned_loss=0.007231, audio_tagging_loss=0.006429, over 14876.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09098, pruned_loss=0.01324, audio_tagging_loss=0.008952, over 3038897.99 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:09:04,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2844286.6666666665, ans=0.1
2023-11-24 13:09:05,929 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426650
2023-11-24 13:09:08,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2844286.6666666665, ans=0.125
2023-11-24 13:09:21,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2844353.3333333335, ans=0.125
2023-11-24 13:09:23,153 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2844353.3333333335, ans=0.125
2023-11-24 13:09:25,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2844420.0, ans=0.125
2023-11-24 13:09:25,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2844420.0, ans=0.125
2023-11-24 13:09:34,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2844420.0, ans=0.2
2023-11-24 13:09:34,341 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2844420.0, ans=0.0
2023-11-24 13:09:44,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2844486.6666666665, ans=0.2
2023-11-24 13:09:49,431 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5850, loss[loss=0.08486, simple_loss=0.1174, pruned_loss=0.01807, audio_tagging_loss=0.008072, over 15677.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09039, pruned_loss=0.01312, audio_tagging_loss=0.008838, over 3036154.99 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:10:02,169 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2844620.0, ans=0.2
2023-11-24 13:10:05,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2844620.0, ans=0.125
2023-11-24 13:10:07,024 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.013e+01 8.489e+01 9.123e+01 9.809e+01 1.343e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-24 13:10:08,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426700
2023-11-24 13:10:29,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2844753.3333333335, ans=0.2
2023-11-24 13:10:52,093 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5900, loss[loss=0.07795, simple_loss=0.1101, pruned_loss=0.01466, audio_tagging_loss=0.008228, over 15281.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09049, pruned_loss=0.01292, audio_tagging_loss=0.008804, over 3040608.76 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:11:11,104 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426750
2023-11-24 13:11:31,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2845086.6666666665, ans=0.125
2023-11-24 13:11:35,842 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=9.34 vs. limit=15.0
2023-11-24 13:11:45,054 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.47 vs. limit=15.0
2023-11-24 13:11:50,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2845153.3333333335, ans=0.125
2023-11-24 13:11:51,342 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.12 vs. limit=15.0
2023-11-24 13:11:53,916 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 5950, loss[loss=0.06854, simple_loss=0.095, pruned_loss=0.01398, audio_tagging_loss=0.007063, over 15194.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09061, pruned_loss=0.01291, audio_tagging_loss=0.008851, over 3041940.18 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:12:05,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2845286.6666666665, ans=0.09899494936611666
2023-11-24 13:12:06,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=2845286.6666666665, ans=15.0
2023-11-24 13:12:11,019 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.386e+01 9.064e+01 9.655e+01 1.412e+02, threshold=1.813e+02, percent-clipped=0.0
2023-11-24 13:12:12,326 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426800
2023-11-24 13:12:21,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2845353.3333333335, ans=0.125
2023-11-24 13:12:21,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2845353.3333333335, ans=0.1
2023-11-24 13:12:34,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2845420.0, ans=0.125
2023-11-24 13:12:34,628 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:12:49,217 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2845486.6666666665, ans=0.125
2023-11-24 13:12:56,170 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6000, loss[loss=0.06219, simple_loss=0.08409, pruned_loss=0.01228, audio_tagging_loss=0.007862, over 14229.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.09078, pruned_loss=0.01292, audio_tagging_loss=0.008741, over 3044068.17 frames. ], batch size: 52, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:12:56,173 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 13:13:35,144 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.4627, 4.5580, 4.4670, 4.5102], device='cuda:0')
2023-11-24 13:13:36,418 INFO [train_asr.py:1253] (0/4) Epoch 36, validation: loss=0.05813, simple_loss=0.0509, pruned_loss=0.005269, audio_tagging_loss=0.02741, over 4681554.00 frames. 
2023-11-24 13:13:36,419 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 13:13:38,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2845553.3333333335, ans=0.125
2023-11-24 13:13:53,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2845620.0, ans=0.2
2023-11-24 13:13:54,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426850
2023-11-24 13:13:57,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2845620.0, ans=0.125
2023-11-24 13:14:20,383 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:14:36,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2845820.0, ans=0.0
2023-11-24 13:14:38,878 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6050, loss[loss=0.06249, simple_loss=0.08251, pruned_loss=0.01021, audio_tagging_loss=0.01102, over 14685.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.091, pruned_loss=0.01284, audio_tagging_loss=0.008758, over 3044833.63 frames. ], batch size: 53, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:14:39,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2845886.6666666665, ans=0.035
2023-11-24 13:14:45,362 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.18 vs. limit=15.0
2023-11-24 13:14:54,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2845953.3333333335, ans=0.035
2023-11-24 13:14:55,410 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.206e+01 8.477e+01 9.229e+01 1.011e+02 1.265e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 13:14:56,717 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426900
2023-11-24 13:15:15,632 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2846086.6666666665, ans=0.125
2023-11-24 13:15:30,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2846153.3333333335, ans=0.125
2023-11-24 13:15:39,759 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6100, loss[loss=0.09006, simple_loss=0.123, pruned_loss=0.02118, audio_tagging_loss=0.00737, over 15575.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09156, pruned_loss=0.01295, audio_tagging_loss=0.008772, over 3046060.39 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:15:46,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2846220.0, ans=0.1
2023-11-24 13:15:57,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2846286.6666666665, ans=0.125
2023-11-24 13:15:59,096 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 426950
2023-11-24 13:16:42,234 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6150, loss[loss=0.05776, simple_loss=0.07735, pruned_loss=0.01126, audio_tagging_loss=0.007823, over 15178.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09112, pruned_loss=0.01295, audio_tagging_loss=0.008812, over 3043676.13 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:16:51,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2846553.3333333335, ans=0.0
2023-11-24 13:17:00,448 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.744e+01 8.380e+01 9.080e+01 9.869e+01 1.269e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 13:17:01,754 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427000
2023-11-24 13:17:02,344 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.42 vs. limit=15.0
2023-11-24 13:17:11,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2846686.6666666665, ans=0.125
2023-11-24 13:17:34,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2846820.0, ans=0.125
2023-11-24 13:17:46,177 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6200, loss[loss=0.06077, simple_loss=0.07178, pruned_loss=0.01645, audio_tagging_loss=0.008424, over 14925.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09106, pruned_loss=0.01302, audio_tagging_loss=0.008847, over 3043899.39 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:17:48,778 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2846886.6666666665, ans=0.125
2023-11-24 13:18:04,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427050
2023-11-24 13:18:04,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2846953.3333333335, ans=0.125
2023-11-24 13:18:11,500 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2847020.0, ans=0.0
2023-11-24 13:18:13,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2847020.0, ans=0.07
2023-11-24 13:18:23,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2847086.6666666665, ans=0.5
2023-11-24 13:18:26,191 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=15.29 vs. limit=15.0
2023-11-24 13:18:39,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2847153.3333333335, ans=0.1
2023-11-24 13:18:48,612 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6250, loss[loss=0.06665, simple_loss=0.08232, pruned_loss=0.01433, audio_tagging_loss=0.01116, over 14955.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09079, pruned_loss=0.01304, audio_tagging_loss=0.00901, over 3043578.12 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:19:00,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2847286.6666666665, ans=0.2
2023-11-24 13:19:04,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2847286.6666666665, ans=0.1
2023-11-24 13:19:07,101 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.177e+01 8.748e+01 9.375e+01 1.005e+02 1.259e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 13:19:07,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427100
2023-11-24 13:19:34,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2847420.0, ans=0.2
2023-11-24 13:19:43,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2847486.6666666665, ans=0.1
2023-11-24 13:19:43,299 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.23 vs. limit=15.0
2023-11-24 13:19:51,312 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6300, loss[loss=0.06206, simple_loss=0.08426, pruned_loss=0.01109, audio_tagging_loss=0.008838, over 14103.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09115, pruned_loss=0.01314, audio_tagging_loss=0.008985, over 3035202.60 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:19:52,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2847553.3333333335, ans=0.1
2023-11-24 13:19:54,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2847553.3333333335, ans=0.0
2023-11-24 13:20:09,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.17 vs. limit=12.0
2023-11-24 13:20:11,586 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427150
2023-11-24 13:20:38,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=12.07 vs. limit=15.0
2023-11-24 13:20:55,129 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6350, loss[loss=0.04366, simple_loss=0.05877, pruned_loss=0.006064, audio_tagging_loss=0.008216, over 14486.00 frames. ], tot_loss[loss=0.06673, simple_loss=0.08972, pruned_loss=0.01273, audio_tagging_loss=0.009145, over 3031846.64 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:21:01,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2847886.6666666665, ans=0.1
2023-11-24 13:21:11,948 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.25 vs. limit=15.0
2023-11-24 13:21:13,180 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.48 vs. limit=15.0
2023-11-24 13:21:13,484 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.901e+01 8.568e+01 9.156e+01 9.751e+01 1.252e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 13:21:13,701 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427200
2023-11-24 13:21:13,792 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2847953.3333333335, ans=0.125
2023-11-24 13:21:37,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2848086.6666666665, ans=0.2
2023-11-24 13:21:57,646 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6400, loss[loss=0.08242, simple_loss=0.1171, pruned_loss=0.01622, audio_tagging_loss=0.007652, over 15456.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.08898, pruned_loss=0.01271, audio_tagging_loss=0.00927, over 3035253.08 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:22:15,644 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427250
2023-11-24 13:22:26,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2848353.3333333335, ans=0.125
2023-11-24 13:22:31,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2848353.3333333335, ans=0.0
2023-11-24 13:22:36,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2848420.0, ans=0.0
2023-11-24 13:22:37,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2848420.0, ans=0.0
2023-11-24 13:22:39,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2848420.0, ans=0.125
2023-11-24 13:22:55,014 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:22:59,740 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6450, loss[loss=0.06083, simple_loss=0.08112, pruned_loss=0.01267, audio_tagging_loss=0.007602, over 15244.00 frames. ], tot_loss[loss=0.0664, simple_loss=0.08888, pruned_loss=0.01269, audio_tagging_loss=0.009275, over 3038488.65 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:23:16,327 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.39 vs. limit=22.5
2023-11-24 13:23:18,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2848620.0, ans=0.125
2023-11-24 13:23:18,988 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.428e+01 8.325e+01 9.052e+01 9.909e+01 1.282e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 13:23:19,156 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427300
2023-11-24 13:23:28,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2848686.6666666665, ans=0.125
2023-11-24 13:23:41,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.20 vs. limit=15.0
2023-11-24 13:23:58,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2848820.0, ans=0.125
2023-11-24 13:24:03,308 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6500, loss[loss=0.0572, simple_loss=0.07766, pruned_loss=0.01035, audio_tagging_loss=0.008026, over 15190.00 frames. ], tot_loss[loss=0.06688, simple_loss=0.08981, pruned_loss=0.01282, audio_tagging_loss=0.009155, over 3042284.28 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:24:11,959 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.53 vs. limit=6.0
2023-11-24 13:24:21,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2848953.3333333335, ans=0.1
2023-11-24 13:24:22,745 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427350
2023-11-24 13:24:22,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2848953.3333333335, ans=0.125
2023-11-24 13:24:24,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2848953.3333333335, ans=0.125
2023-11-24 13:24:41,973 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.65 vs. limit=10.0
2023-11-24 13:24:46,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2849086.6666666665, ans=0.1
2023-11-24 13:24:59,877 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=12.0
2023-11-24 13:25:07,000 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6550, loss[loss=0.07309, simple_loss=0.1021, pruned_loss=0.01446, audio_tagging_loss=0.007572, over 15246.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.0909, pruned_loss=0.01307, audio_tagging_loss=0.009043, over 3039511.71 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:25:15,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2849220.0, ans=0.125
2023-11-24 13:25:18,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2849286.6666666665, ans=0.07
2023-11-24 13:25:21,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2849286.6666666665, ans=0.0
2023-11-24 13:25:25,371 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427400
2023-11-24 13:25:26,380 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.536e+01 9.192e+01 9.836e+01 1.277e+02, threshold=1.838e+02, percent-clipped=0.0
2023-11-24 13:25:28,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2849286.6666666665, ans=0.1
2023-11-24 13:25:42,297 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:26:01,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2849486.6666666665, ans=0.09899494936611666
2023-11-24 13:26:09,477 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6600, loss[loss=0.07417, simple_loss=0.1018, pruned_loss=0.0166, audio_tagging_loss=0.006688, over 15715.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09107, pruned_loss=0.01312, audio_tagging_loss=0.008915, over 3037598.78 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:26:11,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2849553.3333333335, ans=0.0
2023-11-24 13:26:15,126 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.38 vs. limit=6.0
2023-11-24 13:26:17,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2849553.3333333335, ans=10.0
2023-11-24 13:26:28,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427450
2023-11-24 13:26:33,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2849620.0, ans=0.125
2023-11-24 13:27:04,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2849820.0, ans=0.0
2023-11-24 13:27:13,527 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6650, loss[loss=0.06953, simple_loss=0.0939, pruned_loss=0.01301, audio_tagging_loss=0.009575, over 14694.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.09064, pruned_loss=0.01305, audio_tagging_loss=0.008911, over 3042771.28 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:27:22,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2849886.6666666665, ans=0.0
2023-11-24 13:27:31,972 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427500
2023-11-24 13:27:33,576 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.369e+01 8.517e+01 9.117e+01 9.877e+01 1.434e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 13:27:39,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2850020.0, ans=0.125
2023-11-24 13:28:16,075 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6700, loss[loss=0.05819, simple_loss=0.08064, pruned_loss=0.008319, audio_tagging_loss=0.009554, over 15525.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09124, pruned_loss=0.01306, audio_tagging_loss=0.008837, over 3047137.31 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:28:20,296 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.14 vs. limit=15.0
2023-11-24 13:28:27,970 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.20 vs. limit=15.0
2023-11-24 13:28:34,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427550
2023-11-24 13:28:36,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2850286.6666666665, ans=0.0
2023-11-24 13:28:42,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2850353.3333333335, ans=0.125
2023-11-24 13:28:50,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2850353.3333333335, ans=0.125
2023-11-24 13:28:53,543 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.62 vs. limit=22.5
2023-11-24 13:28:54,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=2850420.0, ans=15.0
2023-11-24 13:29:02,512 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.80 vs. limit=22.5
2023-11-24 13:29:15,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2850486.6666666665, ans=0.1
2023-11-24 13:29:19,042 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6750, loss[loss=0.08204, simple_loss=0.1142, pruned_loss=0.01755, audio_tagging_loss=0.007382, over 15977.00 frames. ], tot_loss[loss=0.06777, simple_loss=0.09138, pruned_loss=0.01322, audio_tagging_loss=0.00886, over 3046274.74 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:29:31,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2850620.0, ans=0.1
2023-11-24 13:29:32,067 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.69 vs. limit=15.0
2023-11-24 13:29:38,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427600
2023-11-24 13:29:38,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2850620.0, ans=0.0
2023-11-24 13:29:39,145 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.316e+01 8.434e+01 8.966e+01 9.987e+01 1.535e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 13:29:53,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2850686.6666666665, ans=0.0
2023-11-24 13:30:06,514 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.43 vs. limit=12.0
2023-11-24 13:30:07,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2850753.3333333335, ans=0.1
2023-11-24 13:30:22,663 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6800, loss[loss=0.06479, simple_loss=0.08455, pruned_loss=0.015, audio_tagging_loss=0.007516, over 15046.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09115, pruned_loss=0.0131, audio_tagging_loss=0.008845, over 3048397.05 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:30:25,446 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2850886.6666666665, ans=0.2
2023-11-24 13:30:34,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2850953.3333333335, ans=0.2
2023-11-24 13:30:40,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2850953.3333333335, ans=0.125
2023-11-24 13:30:41,078 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427650
2023-11-24 13:30:45,377 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.27 vs. limit=15.0
2023-11-24 13:30:58,876 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.07 vs. limit=22.5
2023-11-24 13:30:59,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2851086.6666666665, ans=0.1
2023-11-24 13:31:07,732 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.44 vs. limit=15.0
2023-11-24 13:31:08,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2851086.6666666665, ans=0.125
2023-11-24 13:31:22,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2851153.3333333335, ans=0.2
2023-11-24 13:31:24,679 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6850, loss[loss=0.05919, simple_loss=0.08128, pruned_loss=0.009343, audio_tagging_loss=0.009207, over 15330.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09096, pruned_loss=0.01317, audio_tagging_loss=0.008846, over 3040883.51 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:31:30,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2851220.0, ans=0.015
2023-11-24 13:31:43,181 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427700
2023-11-24 13:31:43,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2851286.6666666665, ans=0.07
2023-11-24 13:31:44,240 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.170e+01 8.309e+01 9.090e+01 9.871e+01 1.187e+02, threshold=1.818e+02, percent-clipped=0.0
2023-11-24 13:31:51,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2851353.3333333335, ans=0.125
2023-11-24 13:32:20,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2851486.6666666665, ans=0.125
2023-11-24 13:32:24,404 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.60 vs. limit=22.5
2023-11-24 13:32:26,282 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6900, loss[loss=0.06233, simple_loss=0.08817, pruned_loss=0.01059, audio_tagging_loss=0.007658, over 15169.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09077, pruned_loss=0.01303, audio_tagging_loss=0.008777, over 3041977.01 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:32:44,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2851620.0, ans=0.07
2023-11-24 13:32:45,919 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427750
2023-11-24 13:32:46,053 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:32:55,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2851686.6666666665, ans=0.125
2023-11-24 13:33:11,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2851753.3333333335, ans=0.0
2023-11-24 13:33:14,573 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:33:18,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2851820.0, ans=0.0
2023-11-24 13:33:25,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2851820.0, ans=0.125
2023-11-24 13:33:28,827 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 6950, loss[loss=0.07409, simple_loss=0.09906, pruned_loss=0.01756, audio_tagging_loss=0.007001, over 15829.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09144, pruned_loss=0.01301, audio_tagging_loss=0.008822, over 3040238.38 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:33:47,379 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427800
2023-11-24 13:33:50,032 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.156e+01 8.545e+01 9.109e+01 9.801e+01 1.234e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 13:33:56,664 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.43 vs. limit=15.0
2023-11-24 13:34:31,859 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7000, loss[loss=0.06936, simple_loss=0.09018, pruned_loss=0.01453, audio_tagging_loss=0.009734, over 15909.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09093, pruned_loss=0.01306, audio_tagging_loss=0.008855, over 3039979.42 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:34:35,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2852220.0, ans=0.1
2023-11-24 13:34:36,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2852220.0, ans=0.0
2023-11-24 13:34:49,881 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427850
2023-11-24 13:35:01,985 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2852353.3333333335, ans=0.025
2023-11-24 13:35:04,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2852353.3333333335, ans=0.125
2023-11-24 13:35:15,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2852420.0, ans=0.1
2023-11-24 13:35:20,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2852420.0, ans=0.0
2023-11-24 13:35:34,280 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7050, loss[loss=0.06494, simple_loss=0.08562, pruned_loss=0.01073, audio_tagging_loss=0.0114, over 15074.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.09034, pruned_loss=0.01304, audio_tagging_loss=0.008909, over 3042360.29 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:35:53,298 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427900
2023-11-24 13:35:57,374 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.121e+01 8.577e+01 9.267e+01 9.852e+01 1.175e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 13:36:14,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=2852753.3333333335, ans=15.0
2023-11-24 13:36:32,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2852820.0, ans=0.0
2023-11-24 13:36:37,830 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7100, loss[loss=0.0784, simple_loss=0.1072, pruned_loss=0.01367, audio_tagging_loss=0.01116, over 15686.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09043, pruned_loss=0.01284, audio_tagging_loss=0.008986, over 3034380.78 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 13:36:56,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 427950
2023-11-24 13:37:18,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2853086.6666666665, ans=0.09899494936611666
2023-11-24 13:37:24,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2853086.6666666665, ans=0.0
2023-11-24 13:37:32,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2853153.3333333335, ans=0.125
2023-11-24 13:37:33,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.min_positive, batch_count=2853153.3333333335, ans=0.05
2023-11-24 13:37:40,818 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7150, loss[loss=0.08909, simple_loss=0.1245, pruned_loss=0.01979, audio_tagging_loss=0.007063, over 15494.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09041, pruned_loss=0.0129, audio_tagging_loss=0.009007, over 3039041.73 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 13:37:45,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2853220.0, ans=0.0
2023-11-24 13:37:59,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428000
2023-11-24 13:38:00,893 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-428000.pt
2023-11-24 13:38:05,903 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.129e+01 8.644e+01 9.347e+01 1.029e+02 1.240e+02, threshold=1.869e+02, percent-clipped=0.0
2023-11-24 13:38:06,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2853286.6666666665, ans=0.125
2023-11-24 13:38:18,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2853353.3333333335, ans=0.04949747468305833
2023-11-24 13:38:19,446 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:38:27,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2853420.0, ans=0.125
2023-11-24 13:38:30,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2853420.0, ans=0.125
2023-11-24 13:38:33,289 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2853486.6666666665, ans=0.125
2023-11-24 13:38:38,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2853486.6666666665, ans=0.0
2023-11-24 13:38:44,862 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.02 vs. limit=15.0
2023-11-24 13:38:45,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2853553.3333333335, ans=0.125
2023-11-24 13:38:46,555 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7200, loss[loss=0.06438, simple_loss=0.08083, pruned_loss=0.01248, audio_tagging_loss=0.01149, over 15339.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09111, pruned_loss=0.01306, audio_tagging_loss=0.009038, over 3045870.65 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:39:04,914 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428050
2023-11-24 13:39:08,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2853620.0, ans=0.125
2023-11-24 13:39:16,192 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.58 vs. limit=15.0
2023-11-24 13:39:17,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2853686.6666666665, ans=0.1
2023-11-24 13:39:20,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2853686.6666666665, ans=0.125
2023-11-24 13:39:36,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2853820.0, ans=0.125
2023-11-24 13:39:36,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2853820.0, ans=0.5
2023-11-24 13:39:39,190 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.21 vs. limit=10.0
2023-11-24 13:39:48,303 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7250, loss[loss=0.08346, simple_loss=0.117, pruned_loss=0.01523, audio_tagging_loss=0.009738, over 16696.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09126, pruned_loss=0.01311, audio_tagging_loss=0.009059, over 3046975.20 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:40:08,055 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428100
2023-11-24 13:40:12,048 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.442e+01 8.374e+01 8.915e+01 9.867e+01 1.264e+02, threshold=1.783e+02, percent-clipped=0.0
2023-11-24 13:40:29,460 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.77 vs. limit=15.0
2023-11-24 13:40:38,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2854153.3333333335, ans=0.2
2023-11-24 13:40:46,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2854153.3333333335, ans=0.125
2023-11-24 13:40:51,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2854220.0, ans=0.1
2023-11-24 13:40:51,997 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7300, loss[loss=0.06834, simple_loss=0.09846, pruned_loss=0.01251, audio_tagging_loss=0.006602, over 14834.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09116, pruned_loss=0.01296, audio_tagging_loss=0.008975, over 3048407.37 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:40:54,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2854220.0, ans=0.125
2023-11-24 13:41:10,678 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428150
2023-11-24 13:41:14,496 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:41:14,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2854286.6666666665, ans=0.1
2023-11-24 13:41:19,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2854353.3333333335, ans=15.0
2023-11-24 13:41:30,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2854420.0, ans=0.07
2023-11-24 13:41:38,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2854420.0, ans=0.0
2023-11-24 13:41:53,641 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7350, loss[loss=0.07129, simple_loss=0.09765, pruned_loss=0.01329, audio_tagging_loss=0.00918, over 16728.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09058, pruned_loss=0.0129, audio_tagging_loss=0.008942, over 3049123.00 frames. ], batch size: 63, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:41:58,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=2854553.3333333335, ans=0.5
2023-11-24 13:42:03,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2854553.3333333335, ans=0.0
2023-11-24 13:42:05,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2854620.0, ans=0.0
2023-11-24 13:42:09,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2854620.0, ans=0.2
2023-11-24 13:42:12,271 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428200
2023-11-24 13:42:15,979 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.426e+01 8.548e+01 9.171e+01 1.029e+02 1.460e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 13:42:33,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2854753.3333333335, ans=0.125
2023-11-24 13:42:35,408 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2854753.3333333335, ans=0.125
2023-11-24 13:42:35,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2854753.3333333335, ans=0.125
2023-11-24 13:42:48,916 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.87 vs. limit=15.0
2023-11-24 13:42:55,289 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7400, loss[loss=0.06365, simple_loss=0.08141, pruned_loss=0.01217, audio_tagging_loss=0.01077, over 15014.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09039, pruned_loss=0.01295, audio_tagging_loss=0.008812, over 3047898.42 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:43:07,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2854953.3333333335, ans=0.125
2023-11-24 13:43:14,792 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428250
2023-11-24 13:43:37,207 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.56 vs. limit=6.0
2023-11-24 13:43:49,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2855153.3333333335, ans=0.125
2023-11-24 13:43:58,295 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7450, loss[loss=0.07796, simple_loss=0.1077, pruned_loss=0.01462, audio_tagging_loss=0.009482, over 14094.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09014, pruned_loss=0.013, audio_tagging_loss=0.008735, over 3045478.34 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:44:15,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2855286.6666666665, ans=0.0
2023-11-24 13:44:17,769 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428300
2023-11-24 13:44:21,195 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.449e+01 8.702e+01 9.284e+01 1.003e+02 1.240e+02, threshold=1.857e+02, percent-clipped=0.0
2023-11-24 13:44:24,110 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.01 vs. limit=10.0
2023-11-24 13:44:39,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer_ff3.min_abs, batch_count=2855420.0, ans=0.2
2023-11-24 13:44:53,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2855486.6666666665, ans=0.1
2023-11-24 13:45:01,222 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7500, loss[loss=0.06577, simple_loss=0.08919, pruned_loss=0.009819, audio_tagging_loss=0.01135, over 16263.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.0909, pruned_loss=0.01301, audio_tagging_loss=0.008698, over 3049562.60 frames. ], batch size: 63, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:45:15,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2855620.0, ans=0.125
2023-11-24 13:45:19,277 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428350
2023-11-24 13:45:21,135 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.24 vs. limit=15.0
2023-11-24 13:45:38,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.10 vs. limit=22.5
2023-11-24 13:45:48,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2855753.3333333335, ans=0.0
2023-11-24 13:46:02,717 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7550, loss[loss=0.07338, simple_loss=0.09826, pruned_loss=0.01515, audio_tagging_loss=0.0091, over 16318.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09134, pruned_loss=0.01306, audio_tagging_loss=0.008641, over 3056393.49 frames. ], batch size: 61, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:46:20,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2855953.3333333335, ans=0.125
2023-11-24 13:46:21,633 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428400
2023-11-24 13:46:26,027 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.421e+01 8.555e+01 9.258e+01 9.894e+01 1.443e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 13:46:28,993 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.19 vs. limit=22.5
2023-11-24 13:46:39,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2856086.6666666665, ans=0.125
2023-11-24 13:46:47,437 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.94 vs. limit=10.0
2023-11-24 13:47:05,643 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7600, loss[loss=0.0487, simple_loss=0.05901, pruned_loss=0.008603, audio_tagging_loss=0.01059, over 15384.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09078, pruned_loss=0.01305, audio_tagging_loss=0.00867, over 3052283.04 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:47:05,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2856220.0, ans=0.0
2023-11-24 13:47:24,820 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428450
2023-11-24 13:47:39,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff3.min_abs, batch_count=2856353.3333333335, ans=0.2
2023-11-24 13:47:43,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2856420.0, ans=0.1
2023-11-24 13:47:54,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2856486.6666666665, ans=0.0
2023-11-24 13:48:04,167 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.83 vs. limit=15.0
2023-11-24 13:48:08,991 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7650, loss[loss=0.05867, simple_loss=0.07884, pruned_loss=0.01176, audio_tagging_loss=0.007487, over 15078.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09027, pruned_loss=0.01309, audio_tagging_loss=0.008753, over 3040557.64 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:48:18,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2856553.3333333335, ans=0.125
2023-11-24 13:48:27,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428500
2023-11-24 13:48:30,896 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.493e+01 9.050e+01 9.587e+01 1.815e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 13:48:34,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2856686.6666666665, ans=0.1
2023-11-24 13:48:36,799 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2856686.6666666665, ans=0.125
2023-11-24 13:48:46,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2856753.3333333335, ans=0.1
2023-11-24 13:48:48,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2856753.3333333335, ans=0.0
2023-11-24 13:49:08,359 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.98 vs. limit=15.0
2023-11-24 13:49:12,507 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7700, loss[loss=0.07261, simple_loss=0.09969, pruned_loss=0.01412, audio_tagging_loss=0.008641, over 14878.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09123, pruned_loss=0.01321, audio_tagging_loss=0.008688, over 3036927.22 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:49:31,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428550
2023-11-24 13:49:41,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2857020.0, ans=0.125
2023-11-24 13:49:58,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2857086.6666666665, ans=0.125
2023-11-24 13:50:09,614 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.97 vs. limit=10.0
2023-11-24 13:50:15,612 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7750, loss[loss=0.06224, simple_loss=0.08005, pruned_loss=0.0104, audio_tagging_loss=0.01182, over 14656.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09186, pruned_loss=0.01328, audio_tagging_loss=0.008612, over 3040534.69 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:50:28,031 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.58 vs. limit=15.0
2023-11-24 13:50:34,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428600
2023-11-24 13:50:38,547 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.286e+01 8.354e+01 9.227e+01 9.861e+01 1.458e+02, threshold=1.845e+02, percent-clipped=0.0
2023-11-24 13:51:17,636 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.02 vs. limit=15.0
2023-11-24 13:51:18,251 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7800, loss[loss=0.07232, simple_loss=0.09638, pruned_loss=0.01588, audio_tagging_loss=0.008251, over 15794.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09193, pruned_loss=0.01322, audio_tagging_loss=0.00865, over 3039382.93 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:51:29,128 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.59 vs. limit=12.0
2023-11-24 13:51:36,765 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428650
2023-11-24 13:51:49,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2857686.6666666665, ans=0.0
2023-11-24 13:51:51,027 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2857686.6666666665, ans=0.125
2023-11-24 13:52:09,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2857820.0, ans=0.125
2023-11-24 13:52:18,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2857820.0, ans=0.0
2023-11-24 13:52:20,577 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7850, loss[loss=0.07279, simple_loss=0.09841, pruned_loss=0.01446, audio_tagging_loss=0.00913, over 13538.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09231, pruned_loss=0.01323, audio_tagging_loss=0.008733, over 3041198.92 frames. ], batch size: 52, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:52:27,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2857886.6666666665, ans=0.125
2023-11-24 13:52:36,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2857953.3333333335, ans=0.125
2023-11-24 13:52:39,520 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428700
2023-11-24 13:52:43,087 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.656e+01 8.512e+01 9.149e+01 9.887e+01 1.408e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 13:52:51,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.63 vs. limit=10.0
2023-11-24 13:53:10,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff2.min_abs, batch_count=2858153.3333333335, ans=0.1
2023-11-24 13:53:23,077 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7900, loss[loss=0.08358, simple_loss=0.1179, pruned_loss=0.01503, audio_tagging_loss=0.009616, over 16040.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09142, pruned_loss=0.013, audio_tagging_loss=0.008891, over 3042166.24 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:53:40,380 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.39 vs. limit=10.0
2023-11-24 13:53:42,185 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428750
2023-11-24 13:53:56,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2858353.3333333335, ans=0.0
2023-11-24 13:54:01,424 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.82 vs. limit=22.5
2023-11-24 13:54:17,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2858486.6666666665, ans=0.1
2023-11-24 13:54:18,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2858486.6666666665, ans=0.125
2023-11-24 13:54:26,269 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 7950, loss[loss=0.05199, simple_loss=0.05668, pruned_loss=0.009003, audio_tagging_loss=0.01465, over 13989.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09109, pruned_loss=0.01302, audio_tagging_loss=0.009005, over 3042159.32 frames. ], batch size: 55, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:54:39,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2858620.0, ans=0.0
2023-11-24 13:54:41,142 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:54:43,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2858620.0, ans=0.125
2023-11-24 13:54:44,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428800
2023-11-24 13:54:46,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2858620.0, ans=0.125
2023-11-24 13:54:47,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2858620.0, ans=0.0
2023-11-24 13:54:49,816 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.921e+01 8.877e+01 9.547e+01 1.019e+02 1.290e+02, threshold=1.909e+02, percent-clipped=0.0
2023-11-24 13:54:59,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2858686.6666666665, ans=0.05
2023-11-24 13:55:28,618 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8000, loss[loss=0.06191, simple_loss=0.08573, pruned_loss=0.008105, audio_tagging_loss=0.01093, over 14678.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09046, pruned_loss=0.01289, audio_tagging_loss=0.009112, over 3035209.37 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 13:55:39,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2858953.3333333335, ans=0.0
2023-11-24 13:55:46,854 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428850
2023-11-24 13:55:48,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2858953.3333333335, ans=0.0
2023-11-24 13:55:52,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2859020.0, ans=0.0
2023-11-24 13:56:30,602 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8050, loss[loss=0.06816, simple_loss=0.08843, pruned_loss=0.01377, audio_tagging_loss=0.01018, over 14771.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.09038, pruned_loss=0.0129, audio_tagging_loss=0.009126, over 3039057.97 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:56:32,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2859220.0, ans=0.5
2023-11-24 13:56:44,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2859286.6666666665, ans=0.1
2023-11-24 13:56:49,532 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428900
2023-11-24 13:56:49,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2859286.6666666665, ans=0.2
2023-11-24 13:56:55,202 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.938e+01 8.620e+01 9.250e+01 9.827e+01 1.123e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 13:57:23,870 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.03 vs. limit=15.0
2023-11-24 13:57:24,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2859486.6666666665, ans=0.0
2023-11-24 13:57:24,632 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 13:57:32,554 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8100, loss[loss=0.06571, simple_loss=0.094, pruned_loss=0.01182, audio_tagging_loss=0.006882, over 15979.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09044, pruned_loss=0.01293, audio_tagging_loss=0.009087, over 3040070.55 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:57:45,457 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2859620.0, ans=0.2
2023-11-24 13:57:51,013 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 428950
2023-11-24 13:57:56,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2859686.6666666665, ans=0.025
2023-11-24 13:58:27,931 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.52 vs. limit=15.0
2023-11-24 13:58:34,219 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8150, loss[loss=0.07805, simple_loss=0.1056, pruned_loss=0.01717, audio_tagging_loss=0.008083, over 16339.00 frames. ], tot_loss[loss=0.06773, simple_loss=0.09153, pruned_loss=0.01306, audio_tagging_loss=0.008906, over 3041018.33 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:58:38,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2859886.6666666665, ans=0.0
2023-11-24 13:58:39,240 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2859886.6666666665, ans=0.0
2023-11-24 13:58:53,442 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429000
2023-11-24 13:58:54,228 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.82 vs. limit=15.0
2023-11-24 13:59:00,124 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.313e+01 8.661e+01 9.369e+01 1.006e+02 1.730e+02, threshold=1.874e+02, percent-clipped=0.0
2023-11-24 13:59:00,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2860020.0, ans=0.125
2023-11-24 13:59:14,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2860086.6666666665, ans=0.125
2023-11-24 13:59:16,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2860086.6666666665, ans=0.125
2023-11-24 13:59:25,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2860153.3333333335, ans=0.0
2023-11-24 13:59:36,946 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8200, loss[loss=0.0685, simple_loss=0.08186, pruned_loss=0.01684, audio_tagging_loss=0.01073, over 14492.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.0922, pruned_loss=0.0132, audio_tagging_loss=0.008799, over 3045984.77 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 13:59:37,016 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 13:59:55,737 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429050
2023-11-24 14:00:05,177 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.51 vs. limit=22.5
2023-11-24 14:00:10,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2860353.3333333335, ans=0.125
2023-11-24 14:00:11,725 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2860353.3333333335, ans=0.125
2023-11-24 14:00:39,592 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8250, loss[loss=0.05378, simple_loss=0.07621, pruned_loss=0.008488, audio_tagging_loss=0.007185, over 14090.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09117, pruned_loss=0.01307, audio_tagging_loss=0.008794, over 3040674.00 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:00:57,987 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429100
2023-11-24 14:01:02,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2860686.6666666665, ans=0.0
2023-11-24 14:01:03,755 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.868e+01 8.495e+01 9.085e+01 9.878e+01 1.172e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 14:01:09,319 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.82 vs. limit=15.0
2023-11-24 14:01:16,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.82 vs. limit=15.0
2023-11-24 14:01:20,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2860753.3333333335, ans=0.0
2023-11-24 14:01:21,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2860753.3333333335, ans=0.07
2023-11-24 14:01:23,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2860753.3333333335, ans=0.0
2023-11-24 14:01:37,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2860820.0, ans=0.1
2023-11-24 14:01:41,634 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8300, loss[loss=0.08951, simple_loss=0.1264, pruned_loss=0.02026, audio_tagging_loss=0.006046, over 15585.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09169, pruned_loss=0.01331, audio_tagging_loss=0.00868, over 3042456.40 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 14:01:56,502 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.01 vs. limit=6.0
2023-11-24 14:01:59,629 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429150
2023-11-24 14:02:13,506 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:02:23,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2861086.6666666665, ans=0.035
2023-11-24 14:02:35,454 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=17.56 vs. limit=22.5
2023-11-24 14:02:38,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2861153.3333333335, ans=0.125
2023-11-24 14:02:42,974 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8350, loss[loss=0.07065, simple_loss=0.0895, pruned_loss=0.01548, audio_tagging_loss=0.01042, over 15181.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09163, pruned_loss=0.01342, audio_tagging_loss=0.008637, over 3038501.05 frames. ], batch size: 58, lr: 1.88e-03, grad_scale: 8.0
2023-11-24 14:03:01,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=22.37 vs. limit=22.5
2023-11-24 14:03:02,454 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429200
2023-11-24 14:03:08,746 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.38 vs. limit=15.0
2023-11-24 14:03:10,587 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.055e+01 8.776e+01 9.274e+01 1.012e+02 1.289e+02, threshold=1.855e+02, percent-clipped=0.0
2023-11-24 14:03:18,722 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.83 vs. limit=5.0
2023-11-24 14:03:21,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2861420.0, ans=0.2
2023-11-24 14:03:22,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2861420.0, ans=0.2
2023-11-24 14:03:44,692 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0
2023-11-24 14:03:46,364 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8400, loss[loss=0.06522, simple_loss=0.08656, pruned_loss=0.01436, audio_tagging_loss=0.007577, over 14533.00 frames. ], tot_loss[loss=0.06787, simple_loss=0.09113, pruned_loss=0.01356, audio_tagging_loss=0.008744, over 3035657.23 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:03:52,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2861553.3333333335, ans=0.0
2023-11-24 14:03:54,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2861553.3333333335, ans=0.0
2023-11-24 14:04:01,646 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.68 vs. limit=15.0
2023-11-24 14:04:04,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429250
2023-11-24 14:04:11,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2861686.6666666665, ans=0.1
2023-11-24 14:04:12,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2861686.6666666665, ans=0.0
2023-11-24 14:04:21,683 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.68 vs. limit=15.0
2023-11-24 14:04:36,741 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2861820.0, ans=0.1
2023-11-24 14:04:47,759 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8450, loss[loss=0.07064, simple_loss=0.1012, pruned_loss=0.01255, audio_tagging_loss=0.007476, over 15015.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.0915, pruned_loss=0.01354, audio_tagging_loss=0.008921, over 3039995.99 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:05:05,646 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429300
2023-11-24 14:05:13,168 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.962e+01 8.551e+01 9.201e+01 9.862e+01 3.144e+02, threshold=1.840e+02, percent-clipped=1.0
2023-11-24 14:05:39,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2862153.3333333335, ans=0.125
2023-11-24 14:05:48,434 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8500, loss[loss=0.05943, simple_loss=0.06701, pruned_loss=0.01128, audio_tagging_loss=0.01464, over 15168.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09063, pruned_loss=0.01337, audio_tagging_loss=0.008999, over 3041410.41 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:06:00,499 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.70 vs. limit=15.0
2023-11-24 14:06:08,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429350
2023-11-24 14:06:14,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2862353.3333333335, ans=0.0
2023-11-24 14:06:20,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2862353.3333333335, ans=0.0
2023-11-24 14:06:39,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2862486.6666666665, ans=0.1
2023-11-24 14:06:47,254 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2862486.6666666665, ans=0.0
2023-11-24 14:06:51,115 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8550, loss[loss=0.08122, simple_loss=0.1087, pruned_loss=0.01829, audio_tagging_loss=0.00856, over 15862.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09064, pruned_loss=0.01331, audio_tagging_loss=0.008947, over 3043034.59 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:06:55,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2862553.3333333335, ans=0.0
2023-11-24 14:06:56,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2862553.3333333335, ans=0.125
2023-11-24 14:06:58,398 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.51 vs. limit=12.0
2023-11-24 14:07:00,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2862553.3333333335, ans=0.125
2023-11-24 14:07:10,182 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429400
2023-11-24 14:07:10,402 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2862620.0, ans=0.125
2023-11-24 14:07:17,339 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.305e+01 8.644e+01 9.213e+01 9.720e+01 1.244e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 14:07:23,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2862686.6666666665, ans=0.125
2023-11-24 14:07:25,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2862686.6666666665, ans=0.07
2023-11-24 14:07:54,011 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8600, loss[loss=0.07657, simple_loss=0.1046, pruned_loss=0.01449, audio_tagging_loss=0.009803, over 16359.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09088, pruned_loss=0.01309, audio_tagging_loss=0.008947, over 3046438.14 frames. ], batch size: 62, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:07:56,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2862886.6666666665, ans=0.125
2023-11-24 14:07:56,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2862886.6666666665, ans=0.0
2023-11-24 14:08:01,949 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.31 vs. limit=22.5
2023-11-24 14:08:03,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2862886.6666666665, ans=0.2
2023-11-24 14:08:12,067 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429450
2023-11-24 14:08:17,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2863020.0, ans=0.2
2023-11-24 14:08:21,632 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.25 vs. limit=22.5
2023-11-24 14:08:30,766 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.60 vs. limit=15.0
2023-11-24 14:08:50,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2863153.3333333335, ans=0.125
2023-11-24 14:08:55,753 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8650, loss[loss=0.07256, simple_loss=0.09773, pruned_loss=0.01317, audio_tagging_loss=0.01052, over 14205.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09144, pruned_loss=0.01309, audio_tagging_loss=0.008963, over 3050279.70 frames. ], batch size: 53, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:09:07,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2863286.6666666665, ans=0.125
2023-11-24 14:09:14,502 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429500
2023-11-24 14:09:22,000 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.258e+01 8.469e+01 9.080e+01 9.893e+01 1.324e+02, threshold=1.816e+02, percent-clipped=0.0
2023-11-24 14:09:37,105 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:09:41,918 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:09:53,433 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2863486.6666666665, ans=0.125
2023-11-24 14:09:56,216 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.62 vs. limit=22.5
2023-11-24 14:09:57,745 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8700, loss[loss=0.05064, simple_loss=0.06694, pruned_loss=0.008148, audio_tagging_loss=0.009022, over 15803.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09115, pruned_loss=0.01317, audio_tagging_loss=0.009033, over 3054978.51 frames. ], batch size: 60, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:10:17,286 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429550
2023-11-24 14:10:44,258 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=13.98 vs. limit=22.5
2023-11-24 14:10:53,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2863820.0, ans=0.1
2023-11-24 14:10:59,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2863886.6666666665, ans=0.0
2023-11-24 14:11:00,207 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8750, loss[loss=0.09753, simple_loss=0.1285, pruned_loss=0.02454, audio_tagging_loss=0.008722, over 15178.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09177, pruned_loss=0.0133, audio_tagging_loss=0.009101, over 3052658.10 frames. ], batch size: 54, lr: 1.88e-03, grad_scale: 16.0
2023-11-24 14:11:04,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2863886.6666666665, ans=0.125
2023-11-24 14:11:08,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2863886.6666666665, ans=0.09899494936611666
2023-11-24 14:11:17,929 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429600
2023-11-24 14:11:25,143 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.456e+01 8.680e+01 9.303e+01 1.028e+02 1.677e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-24 14:12:01,358 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=12.31 vs. limit=15.0
2023-11-24 14:12:02,034 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8800, loss[loss=0.07115, simple_loss=0.08912, pruned_loss=0.01627, audio_tagging_loss=0.01033, over 15681.00 frames. ], tot_loss[loss=0.0685, simple_loss=0.09209, pruned_loss=0.01332, audio_tagging_loss=0.009141, over 3047198.48 frames. ], batch size: 59, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 14:12:17,805 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.19 vs. limit=6.0
2023-11-24 14:12:21,452 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429650
2023-11-24 14:12:32,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2864353.3333333335, ans=0.04949747468305833
2023-11-24 14:12:53,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2864486.6666666665, ans=0.0
2023-11-24 14:12:54,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2864486.6666666665, ans=0.0
2023-11-24 14:13:04,078 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8850, loss[loss=0.06379, simple_loss=0.08856, pruned_loss=0.011, audio_tagging_loss=0.008503, over 15417.00 frames. ], tot_loss[loss=0.06807, simple_loss=0.09138, pruned_loss=0.01323, audio_tagging_loss=0.009144, over 3050764.37 frames. ], batch size: 57, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 14:13:08,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2864553.3333333335, ans=0.125
2023-11-24 14:13:08,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2864553.3333333335, ans=0.2
2023-11-24 14:13:08,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2864553.3333333335, ans=0.125
2023-11-24 14:13:15,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2864620.0, ans=0.125
2023-11-24 14:13:16,433 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:13:23,741 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429700
2023-11-24 14:13:30,787 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.004e+01 8.562e+01 9.103e+01 9.836e+01 1.222e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 14:13:32,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2864686.6666666665, ans=0.125
2023-11-24 14:13:58,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2864820.0, ans=0.1
2023-11-24 14:14:01,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2864820.0, ans=0.125
2023-11-24 14:14:01,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2864820.0, ans=0.125
2023-11-24 14:14:07,393 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8900, loss[loss=0.06596, simple_loss=0.08997, pruned_loss=0.0137, audio_tagging_loss=0.007268, over 15286.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.0918, pruned_loss=0.01328, audio_tagging_loss=0.008946, over 3048856.60 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 14:14:15,338 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:14:22,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2864953.3333333335, ans=0.125
2023-11-24 14:14:22,721 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.71 vs. limit=15.0
2023-11-24 14:14:25,785 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429750
2023-11-24 14:14:25,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2864953.3333333335, ans=0.1
2023-11-24 14:14:26,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2864953.3333333335, ans=10.0
2023-11-24 14:14:31,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2865020.0, ans=0.0
2023-11-24 14:14:41,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2865020.0, ans=0.125
2023-11-24 14:15:05,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2865153.3333333335, ans=0.125
2023-11-24 14:15:05,480 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.95 vs. limit=15.0
2023-11-24 14:15:09,583 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 8950, loss[loss=0.08246, simple_loss=0.121, pruned_loss=0.0143, audio_tagging_loss=0.007668, over 15221.00 frames. ], tot_loss[loss=0.0687, simple_loss=0.09296, pruned_loss=0.01345, audio_tagging_loss=0.008763, over 3048864.25 frames. ], batch size: 56, lr: 1.88e-03, grad_scale: 32.0
2023-11-24 14:15:27,803 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429800
2023-11-24 14:15:35,387 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.838e+01 8.550e+01 9.254e+01 9.924e+01 1.359e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 14:15:43,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2865353.3333333335, ans=0.125
2023-11-24 14:15:55,412 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.76 vs. limit=15.0
2023-11-24 14:15:57,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2865420.0, ans=0.125
2023-11-24 14:16:11,697 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9000, loss[loss=0.06298, simple_loss=0.08234, pruned_loss=0.0117, audio_tagging_loss=0.01011, over 15344.00 frames. ], tot_loss[loss=0.06912, simple_loss=0.09382, pruned_loss=0.01355, audio_tagging_loss=0.008655, over 3056917.51 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:16:11,700 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 14:16:39,092 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.9630, 3.8133, 4.8159, 4.4849], device='cuda:0')
2023-11-24 14:16:50,253 INFO [train_asr.py:1253] (0/4) Epoch 36, validation: loss=0.05864, simple_loss=0.05081, pruned_loss=0.005226, audio_tagging_loss=0.02801, over 4681554.00 frames. 
2023-11-24 14:16:50,254 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 14:17:03,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2865620.0, ans=0.0
2023-11-24 14:17:08,458 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429850
2023-11-24 14:17:15,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2865686.6666666665, ans=0.0
2023-11-24 14:17:19,303 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.17 vs. limit=15.0
2023-11-24 14:17:35,652 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.76 vs. limit=15.0
2023-11-24 14:17:43,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2865820.0, ans=0.2
2023-11-24 14:17:44,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2865820.0, ans=0.0
2023-11-24 14:17:52,340 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9050, loss[loss=0.06563, simple_loss=0.09114, pruned_loss=0.01257, audio_tagging_loss=0.00749, over 15592.00 frames. ], tot_loss[loss=0.06909, simple_loss=0.09391, pruned_loss=0.01355, audio_tagging_loss=0.008582, over 3055599.67 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:18:04,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2865953.3333333335, ans=0.125
2023-11-24 14:18:10,813 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429900
2023-11-24 14:18:13,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2865953.3333333335, ans=0.0
2023-11-24 14:18:19,340 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.179e+01 8.711e+01 9.316e+01 9.957e+01 1.805e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 14:18:38,909 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.68 vs. limit=10.0
2023-11-24 14:18:42,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2866153.3333333335, ans=10.0
2023-11-24 14:18:53,964 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9100, loss[loss=0.07609, simple_loss=0.1071, pruned_loss=0.01539, audio_tagging_loss=0.007151, over 14192.00 frames. ], tot_loss[loss=0.06878, simple_loss=0.09361, pruned_loss=0.01348, audio_tagging_loss=0.008495, over 3053443.69 frames. ], batch size: 53, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:19:07,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2866286.6666666665, ans=0.0
2023-11-24 14:19:13,310 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 429950
2023-11-24 14:19:18,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2866353.3333333335, ans=0.1
2023-11-24 14:19:20,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2866353.3333333335, ans=0.125
2023-11-24 14:19:22,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2866353.3333333335, ans=0.125
2023-11-24 14:19:24,811 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.39 vs. limit=22.5
2023-11-24 14:19:52,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2866486.6666666665, ans=0.0
2023-11-24 14:19:55,080 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2866553.3333333335, ans=0.04949747468305833
2023-11-24 14:19:56,570 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9150, loss[loss=0.06046, simple_loss=0.07842, pruned_loss=0.009844, audio_tagging_loss=0.01141, over 16581.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09269, pruned_loss=0.01331, audio_tagging_loss=0.008555, over 3055345.49 frames. ], batch size: 63, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:19:58,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2866553.3333333335, ans=0.025
2023-11-24 14:20:15,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430000
2023-11-24 14:20:18,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2866620.0, ans=0.0
2023-11-24 14:20:22,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2866686.6666666665, ans=0.0
2023-11-24 14:20:23,781 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.464e+01 8.685e+01 9.383e+01 1.043e+02 1.576e+02, threshold=1.877e+02, percent-clipped=0.0
2023-11-24 14:20:27,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2866686.6666666665, ans=0.125
2023-11-24 14:20:39,830 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.33 vs. limit=15.0
2023-11-24 14:20:59,767 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9200, loss[loss=0.08971, simple_loss=0.1319, pruned_loss=0.01674, audio_tagging_loss=0.007028, over 16449.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09162, pruned_loss=0.01322, audio_tagging_loss=0.008584, over 3065658.22 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:21:06,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2866886.6666666665, ans=0.125
2023-11-24 14:21:08,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2866886.6666666665, ans=0.2
2023-11-24 14:21:18,565 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430050
2023-11-24 14:21:26,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2867020.0, ans=0.125
2023-11-24 14:21:34,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2867020.0, ans=0.2
2023-11-24 14:21:35,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2867020.0, ans=0.1
2023-11-24 14:21:37,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2867086.6666666665, ans=0.1
2023-11-24 14:21:54,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2867153.3333333335, ans=0.2
2023-11-24 14:22:02,251 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9250, loss[loss=0.07758, simple_loss=0.1098, pruned_loss=0.01731, audio_tagging_loss=0.005379, over 14967.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09187, pruned_loss=0.01317, audio_tagging_loss=0.008593, over 3064287.17 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:22:02,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2867220.0, ans=0.1
2023-11-24 14:22:21,300 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430100
2023-11-24 14:22:29,122 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.50 vs. limit=22.5
2023-11-24 14:22:30,813 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.050e+01 8.538e+01 9.195e+01 9.828e+01 1.249e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 14:22:53,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2867486.6666666665, ans=0.0
2023-11-24 14:22:53,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2867486.6666666665, ans=0.125
2023-11-24 14:23:04,574 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9300, loss[loss=0.05898, simple_loss=0.08463, pruned_loss=0.009372, audio_tagging_loss=0.007297, over 14986.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.0913, pruned_loss=0.01293, audio_tagging_loss=0.008603, over 3063315.91 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:23:04,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2867553.3333333335, ans=0.125
2023-11-24 14:23:10,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=21.16 vs. limit=22.5
2023-11-24 14:23:13,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2867553.3333333335, ans=0.1
2023-11-24 14:23:22,973 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430150
2023-11-24 14:23:23,051 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2867620.0, ans=0.1
2023-11-24 14:23:30,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2867686.6666666665, ans=0.0
2023-11-24 14:24:06,050 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9350, loss[loss=0.06047, simple_loss=0.07633, pruned_loss=0.01151, audio_tagging_loss=0.0108, over 14812.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09145, pruned_loss=0.01304, audio_tagging_loss=0.00869, over 3059055.97 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:24:25,385 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430200
2023-11-24 14:24:35,739 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.521e+01 8.556e+01 8.988e+01 9.800e+01 1.410e+02, threshold=1.798e+02, percent-clipped=0.0
2023-11-24 14:24:36,454 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=6.03 vs. limit=6.0
2023-11-24 14:24:50,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2868086.6666666665, ans=0.2
2023-11-24 14:25:03,550 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.67 vs. limit=15.0
2023-11-24 14:25:07,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2868220.0, ans=0.125
2023-11-24 14:25:08,885 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9400, loss[loss=0.06104, simple_loss=0.07637, pruned_loss=0.01438, audio_tagging_loss=0.008479, over 14791.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09188, pruned_loss=0.01317, audio_tagging_loss=0.008867, over 3061715.19 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:25:09,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2868220.0, ans=0.2
2023-11-24 14:25:10,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2868220.0, ans=0.125
2023-11-24 14:25:21,195 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.93 vs. limit=15.0
2023-11-24 14:25:27,844 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430250
2023-11-24 14:25:46,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2868420.0, ans=0.0
2023-11-24 14:25:48,634 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:25:49,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2868420.0, ans=0.125
2023-11-24 14:25:52,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2868420.0, ans=0.125
2023-11-24 14:25:56,486 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2868420.0, ans=0.125
2023-11-24 14:26:03,276 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0
2023-11-24 14:26:10,695 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:26:11,856 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9450, loss[loss=0.06158, simple_loss=0.0785, pruned_loss=0.01278, audio_tagging_loss=0.009544, over 14460.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.0919, pruned_loss=0.01319, audio_tagging_loss=0.008913, over 3062180.34 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:26:22,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2868553.3333333335, ans=0.0
2023-11-24 14:26:30,261 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430300
2023-11-24 14:26:34,343 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.97 vs. limit=15.0
2023-11-24 14:26:37,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2868686.6666666665, ans=0.1
2023-11-24 14:26:39,331 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.666e+01 9.464e+01 1.022e+02 1.388e+02, threshold=1.893e+02, percent-clipped=0.0
2023-11-24 14:26:41,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2868686.6666666665, ans=0.125
2023-11-24 14:26:50,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2868753.3333333335, ans=0.1
2023-11-24 14:26:50,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2868753.3333333335, ans=0.2
2023-11-24 14:27:02,470 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.76 vs. limit=15.0
2023-11-24 14:27:13,075 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9500, loss[loss=0.05743, simple_loss=0.0757, pruned_loss=0.007856, audio_tagging_loss=0.01173, over 15327.00 frames. ], tot_loss[loss=0.06812, simple_loss=0.09181, pruned_loss=0.01323, audio_tagging_loss=0.008974, over 3056327.52 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:27:31,256 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430350
2023-11-24 14:28:14,253 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9550, loss[loss=0.07638, simple_loss=0.1054, pruned_loss=0.01423, audio_tagging_loss=0.009437, over 14868.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09151, pruned_loss=0.0132, audio_tagging_loss=0.009044, over 3049975.12 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:28:17,154 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.67 vs. limit=22.5
2023-11-24 14:28:29,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2869286.6666666665, ans=0.1
2023-11-24 14:28:33,382 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430400
2023-11-24 14:28:43,140 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.232e+01 8.513e+01 9.117e+01 9.826e+01 1.260e+02, threshold=1.823e+02, percent-clipped=0.0
2023-11-24 14:28:43,903 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.77 vs. limit=12.0
2023-11-24 14:29:14,277 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2869486.6666666665, ans=0.1
2023-11-24 14:29:16,897 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9600, loss[loss=0.06787, simple_loss=0.08838, pruned_loss=0.01249, audio_tagging_loss=0.01118, over 15318.00 frames. ], tot_loss[loss=0.06783, simple_loss=0.09115, pruned_loss=0.01302, audio_tagging_loss=0.009229, over 3042559.50 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:29:23,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2869553.3333333335, ans=0.125
2023-11-24 14:29:35,232 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430450
2023-11-24 14:30:19,692 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9650, loss[loss=0.07601, simple_loss=0.09853, pruned_loss=0.01718, audio_tagging_loss=0.009564, over 15653.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09111, pruned_loss=0.01306, audio_tagging_loss=0.009246, over 3043300.91 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:30:37,841 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430500
2023-11-24 14:30:38,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2869953.3333333335, ans=0.0
2023-11-24 14:30:49,802 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.279e+01 8.447e+01 9.267e+01 9.901e+01 1.317e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 14:30:55,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2870020.0, ans=0.0
2023-11-24 14:31:07,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2870086.6666666665, ans=0.04949747468305833
2023-11-24 14:31:08,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2870086.6666666665, ans=0.125
2023-11-24 14:31:10,037 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=11.90 vs. limit=12.0
2023-11-24 14:31:22,779 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9700, loss[loss=0.07893, simple_loss=0.1061, pruned_loss=0.01769, audio_tagging_loss=0.008191, over 16340.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09153, pruned_loss=0.01323, audio_tagging_loss=0.009045, over 3042843.90 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:31:42,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430550
2023-11-24 14:31:59,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2870353.3333333335, ans=0.07
2023-11-24 14:31:59,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2870353.3333333335, ans=0.0
2023-11-24 14:32:13,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2870486.6666666665, ans=0.1
2023-11-24 14:32:14,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2870486.6666666665, ans=0.2
2023-11-24 14:32:25,791 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9750, loss[loss=0.0619, simple_loss=0.0788, pruned_loss=0.009673, audio_tagging_loss=0.01283, over 14333.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09112, pruned_loss=0.01305, audio_tagging_loss=0.008894, over 3033703.84 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:32:42,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2870620.0, ans=0.1
2023-11-24 14:32:46,513 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430600
2023-11-24 14:32:53,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2870686.6666666665, ans=10.0
2023-11-24 14:32:54,487 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2870686.6666666665, ans=0.1
2023-11-24 14:32:57,818 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.545e+01 8.594e+01 9.328e+01 1.010e+02 1.236e+02, threshold=1.866e+02, percent-clipped=0.0
2023-11-24 14:33:24,551 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.80 vs. limit=15.0
2023-11-24 14:33:28,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2870820.0, ans=0.125
2023-11-24 14:33:31,832 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9800, loss[loss=0.06587, simple_loss=0.09235, pruned_loss=0.01186, audio_tagging_loss=0.007834, over 16454.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09103, pruned_loss=0.01302, audio_tagging_loss=0.008838, over 3033812.30 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:33:46,884 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2870953.3333333335, ans=0.1
2023-11-24 14:33:48,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2870953.3333333335, ans=0.2
2023-11-24 14:33:50,282 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430650
2023-11-24 14:33:57,452 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.46 vs. limit=6.0
2023-11-24 14:34:12,410 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.05 vs. limit=22.5
2023-11-24 14:34:15,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2871086.6666666665, ans=0.125
2023-11-24 14:34:28,079 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:34:30,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2871153.3333333335, ans=0.125
2023-11-24 14:34:32,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.25 vs. limit=15.0
2023-11-24 14:34:34,256 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9850, loss[loss=0.0728, simple_loss=0.0981, pruned_loss=0.01315, audio_tagging_loss=0.01059, over 15300.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09147, pruned_loss=0.01302, audio_tagging_loss=0.008808, over 3035539.94 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:34:47,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2871286.6666666665, ans=0.1
2023-11-24 14:34:53,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430700
2023-11-24 14:35:05,554 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.199e+01 8.366e+01 9.070e+01 9.726e+01 1.220e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 14:35:12,320 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.90 vs. limit=15.0
2023-11-24 14:35:13,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2871420.0, ans=0.125
2023-11-24 14:35:16,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2871420.0, ans=0.5
2023-11-24 14:35:24,349 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.57 vs. limit=15.0
2023-11-24 14:35:37,178 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9900, loss[loss=0.05223, simple_loss=0.06592, pruned_loss=0.008332, audio_tagging_loss=0.01094, over 15260.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09103, pruned_loss=0.01293, audio_tagging_loss=0.008753, over 3044897.71 frames. ], batch size: 60, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:35:43,943 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2871553.3333333335, ans=0.125
2023-11-24 14:35:48,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2871553.3333333335, ans=0.0
2023-11-24 14:35:56,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2871620.0, ans=0.1
2023-11-24 14:35:56,909 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430750
2023-11-24 14:36:00,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2871620.0, ans=0.1
2023-11-24 14:36:29,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2871820.0, ans=0.125
2023-11-24 14:36:35,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2871820.0, ans=0.2
2023-11-24 14:36:35,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2871820.0, ans=0.125
2023-11-24 14:36:40,994 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 9950, loss[loss=0.08484, simple_loss=0.1217, pruned_loss=0.0177, audio_tagging_loss=0.006276, over 15201.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09173, pruned_loss=0.01313, audio_tagging_loss=0.008784, over 3041579.72 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:36:50,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=2871886.6666666665, ans=0.2
2023-11-24 14:36:59,247 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430800
2023-11-24 14:37:01,594 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.31 vs. limit=22.5
2023-11-24 14:37:11,022 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.477e+01 9.166e+01 9.822e+01 1.194e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 14:37:11,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2872020.0, ans=0.125
2023-11-24 14:37:12,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2872020.0, ans=0.05
2023-11-24 14:37:16,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2872020.0, ans=0.0
2023-11-24 14:37:37,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.min_abs, batch_count=2872153.3333333335, ans=0.5
2023-11-24 14:37:41,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=2872153.3333333335, ans=0.02
2023-11-24 14:37:44,597 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10000, loss[loss=0.07708, simple_loss=0.1101, pruned_loss=0.01583, audio_tagging_loss=0.006189, over 16762.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09139, pruned_loss=0.01304, audio_tagging_loss=0.008784, over 3044861.17 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:37:50,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2872220.0, ans=0.1
2023-11-24 14:38:04,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430850
2023-11-24 14:38:43,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2872486.6666666665, ans=0.125
2023-11-24 14:38:47,931 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:38:50,179 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10050, loss[loss=0.05821, simple_loss=0.07234, pruned_loss=0.01401, audio_tagging_loss=0.008031, over 14393.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09042, pruned_loss=0.01287, audio_tagging_loss=0.00883, over 3047407.73 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:38:51,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2872553.3333333335, ans=0.125
2023-11-24 14:38:54,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2872553.3333333335, ans=0.1
2023-11-24 14:39:00,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2872553.3333333335, ans=0.125
2023-11-24 14:39:06,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2872620.0, ans=0.07
2023-11-24 14:39:09,670 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430900
2023-11-24 14:39:12,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2872620.0, ans=0.0
2023-11-24 14:39:20,969 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.464e+01 8.558e+01 9.038e+01 9.635e+01 1.199e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 14:39:43,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2872820.0, ans=0.125
2023-11-24 14:39:48,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2872820.0, ans=0.125
2023-11-24 14:39:53,170 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.88 vs. limit=15.0
2023-11-24 14:39:54,125 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10100, loss[loss=0.07142, simple_loss=0.09375, pruned_loss=0.01675, audio_tagging_loss=0.007791, over 14994.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09089, pruned_loss=0.01295, audio_tagging_loss=0.008899, over 3047273.50 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:39:55,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2872886.6666666665, ans=0.0
2023-11-24 14:40:13,339 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 430950
2023-11-24 14:40:20,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2873020.0, ans=0.5
2023-11-24 14:40:30,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2873020.0, ans=0.125
2023-11-24 14:40:46,853 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:40:58,564 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10150, loss[loss=0.07652, simple_loss=0.1008, pruned_loss=0.01783, audio_tagging_loss=0.008269, over 15754.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09018, pruned_loss=0.01287, audio_tagging_loss=0.00901, over 3048190.52 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:40:58,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2873220.0, ans=0.2
2023-11-24 14:41:17,707 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431000
2023-11-24 14:41:24,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2873353.3333333335, ans=0.125
2023-11-24 14:41:29,399 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.382e+01 8.738e+01 9.315e+01 1.037e+02 1.393e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 14:41:29,481 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:41:32,980 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.52 vs. limit=15.0
2023-11-24 14:42:01,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2873553.3333333335, ans=0.125
2023-11-24 14:42:03,169 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10200, loss[loss=0.0733, simple_loss=0.09547, pruned_loss=0.01306, audio_tagging_loss=0.01251, over 15909.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.08984, pruned_loss=0.01292, audio_tagging_loss=0.009027, over 3055180.43 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:42:03,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2873553.3333333335, ans=0.0
2023-11-24 14:42:18,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2873620.0, ans=0.125
2023-11-24 14:42:20,492 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.02 vs. limit=6.0
2023-11-24 14:42:22,845 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431050
2023-11-24 14:42:27,533 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:42:27,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2873686.6666666665, ans=0.0
2023-11-24 14:42:31,060 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2873686.6666666665, ans=0.0
2023-11-24 14:42:33,741 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.72 vs. limit=15.0
2023-11-24 14:42:43,795 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:42:56,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2873820.0, ans=0.125
2023-11-24 14:42:57,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2873820.0, ans=0.125
2023-11-24 14:43:04,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2873820.0, ans=0.125
2023-11-24 14:43:06,382 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10250, loss[loss=0.05839, simple_loss=0.08188, pruned_loss=0.007954, audio_tagging_loss=0.009497, over 14645.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09136, pruned_loss=0.0132, audio_tagging_loss=0.008928, over 3058908.04 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 14:43:09,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2873886.6666666665, ans=0.1
2023-11-24 14:43:11,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.55 vs. limit=15.0
2023-11-24 14:43:15,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2873886.6666666665, ans=0.0
2023-11-24 14:43:25,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431100
2023-11-24 14:43:27,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2873953.3333333335, ans=0.125
2023-11-24 14:43:29,432 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2873953.3333333335, ans=0.0
2023-11-24 14:43:35,888 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.16 vs. limit=6.0
2023-11-24 14:43:39,459 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.315e+01 8.593e+01 9.288e+01 9.901e+01 1.156e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-24 14:43:53,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2874086.6666666665, ans=0.125
2023-11-24 14:44:10,538 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10300, loss[loss=0.08277, simple_loss=0.09811, pruned_loss=0.02158, audio_tagging_loss=0.01214, over 16736.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09101, pruned_loss=0.01333, audio_tagging_loss=0.008988, over 3055645.83 frames. ], batch size: 63, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 14:44:29,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431150
2023-11-24 14:44:38,569 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.97 vs. limit=22.5
2023-11-24 14:44:44,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2874353.3333333335, ans=0.125
2023-11-24 14:45:05,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2874486.6666666665, ans=0.0
2023-11-24 14:45:11,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2874553.3333333335, ans=0.125
2023-11-24 14:45:12,591 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10350, loss[loss=0.06692, simple_loss=0.08691, pruned_loss=0.01309, audio_tagging_loss=0.01037, over 15085.00 frames. ], tot_loss[loss=0.0681, simple_loss=0.09154, pruned_loss=0.01336, audio_tagging_loss=0.008965, over 3047567.74 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 14:45:21,131 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=22.05 vs. limit=22.5
2023-11-24 14:45:32,542 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431200
2023-11-24 14:45:46,000 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.769e+01 8.642e+01 9.293e+01 9.950e+01 1.145e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 14:45:47,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2874686.6666666665, ans=0.2
2023-11-24 14:45:58,059 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2874753.3333333335, ans=0.125
2023-11-24 14:46:11,178 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.37 vs. limit=15.0
2023-11-24 14:46:12,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2874820.0, ans=0.0
2023-11-24 14:46:17,029 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10400, loss[loss=0.07484, simple_loss=0.1008, pruned_loss=0.015, audio_tagging_loss=0.009414, over 14757.00 frames. ], tot_loss[loss=0.06809, simple_loss=0.09145, pruned_loss=0.01331, audio_tagging_loss=0.009057, over 3048165.68 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:46:26,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2874886.6666666665, ans=0.0
2023-11-24 14:46:32,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2874953.3333333335, ans=0.0
2023-11-24 14:46:36,174 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431250
2023-11-24 14:46:37,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2874953.3333333335, ans=0.125
2023-11-24 14:46:44,652 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.56 vs. limit=15.0
2023-11-24 14:46:45,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2875020.0, ans=0.2
2023-11-24 14:46:48,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2875020.0, ans=0.0
2023-11-24 14:47:05,735 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.71 vs. limit=10.0
2023-11-24 14:47:07,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2875153.3333333335, ans=0.0
2023-11-24 14:47:07,955 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.57 vs. limit=15.0
2023-11-24 14:47:21,119 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10450, loss[loss=0.07162, simple_loss=0.09811, pruned_loss=0.01378, audio_tagging_loss=0.008787, over 15800.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09148, pruned_loss=0.01327, audio_tagging_loss=0.009036, over 3054060.28 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:47:40,001 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431300
2023-11-24 14:47:43,218 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2875286.6666666665, ans=0.0
2023-11-24 14:47:49,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.73 vs. limit=22.5
2023-11-24 14:47:54,050 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.123e+01 8.463e+01 9.315e+01 9.862e+01 1.314e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 14:48:24,016 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10500, loss[loss=0.0788, simple_loss=0.1042, pruned_loss=0.01749, audio_tagging_loss=0.009196, over 14378.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.0911, pruned_loss=0.01331, audio_tagging_loss=0.008928, over 3057757.99 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:48:26,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2875553.3333333335, ans=0.0
2023-11-24 14:48:40,161 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.41 vs. limit=10.0
2023-11-24 14:48:42,977 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431350
2023-11-24 14:48:45,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2875620.0, ans=0.125
2023-11-24 14:48:47,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2875620.0, ans=0.125
2023-11-24 14:48:58,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2875686.6666666665, ans=0.2
2023-11-24 14:49:26,978 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10550, loss[loss=0.05624, simple_loss=0.07282, pruned_loss=0.01243, audio_tagging_loss=0.007402, over 15235.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.08995, pruned_loss=0.01311, audio_tagging_loss=0.008856, over 3048918.25 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:49:40,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2875953.3333333335, ans=0.0
2023-11-24 14:49:45,339 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431400
2023-11-24 14:49:58,958 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.301e+01 8.652e+01 9.234e+01 9.954e+01 1.298e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 14:50:29,164 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10600, loss[loss=0.0699, simple_loss=0.09902, pruned_loss=0.01191, audio_tagging_loss=0.008482, over 15486.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.09013, pruned_loss=0.01297, audio_tagging_loss=0.008792, over 3041176.35 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:50:32,304 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.91 vs. limit=10.0
2023-11-24 14:50:34,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2876220.0, ans=0.125
2023-11-24 14:50:47,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2876286.6666666665, ans=0.0
2023-11-24 14:50:48,110 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431450
2023-11-24 14:50:56,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2876353.3333333335, ans=0.0
2023-11-24 14:51:31,310 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10650, loss[loss=0.07316, simple_loss=0.09747, pruned_loss=0.01556, audio_tagging_loss=0.008864, over 14865.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09076, pruned_loss=0.01318, audio_tagging_loss=0.008845, over 3044189.42 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:51:46,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2876620.0, ans=0.0
2023-11-24 14:51:48,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2876620.0, ans=0.125
2023-11-24 14:51:51,505 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431500
2023-11-24 14:52:04,979 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.032e+01 8.499e+01 9.165e+01 9.904e+01 1.171e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 14:52:06,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2876686.6666666665, ans=0.125
2023-11-24 14:52:11,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2876753.3333333335, ans=0.2
2023-11-24 14:52:11,957 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.11 vs. limit=6.0
2023-11-24 14:52:23,557 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2876820.0, ans=0.125
2023-11-24 14:52:36,257 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10700, loss[loss=0.05167, simple_loss=0.06218, pruned_loss=0.01061, audio_tagging_loss=0.009972, over 13914.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.0903, pruned_loss=0.01302, audio_tagging_loss=0.00885, over 3041952.70 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:52:37,992 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=11.90 vs. limit=15.0
2023-11-24 14:52:47,108 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 14:52:48,483 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2876953.3333333335, ans=0.025
2023-11-24 14:52:55,513 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431550
2023-11-24 14:53:39,955 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2877220.0, ans=0.125
2023-11-24 14:53:40,864 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10750, loss[loss=0.07933, simple_loss=0.1072, pruned_loss=0.01586, audio_tagging_loss=0.009868, over 15535.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09042, pruned_loss=0.01305, audio_tagging_loss=0.008806, over 3050007.17 frames. ], batch size: 59, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 14:53:52,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2877286.6666666665, ans=0.0
2023-11-24 14:53:59,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431600
2023-11-24 14:54:14,493 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.208e+01 8.628e+01 9.351e+01 9.677e+01 1.250e+02, threshold=1.870e+02, percent-clipped=0.0
2023-11-24 14:54:23,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2877420.0, ans=0.0
2023-11-24 14:54:28,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2877420.0, ans=0.125
2023-11-24 14:54:35,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2877486.6666666665, ans=0.125
2023-11-24 14:54:40,412 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2877486.6666666665, ans=0.1
2023-11-24 14:54:43,930 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10800, loss[loss=0.04503, simple_loss=0.06291, pruned_loss=0.00776, audio_tagging_loss=0.005812, over 15866.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09064, pruned_loss=0.01306, audio_tagging_loss=0.00868, over 3052384.76 frames. ], batch size: 62, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:54:46,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2877553.3333333335, ans=0.125
2023-11-24 14:54:57,744 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2877620.0, ans=0.0
2023-11-24 14:55:02,985 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431650
2023-11-24 14:55:05,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2877620.0, ans=0.125
2023-11-24 14:55:18,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.39 vs. limit=5.0
2023-11-24 14:55:19,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2877686.6666666665, ans=22.5
2023-11-24 14:55:36,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2877820.0, ans=0.0
2023-11-24 14:55:46,754 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10850, loss[loss=0.05147, simple_loss=0.07239, pruned_loss=0.007085, audio_tagging_loss=0.008194, over 15236.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09145, pruned_loss=0.01318, audio_tagging_loss=0.008605, over 3053245.75 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:56:00,418 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.42 vs. limit=15.0
2023-11-24 14:56:01,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2877953.3333333335, ans=0.125
2023-11-24 14:56:05,722 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431700
2023-11-24 14:56:12,267 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.09 vs. limit=15.0
2023-11-24 14:56:18,676 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.433e+01 8.559e+01 9.138e+01 9.969e+01 1.288e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 14:56:23,192 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.76 vs. limit=22.5
2023-11-24 14:56:23,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2878086.6666666665, ans=0.125
2023-11-24 14:56:45,140 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.13 vs. limit=15.0
2023-11-24 14:56:45,770 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:56:49,788 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10900, loss[loss=0.06863, simple_loss=0.09794, pruned_loss=0.01407, audio_tagging_loss=0.005592, over 17501.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09115, pruned_loss=0.01325, audio_tagging_loss=0.008695, over 3056851.66 frames. ], batch size: 67, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:56:57,655 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.27 vs. limit=15.0
2023-11-24 14:57:02,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2878286.6666666665, ans=0.125
2023-11-24 14:57:07,895 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431750
2023-11-24 14:57:10,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=2878286.6666666665, ans=0.025
2023-11-24 14:57:12,028 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.20 vs. limit=22.5
2023-11-24 14:57:23,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2878353.3333333335, ans=0.125
2023-11-24 14:57:27,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2878420.0, ans=0.125
2023-11-24 14:57:28,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2878420.0, ans=0.1
2023-11-24 14:57:33,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2878420.0, ans=0.125
2023-11-24 14:57:38,075 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.31 vs. limit=15.0
2023-11-24 14:57:51,583 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 10950, loss[loss=0.08687, simple_loss=0.1249, pruned_loss=0.01754, audio_tagging_loss=0.006853, over 15855.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09085, pruned_loss=0.01315, audio_tagging_loss=0.008781, over 3055429.72 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:58:10,009 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431800
2023-11-24 14:58:12,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2878620.0, ans=0.0
2023-11-24 14:58:24,305 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.107e+01 8.538e+01 9.236e+01 1.007e+02 1.326e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 14:58:30,661 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2878753.3333333335, ans=0.95
2023-11-24 14:58:36,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2878753.3333333335, ans=0.2
2023-11-24 14:58:42,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2878820.0, ans=0.2
2023-11-24 14:58:45,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2878820.0, ans=0.05
2023-11-24 14:58:53,764 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11000, loss[loss=0.0766, simple_loss=0.1023, pruned_loss=0.01806, audio_tagging_loss=0.007365, over 15248.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.08987, pruned_loss=0.01284, audio_tagging_loss=0.008887, over 3061205.95 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 14:59:05,659 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 14:59:13,478 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431850
2023-11-24 14:59:23,301 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.81 vs. limit=15.0
2023-11-24 14:59:36,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2879086.6666666665, ans=0.125
2023-11-24 14:59:39,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2879086.6666666665, ans=0.09899494936611666
2023-11-24 14:59:56,746 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11050, loss[loss=0.07331, simple_loss=0.09324, pruned_loss=0.01486, audio_tagging_loss=0.01183, over 15327.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.08922, pruned_loss=0.01287, audio_tagging_loss=0.009137, over 3054278.26 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:00:00,054 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:00:12,432 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.45 vs. limit=10.0
2023-11-24 15:00:15,361 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431900
2023-11-24 15:00:20,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2879353.3333333335, ans=0.125
2023-11-24 15:00:22,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2879353.3333333335, ans=0.0
2023-11-24 15:00:28,282 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.634e+01 9.324e+01 1.017e+02 1.244e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 15:00:37,173 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2879420.0, ans=0.125
2023-11-24 15:00:59,013 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11100, loss[loss=0.0684, simple_loss=0.08522, pruned_loss=0.0145, audio_tagging_loss=0.01129, over 14745.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09048, pruned_loss=0.01307, audio_tagging_loss=0.009181, over 3053220.60 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:00:59,573 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=15.44 vs. limit=22.5
2023-11-24 15:01:00,871 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.04 vs. limit=10.0
2023-11-24 15:01:03,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.17 vs. limit=15.0
2023-11-24 15:01:17,723 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 431950
2023-11-24 15:01:25,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2879686.6666666665, ans=0.125
2023-11-24 15:01:35,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2879686.6666666665, ans=0.07
2023-11-24 15:02:00,902 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11150, loss[loss=0.08969, simple_loss=0.1225, pruned_loss=0.0198, audio_tagging_loss=0.008629, over 14945.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09062, pruned_loss=0.01312, audio_tagging_loss=0.009278, over 3044534.75 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:02:09,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2879886.6666666665, ans=0.035
2023-11-24 15:02:11,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2879886.6666666665, ans=15.0
2023-11-24 15:02:18,911 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.09 vs. limit=6.0
2023-11-24 15:02:21,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432000
2023-11-24 15:02:22,818 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-432000.pt
2023-11-24 15:02:29,297 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2880020.0, ans=0.125
2023-11-24 15:02:37,216 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.937e+01 8.446e+01 9.218e+01 9.993e+01 1.214e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 15:03:01,128 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2880153.3333333335, ans=0.125
2023-11-24 15:03:07,433 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11200, loss[loss=0.06946, simple_loss=0.0953, pruned_loss=0.011, audio_tagging_loss=0.01081, over 15076.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09042, pruned_loss=0.01295, audio_tagging_loss=0.009251, over 3044599.83 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:03:17,855 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2880220.0, ans=0.0
2023-11-24 15:03:25,993 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432050
2023-11-24 15:03:29,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2880286.6666666665, ans=0.125
2023-11-24 15:03:45,302 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.02 vs. limit=15.0
2023-11-24 15:04:03,362 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.27 vs. limit=15.0
2023-11-24 15:04:09,708 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11250, loss[loss=0.05256, simple_loss=0.07667, pruned_loss=0.006431, audio_tagging_loss=0.00779, over 15666.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.08985, pruned_loss=0.01269, audio_tagging_loss=0.009252, over 3053254.71 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:04:27,475 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432100
2023-11-24 15:04:41,561 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.423e+01 8.402e+01 9.085e+01 9.896e+01 1.491e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 15:04:43,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2880686.6666666665, ans=0.0
2023-11-24 15:04:47,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2880753.3333333335, ans=0.0
2023-11-24 15:04:59,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2880820.0, ans=0.125
2023-11-24 15:04:59,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2880820.0, ans=0.025
2023-11-24 15:05:10,590 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11300, loss[loss=0.07433, simple_loss=0.09344, pruned_loss=0.01887, audio_tagging_loss=0.00874, over 14274.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09025, pruned_loss=0.01294, audio_tagging_loss=0.009104, over 3046281.08 frames. ], batch size: 54, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:05:20,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2880886.6666666665, ans=0.0
2023-11-24 15:05:22,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2880953.3333333335, ans=0.0
2023-11-24 15:05:23,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2880953.3333333335, ans=0.125
2023-11-24 15:05:29,027 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.54 vs. limit=15.0
2023-11-24 15:05:29,604 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432150
2023-11-24 15:05:33,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2880953.3333333335, ans=0.04949747468305833
2023-11-24 15:05:53,485 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2881086.6666666665, ans=0.0
2023-11-24 15:05:57,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2881086.6666666665, ans=0.125
2023-11-24 15:05:57,375 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.07 vs. limit=10.0
2023-11-24 15:06:07,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2881153.3333333335, ans=0.1
2023-11-24 15:06:11,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.59 vs. limit=15.0
2023-11-24 15:06:13,229 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11350, loss[loss=0.0614, simple_loss=0.08056, pruned_loss=0.01162, audio_tagging_loss=0.009493, over 16848.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09063, pruned_loss=0.01306, audio_tagging_loss=0.008954, over 3048959.37 frames. ], batch size: 66, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:06:13,713 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.46 vs. limit=15.0
2023-11-24 15:06:32,161 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432200
2023-11-24 15:06:32,691 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.69 vs. limit=15.0
2023-11-24 15:06:45,423 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.945e+01 8.608e+01 9.253e+01 9.930e+01 1.315e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 15:07:00,309 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.56 vs. limit=12.0
2023-11-24 15:07:01,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2881420.0, ans=0.0
2023-11-24 15:07:16,364 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11400, loss[loss=0.05753, simple_loss=0.08046, pruned_loss=0.01049, audio_tagging_loss=0.006816, over 14847.00 frames. ], tot_loss[loss=0.06788, simple_loss=0.09165, pruned_loss=0.01326, audio_tagging_loss=0.008802, over 3045842.04 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:07:16,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2881553.3333333335, ans=0.125
2023-11-24 15:07:24,180 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.65 vs. limit=10.0
2023-11-24 15:07:26,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2881553.3333333335, ans=0.09899494936611666
2023-11-24 15:07:27,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2881620.0, ans=0.2
2023-11-24 15:07:29,866 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=7.27 vs. limit=15.0
2023-11-24 15:07:31,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2881620.0, ans=0.0
2023-11-24 15:07:34,323 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432250
2023-11-24 15:07:52,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2881753.3333333335, ans=0.1
2023-11-24 15:07:56,766 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2881753.3333333335, ans=0.125
2023-11-24 15:08:01,823 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:08:01,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2881753.3333333335, ans=0.0
2023-11-24 15:08:15,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2881820.0, ans=0.07
2023-11-24 15:08:18,280 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11450, loss[loss=0.06801, simple_loss=0.08914, pruned_loss=0.01164, audio_tagging_loss=0.01179, over 15669.00 frames. ], tot_loss[loss=0.06814, simple_loss=0.09205, pruned_loss=0.01329, audio_tagging_loss=0.008818, over 3046764.17 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:08:19,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2881886.6666666665, ans=0.125
2023-11-24 15:08:27,838 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.59 vs. limit=22.5
2023-11-24 15:08:37,397 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432300
2023-11-24 15:08:47,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2882020.0, ans=0.125
2023-11-24 15:08:51,397 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.204e+01 8.606e+01 9.263e+01 9.916e+01 1.290e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 15:09:08,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2882153.3333333335, ans=0.125
2023-11-24 15:09:20,630 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11500, loss[loss=0.08767, simple_loss=0.1201, pruned_loss=0.02075, audio_tagging_loss=0.006893, over 15221.00 frames. ], tot_loss[loss=0.06829, simple_loss=0.09232, pruned_loss=0.01338, audio_tagging_loss=0.008753, over 3048497.86 frames. ], batch size: 56, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 15:09:39,332 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432350
2023-11-24 15:09:39,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2882286.6666666665, ans=0.125
2023-11-24 15:09:52,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2882353.3333333335, ans=0.125
2023-11-24 15:10:07,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2882420.0, ans=0.0
2023-11-24 15:10:10,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2882486.6666666665, ans=0.125
2023-11-24 15:10:11,318 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2882486.6666666665, ans=0.2
2023-11-24 15:10:21,968 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11550, loss[loss=0.06475, simple_loss=0.08671, pruned_loss=0.01124, audio_tagging_loss=0.01015, over 16025.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09109, pruned_loss=0.0132, audio_tagging_loss=0.008935, over 3052473.18 frames. ], batch size: 64, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 15:10:40,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432400
2023-11-24 15:10:46,737 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2882686.6666666665, ans=0.125
2023-11-24 15:10:55,840 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.162e+01 8.536e+01 9.084e+01 9.886e+01 1.170e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 15:11:01,159 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 15:11:03,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2882753.3333333335, ans=0.125
2023-11-24 15:11:07,639 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.78 vs. limit=15.0
2023-11-24 15:11:10,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2882820.0, ans=0.1
2023-11-24 15:11:20,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2882820.0, ans=0.0
2023-11-24 15:11:20,888 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2882820.0, ans=0.1
2023-11-24 15:11:24,129 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11600, loss[loss=0.06316, simple_loss=0.08196, pruned_loss=0.01261, audio_tagging_loss=0.009569, over 16071.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.0912, pruned_loss=0.0131, audio_tagging_loss=0.008858, over 3051915.44 frames. ], batch size: 61, lr: 1.87e-03, grad_scale: 32.0
2023-11-24 15:11:43,046 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432450
2023-11-24 15:11:44,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2882953.3333333335, ans=0.125
2023-11-24 15:11:54,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2883020.0, ans=0.5
2023-11-24 15:11:57,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2883020.0, ans=0.125
2023-11-24 15:11:57,178 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:12:00,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2883086.6666666665, ans=0.0
2023-11-24 15:12:02,813 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.83 vs. limit=6.0
2023-11-24 15:12:18,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2883153.3333333335, ans=0.2
2023-11-24 15:12:19,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2883153.3333333335, ans=15.0
2023-11-24 15:12:19,025 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.27 vs. limit=15.0
2023-11-24 15:12:26,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.42 vs. limit=12.0
2023-11-24 15:12:26,852 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11650, loss[loss=0.0635, simple_loss=0.09508, pruned_loss=0.007071, audio_tagging_loss=0.008883, over 14467.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09098, pruned_loss=0.0131, audio_tagging_loss=0.008928, over 3054257.58 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 15:12:28,684 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=13.57 vs. limit=15.0
2023-11-24 15:12:29,797 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.94 vs. limit=15.0
2023-11-24 15:12:45,589 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432500
2023-11-24 15:12:52,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2883353.3333333335, ans=0.0
2023-11-24 15:12:59,387 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2883353.3333333335, ans=0.125
2023-11-24 15:13:01,374 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.376e+01 8.463e+01 9.086e+01 9.673e+01 1.142e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 15:13:04,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2883420.0, ans=0.125
2023-11-24 15:13:06,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2883420.0, ans=0.2
2023-11-24 15:13:28,776 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11700, loss[loss=0.05763, simple_loss=0.07173, pruned_loss=0.01278, audio_tagging_loss=0.00899, over 14221.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.08986, pruned_loss=0.01283, audio_tagging_loss=0.008912, over 3053064.58 frames. ], batch size: 53, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:13:34,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2883553.3333333335, ans=0.125
2023-11-24 15:13:39,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.60 vs. limit=6.0
2023-11-24 15:13:47,045 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432550
2023-11-24 15:14:00,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2883686.6666666665, ans=0.04949747468305833
2023-11-24 15:14:11,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2883753.3333333335, ans=0.0
2023-11-24 15:14:14,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2883753.3333333335, ans=0.125
2023-11-24 15:14:15,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.01 vs. limit=10.0
2023-11-24 15:14:31,234 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11750, loss[loss=0.06458, simple_loss=0.08292, pruned_loss=0.01335, audio_tagging_loss=0.009772, over 15764.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09032, pruned_loss=0.01293, audio_tagging_loss=0.008941, over 3058542.77 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:14:41,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2883953.3333333335, ans=0.125
2023-11-24 15:14:43,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2883953.3333333335, ans=0.0
2023-11-24 15:14:50,164 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432600
2023-11-24 15:14:52,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2883953.3333333335, ans=0.1
2023-11-24 15:15:07,583 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.692e+01 8.388e+01 9.250e+01 9.905e+01 1.592e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 15:15:11,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2884086.6666666665, ans=0.0
2023-11-24 15:15:29,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2884153.3333333335, ans=0.05
2023-11-24 15:15:34,419 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11800, loss[loss=0.07449, simple_loss=0.1055, pruned_loss=0.01579, audio_tagging_loss=0.005966, over 15553.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09061, pruned_loss=0.01298, audio_tagging_loss=0.008917, over 3053891.76 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:15:43,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2884220.0, ans=0.0
2023-11-24 15:15:47,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2884286.6666666665, ans=0.125
2023-11-24 15:15:52,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432650
2023-11-24 15:15:53,383 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.01 vs. limit=22.5
2023-11-24 15:15:55,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=2884286.6666666665, ans=0.05
2023-11-24 15:15:55,722 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.77 vs. limit=15.0
2023-11-24 15:16:15,567 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.89 vs. limit=10.0
2023-11-24 15:16:21,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2884420.0, ans=0.09899494936611666
2023-11-24 15:16:36,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2884553.3333333335, ans=0.125
2023-11-24 15:16:36,946 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11850, loss[loss=0.04985, simple_loss=0.05385, pruned_loss=0.009353, audio_tagging_loss=0.01358, over 14069.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09127, pruned_loss=0.0132, audio_tagging_loss=0.008988, over 3046707.95 frames. ], batch size: 55, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:16:51,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2884620.0, ans=0.2
2023-11-24 15:16:54,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432700
2023-11-24 15:17:13,152 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.609e+01 8.479e+01 9.210e+01 9.675e+01 1.207e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 15:17:20,123 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.22 vs. limit=22.5
2023-11-24 15:17:38,048 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2884886.6666666665, ans=0.125
2023-11-24 15:17:38,785 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11900, loss[loss=0.05664, simple_loss=0.0709, pruned_loss=0.01051, audio_tagging_loss=0.01069, over 14646.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.0903, pruned_loss=0.01308, audio_tagging_loss=0.009043, over 3048962.67 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:17:58,157 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432750
2023-11-24 15:18:40,542 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 11950, loss[loss=0.06627, simple_loss=0.09442, pruned_loss=0.008007, audio_tagging_loss=0.01105, over 16092.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.0905, pruned_loss=0.0131, audio_tagging_loss=0.009152, over 3053998.42 frames. ], batch size: 58, lr: 1.87e-03, grad_scale: 8.0
2023-11-24 15:18:40,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2885220.0, ans=0.125
2023-11-24 15:19:00,170 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432800
2023-11-24 15:19:11,330 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:19:16,920 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 8.459e+01 8.985e+01 9.590e+01 1.349e+02, threshold=1.797e+02, percent-clipped=0.0
2023-11-24 15:19:35,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2885486.6666666665, ans=0.125
2023-11-24 15:19:41,564 INFO [train_asr.py:1221] (0/4) Epoch 36, batch 12000, loss[loss=0.0662, simple_loss=0.09406, pruned_loss=0.01032, audio_tagging_loss=0.008856, over 15746.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09098, pruned_loss=0.0131, audio_tagging_loss=0.009193, over 3051933.51 frames. ], batch size: 57, lr: 1.87e-03, grad_scale: 16.0
2023-11-24 15:19:41,567 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 15:20:23,128 INFO [train_asr.py:1253] (0/4) Epoch 36, validation: loss=0.05822, simple_loss=0.05085, pruned_loss=0.005219, audio_tagging_loss=0.02757, over 4681554.00 frames. 
2023-11-24 15:20:23,129 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 15:20:33,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2885620.0, ans=0.025
2023-11-24 15:20:37,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2885620.0, ans=0.125
2023-11-24 15:20:38,185 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.38 vs. limit=15.0
2023-11-24 15:20:39,985 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432850
2023-11-24 15:20:43,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2885620.0, ans=0.125
2023-11-24 15:20:52,515 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-36.pt
2023-11-24 15:21:26,947 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 0, loss[loss=0.08145, simple_loss=0.1035, pruned_loss=0.01247, audio_tagging_loss=0.01722, over 15297.00 frames. ], tot_loss[loss=0.08145, simple_loss=0.1035, pruned_loss=0.01247, audio_tagging_loss=0.01722, over 15297.00 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:21:26,950 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 15:22:03,072 INFO [train_asr.py:1253] (0/4) Epoch 37, validation: loss=0.05797, simple_loss=0.05085, pruned_loss=0.005252, audio_tagging_loss=0.02729, over 4681554.00 frames. 
2023-11-24 15:22:03,073 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 15:22:05,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2885720.0, ans=0.125
2023-11-24 15:22:08,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2885720.0, ans=0.0
2023-11-24 15:22:18,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2885786.6666666665, ans=0.0
2023-11-24 15:22:24,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2885786.6666666665, ans=0.125
2023-11-24 15:22:36,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2885853.3333333335, ans=0.0
2023-11-24 15:22:44,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2885920.0, ans=0.0
2023-11-24 15:22:53,050 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432900
2023-11-24 15:22:53,539 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=8.33 vs. limit=15.0
2023-11-24 15:23:01,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2885986.6666666665, ans=0.125
2023-11-24 15:23:06,062 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 50, loss[loss=0.07736, simple_loss=0.09766, pruned_loss=0.01584, audio_tagging_loss=0.01269, over 15494.00 frames. ], tot_loss[loss=0.07713, simple_loss=0.09319, pruned_loss=0.01354, audio_tagging_loss=0.017, over 693850.34 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:23:10,794 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.891e+01 9.190e+01 9.778e+01 1.072e+02 1.495e+02, threshold=1.956e+02, percent-clipped=0.0
2023-11-24 15:23:16,428 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2886053.3333333335, ans=0.95
2023-11-24 15:23:23,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.34 vs. limit=15.0
2023-11-24 15:23:32,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2886186.6666666665, ans=0.1
2023-11-24 15:23:39,219 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2886186.6666666665, ans=0.125
2023-11-24 15:23:45,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2886253.3333333335, ans=0.125
2023-11-24 15:23:51,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2886253.3333333335, ans=0.0
2023-11-24 15:23:55,637 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 432950
2023-11-24 15:23:56,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2886320.0, ans=0.2
2023-11-24 15:24:07,980 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 100, loss[loss=0.047, simple_loss=0.05247, pruned_loss=0.004232, audio_tagging_loss=0.01653, over 14953.00 frames. ], tot_loss[loss=0.07482, simple_loss=0.09159, pruned_loss=0.01277, audio_tagging_loss=0.01625, over 1214004.48 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:24:53,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2886586.6666666665, ans=0.125
2023-11-24 15:24:57,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433000
2023-11-24 15:25:04,681 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.52 vs. limit=15.0
2023-11-24 15:25:05,798 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.02 vs. limit=15.0
2023-11-24 15:25:09,905 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 150, loss[loss=0.07659, simple_loss=0.1085, pruned_loss=0.01149, audio_tagging_loss=0.01085, over 16607.00 frames. ], tot_loss[loss=0.07217, simple_loss=0.09013, pruned_loss=0.01252, audio_tagging_loss=0.01459, over 1629282.41 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:25:16,640 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.469e+01 8.996e+01 9.560e+01 1.019e+02 1.193e+02, threshold=1.912e+02, percent-clipped=0.0
2023-11-24 15:25:40,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2886853.3333333335, ans=0.0
2023-11-24 15:25:45,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2886853.3333333335, ans=0.2
2023-11-24 15:25:59,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433050
2023-11-24 15:26:12,776 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 200, loss[loss=0.07367, simple_loss=0.1046, pruned_loss=0.01368, audio_tagging_loss=0.007677, over 16338.00 frames. ], tot_loss[loss=0.0715, simple_loss=0.09149, pruned_loss=0.01292, audio_tagging_loss=0.01284, over 1947094.86 frames. ], batch size: 63, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:26:37,878 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.48 vs. limit=6.0
2023-11-24 15:26:45,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2887186.6666666665, ans=0.125
2023-11-24 15:27:02,946 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433100
2023-11-24 15:27:14,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2887386.6666666665, ans=0.0
2023-11-24 15:27:15,144 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 250, loss[loss=0.09076, simple_loss=0.124, pruned_loss=0.02169, audio_tagging_loss=0.007071, over 15608.00 frames. ], tot_loss[loss=0.07009, simple_loss=0.09085, pruned_loss=0.01291, audio_tagging_loss=0.01176, over 2194907.97 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:27:20,917 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.613e+01 8.812e+01 9.417e+01 1.032e+02 1.666e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-24 15:27:42,635 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2887520.0, ans=0.0
2023-11-24 15:27:50,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2887520.0, ans=0.0
2023-11-24 15:28:03,575 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:28:04,527 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433150
2023-11-24 15:28:16,220 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 300, loss[loss=0.07354, simple_loss=0.09722, pruned_loss=0.01708, audio_tagging_loss=0.007846, over 16055.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.09026, pruned_loss=0.01275, audio_tagging_loss=0.01092, over 2381844.85 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:28:25,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2887720.0, ans=0.05
2023-11-24 15:28:25,459 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2887720.0, ans=0.2
2023-11-24 15:28:32,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2887786.6666666665, ans=0.125
2023-11-24 15:28:35,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2887786.6666666665, ans=0.2
2023-11-24 15:28:36,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2887786.6666666665, ans=0.0
2023-11-24 15:29:03,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2887920.0, ans=0.125
2023-11-24 15:29:05,470 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433200
2023-11-24 15:29:09,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2887986.6666666665, ans=0.125
2023-11-24 15:29:17,663 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2888053.3333333335, ans=0.125
2023-11-24 15:29:18,710 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 350, loss[loss=0.06471, simple_loss=0.08375, pruned_loss=0.01369, audio_tagging_loss=0.00914, over 14910.00 frames. ], tot_loss[loss=0.06966, simple_loss=0.0925, pruned_loss=0.01317, audio_tagging_loss=0.01024, over 2532614.64 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:29:25,201 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.488e+01 8.531e+01 9.197e+01 9.925e+01 1.241e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 15:29:30,736 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.68 vs. limit=15.0
2023-11-24 15:29:36,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2888120.0, ans=0.0
2023-11-24 15:29:41,912 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.28 vs. limit=22.5
2023-11-24 15:29:52,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=2888186.6666666665, ans=15.0
2023-11-24 15:30:09,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433250
2023-11-24 15:30:12,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2888320.0, ans=0.0
2023-11-24 15:30:21,524 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 400, loss[loss=0.05869, simple_loss=0.07302, pruned_loss=0.01261, audio_tagging_loss=0.009564, over 14828.00 frames. ], tot_loss[loss=0.06876, simple_loss=0.09169, pruned_loss=0.013, audio_tagging_loss=0.009915, over 2655209.17 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:30:40,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2888453.3333333335, ans=0.1
2023-11-24 15:30:45,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2888520.0, ans=0.1
2023-11-24 15:30:52,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2888520.0, ans=0.125
2023-11-24 15:31:11,481 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433300
2023-11-24 15:31:12,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2888653.3333333335, ans=0.2
2023-11-24 15:31:12,987 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2888653.3333333335, ans=0.125
2023-11-24 15:31:13,368 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=14.61 vs. limit=15.0
2023-11-24 15:31:21,745 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.25 vs. limit=15.0
2023-11-24 15:31:23,319 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 450, loss[loss=0.07055, simple_loss=0.1047, pruned_loss=0.01055, audio_tagging_loss=0.007632, over 15786.00 frames. ], tot_loss[loss=0.06867, simple_loss=0.09184, pruned_loss=0.0131, audio_tagging_loss=0.009642, over 2742030.83 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:31:24,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2888720.0, ans=0.125
2023-11-24 15:31:30,273 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.008e+01 8.409e+01 9.030e+01 9.754e+01 1.188e+02, threshold=1.806e+02, percent-clipped=0.0
2023-11-24 15:31:37,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2888786.6666666665, ans=0.0
2023-11-24 15:31:52,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2888853.3333333335, ans=0.125
2023-11-24 15:32:12,966 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433350
2023-11-24 15:32:25,463 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 500, loss[loss=0.06196, simple_loss=0.0819, pruned_loss=0.01111, audio_tagging_loss=0.009894, over 15351.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.0909, pruned_loss=0.01304, audio_tagging_loss=0.009455, over 2810826.44 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:32:28,495 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2889053.3333333335, ans=0.125
2023-11-24 15:32:55,836 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2889186.6666666665, ans=0.1
2023-11-24 15:33:13,272 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2889253.3333333335, ans=0.0
2023-11-24 15:33:16,184 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433400
2023-11-24 15:33:16,699 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.65 vs. limit=15.0
2023-11-24 15:33:23,877 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2889320.0, ans=0.0
2023-11-24 15:33:28,902 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 550, loss[loss=0.05962, simple_loss=0.07666, pruned_loss=0.01334, audio_tagging_loss=0.007947, over 14505.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09117, pruned_loss=0.01309, audio_tagging_loss=0.00937, over 2866863.95 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:33:35,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2889386.6666666665, ans=0.1
2023-11-24 15:33:36,625 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.673e+01 8.403e+01 9.014e+01 9.757e+01 1.245e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 15:34:12,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2889586.6666666665, ans=0.125
2023-11-24 15:34:12,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2889586.6666666665, ans=0.125
2023-11-24 15:34:17,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2889653.3333333335, ans=0.125
2023-11-24 15:34:18,810 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433450
2023-11-24 15:34:30,540 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 600, loss[loss=0.06161, simple_loss=0.08143, pruned_loss=0.01177, audio_tagging_loss=0.009126, over 15974.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.0919, pruned_loss=0.01318, audio_tagging_loss=0.009228, over 2906021.66 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:34:40,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2889720.0, ans=0.1
2023-11-24 15:35:00,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2889853.3333333335, ans=0.125
2023-11-24 15:35:07,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2889920.0, ans=0.2
2023-11-24 15:35:09,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2889920.0, ans=0.04949747468305833
2023-11-24 15:35:17,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2889920.0, ans=10.0
2023-11-24 15:35:18,885 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.71 vs. limit=15.0
2023-11-24 15:35:20,523 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433500
2023-11-24 15:35:33,127 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 650, loss[loss=0.06442, simple_loss=0.08587, pruned_loss=0.01346, audio_tagging_loss=0.008021, over 15139.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09167, pruned_loss=0.01321, audio_tagging_loss=0.00927, over 2933326.02 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:35:40,749 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.769e+01 8.381e+01 9.134e+01 9.904e+01 1.320e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 15:36:05,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2890186.6666666665, ans=0.125
2023-11-24 15:36:23,154 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433550
2023-11-24 15:36:35,538 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 700, loss[loss=0.05791, simple_loss=0.07261, pruned_loss=0.0129, audio_tagging_loss=0.008701, over 14546.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.09175, pruned_loss=0.0131, audio_tagging_loss=0.00918, over 2962354.28 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:36:52,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2890453.3333333335, ans=0.125
2023-11-24 15:36:56,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2890453.3333333335, ans=0.125
2023-11-24 15:36:57,738 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.37 vs. limit=12.0
2023-11-24 15:37:06,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2890520.0, ans=0.0
2023-11-24 15:37:16,497 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2890586.6666666665, ans=0.125
2023-11-24 15:37:17,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2890586.6666666665, ans=0.1
2023-11-24 15:37:18,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2890586.6666666665, ans=0.1
2023-11-24 15:37:25,578 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433600
2023-11-24 15:37:25,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2890653.3333333335, ans=0.0
2023-11-24 15:37:38,610 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 750, loss[loss=0.08427, simple_loss=0.1119, pruned_loss=0.01793, audio_tagging_loss=0.01036, over 14841.00 frames. ], tot_loss[loss=0.0686, simple_loss=0.09254, pruned_loss=0.01324, audio_tagging_loss=0.00908, over 2987067.66 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:37:40,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2890720.0, ans=0.2
2023-11-24 15:37:44,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2890720.0, ans=0.125
2023-11-24 15:37:45,698 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.386e+01 8.622e+01 9.264e+01 1.020e+02 2.352e+02, threshold=1.853e+02, percent-clipped=1.0
2023-11-24 15:37:46,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2890720.0, ans=0.0
2023-11-24 15:37:49,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2890786.6666666665, ans=0.125
2023-11-24 15:38:28,495 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433650
2023-11-24 15:38:34,677 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.min_positive, batch_count=2890986.6666666665, ans=0.025
2023-11-24 15:38:40,569 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 800, loss[loss=0.04053, simple_loss=0.04896, pruned_loss=0.005966, audio_tagging_loss=0.01008, over 14440.00 frames. ], tot_loss[loss=0.06852, simple_loss=0.09233, pruned_loss=0.01322, audio_tagging_loss=0.009138, over 3004176.04 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:39:07,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2891186.6666666665, ans=0.0
2023-11-24 15:39:16,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2891186.6666666665, ans=0.1
2023-11-24 15:39:16,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2891186.6666666665, ans=0.1
2023-11-24 15:39:28,656 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.74 vs. limit=22.5
2023-11-24 15:39:31,224 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433700
2023-11-24 15:39:38,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2891320.0, ans=0.125
2023-11-24 15:39:43,402 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 850, loss[loss=0.05672, simple_loss=0.06109, pruned_loss=0.01573, audio_tagging_loss=0.01045, over 14378.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09182, pruned_loss=0.01323, audio_tagging_loss=0.009134, over 3018140.97 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:39:46,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2891386.6666666665, ans=0.0
2023-11-24 15:39:51,006 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.831e+01 8.494e+01 9.223e+01 9.765e+01 1.207e+02, threshold=1.845e+02, percent-clipped=0.0
2023-11-24 15:40:05,089 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:40:18,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2891520.0, ans=0.1
2023-11-24 15:40:26,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2891586.6666666665, ans=0.025
2023-11-24 15:40:27,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2891586.6666666665, ans=0.2
2023-11-24 15:40:32,711 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.55 vs. limit=22.5
2023-11-24 15:40:33,386 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433750
2023-11-24 15:40:40,029 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2891653.3333333335, ans=0.2
2023-11-24 15:40:45,662 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 900, loss[loss=0.06217, simple_loss=0.08546, pruned_loss=0.01138, audio_tagging_loss=0.008057, over 15179.00 frames. ], tot_loss[loss=0.06868, simple_loss=0.09243, pruned_loss=0.01326, audio_tagging_loss=0.009199, over 3027241.19 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:40:51,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2891720.0, ans=0.125
2023-11-24 15:40:53,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2891720.0, ans=0.0
2023-11-24 15:40:57,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2891786.6666666665, ans=0.125
2023-11-24 15:41:03,026 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2891786.6666666665, ans=0.1
2023-11-24 15:41:24,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2891920.0, ans=0.0
2023-11-24 15:41:35,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433800
2023-11-24 15:41:48,029 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 950, loss[loss=0.05681, simple_loss=0.07845, pruned_loss=0.0108, audio_tagging_loss=0.006786, over 15036.00 frames. ], tot_loss[loss=0.06853, simple_loss=0.09257, pruned_loss=0.0132, audio_tagging_loss=0.009039, over 3037248.64 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:41:51,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2892053.3333333335, ans=0.125
2023-11-24 15:41:57,417 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.340e+01 8.595e+01 9.194e+01 9.952e+01 1.132e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 15:42:06,390 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.30 vs. limit=22.5
2023-11-24 15:42:25,669 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.90 vs. limit=15.0
2023-11-24 15:42:32,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2892253.3333333335, ans=0.07
2023-11-24 15:42:38,336 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433850
2023-11-24 15:42:51,326 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1000, loss[loss=0.07144, simple_loss=0.08911, pruned_loss=0.01754, audio_tagging_loss=0.009337, over 15300.00 frames. ], tot_loss[loss=0.0688, simple_loss=0.093, pruned_loss=0.01339, audio_tagging_loss=0.008907, over 3034385.19 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:42:54,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2892386.6666666665, ans=0.0
2023-11-24 15:43:05,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2892453.3333333335, ans=0.2
2023-11-24 15:43:10,553 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.12 vs. limit=15.0
2023-11-24 15:43:17,447 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 15:43:34,251 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.24 vs. limit=15.0
2023-11-24 15:43:40,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433900
2023-11-24 15:43:52,919 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1050, loss[loss=0.08085, simple_loss=0.1142, pruned_loss=0.01665, audio_tagging_loss=0.007113, over 14457.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09175, pruned_loss=0.01323, audio_tagging_loss=0.00894, over 3029144.99 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:43:54,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2892720.0, ans=0.1
2023-11-24 15:43:54,431 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2892720.0, ans=0.2
2023-11-24 15:44:01,360 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.90 vs. limit=6.0
2023-11-24 15:44:01,784 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 8.504e+01 9.256e+01 1.007e+02 1.365e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 15:44:17,583 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.31 vs. limit=15.0
2023-11-24 15:44:28,747 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.16 vs. limit=12.0
2023-11-24 15:44:30,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2892920.0, ans=0.1
2023-11-24 15:44:42,852 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 433950
2023-11-24 15:44:55,284 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1100, loss[loss=0.06668, simple_loss=0.09388, pruned_loss=0.01115, audio_tagging_loss=0.008589, over 15444.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.0916, pruned_loss=0.0132, audio_tagging_loss=0.008849, over 3032035.05 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:44:57,732 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 15:44:59,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2893053.3333333335, ans=0.1
2023-11-24 15:45:00,619 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=12.39 vs. limit=22.5
2023-11-24 15:45:13,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2893120.0, ans=0.0
2023-11-24 15:45:27,055 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=15.0
2023-11-24 15:45:45,359 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434000
2023-11-24 15:45:58,263 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1150, loss[loss=0.06976, simple_loss=0.09832, pruned_loss=0.01089, audio_tagging_loss=0.009709, over 15207.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09186, pruned_loss=0.01324, audio_tagging_loss=0.008797, over 3042758.98 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:46:06,382 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.594e+01 8.608e+01 9.158e+01 9.709e+01 1.539e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 15:46:13,678 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.81 vs. limit=15.0
2023-11-24 15:46:43,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2893586.6666666665, ans=0.125
2023-11-24 15:46:48,194 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434050
2023-11-24 15:47:00,408 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1200, loss[loss=0.0622, simple_loss=0.08128, pruned_loss=0.01142, audio_tagging_loss=0.01014, over 16028.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09096, pruned_loss=0.01312, audio_tagging_loss=0.008794, over 3041652.87 frames. ], batch size: 62, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:47:02,441 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.19 vs. limit=6.0
2023-11-24 15:47:13,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2893786.6666666665, ans=0.0
2023-11-24 15:47:50,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434100
2023-11-24 15:48:01,998 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1250, loss[loss=0.0719, simple_loss=0.101, pruned_loss=0.01491, audio_tagging_loss=0.006487, over 16853.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09144, pruned_loss=0.01322, audio_tagging_loss=0.00866, over 3040150.48 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:48:11,440 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.976e+01 8.496e+01 8.998e+01 9.780e+01 2.107e+02, threshold=1.800e+02, percent-clipped=1.0
2023-11-24 15:48:21,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2894120.0, ans=0.0
2023-11-24 15:48:26,563 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:48:51,843 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434150
2023-11-24 15:49:00,619 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0
2023-11-24 15:49:05,244 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1300, loss[loss=0.06773, simple_loss=0.09256, pruned_loss=0.0115, audio_tagging_loss=0.009946, over 16294.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.0912, pruned_loss=0.01315, audio_tagging_loss=0.00871, over 3040309.87 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:49:11,780 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.14 vs. limit=15.0
2023-11-24 15:49:16,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2894453.3333333335, ans=0.125
2023-11-24 15:49:32,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2894520.0, ans=0.1
2023-11-24 15:49:35,042 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.16 vs. limit=8.0
2023-11-24 15:49:54,824 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434200
2023-11-24 15:49:55,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2894653.3333333335, ans=0.0
2023-11-24 15:50:07,494 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1350, loss[loss=0.05659, simple_loss=0.07397, pruned_loss=0.009058, audio_tagging_loss=0.01055, over 15425.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09062, pruned_loss=0.0128, audio_tagging_loss=0.008817, over 3043626.78 frames. ], batch size: 62, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:50:11,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2894720.0, ans=0.125
2023-11-24 15:50:15,721 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.023e+01 8.208e+01 8.839e+01 9.831e+01 1.172e+02, threshold=1.768e+02, percent-clipped=0.0
2023-11-24 15:50:19,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2894786.6666666665, ans=0.2
2023-11-24 15:50:24,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=2894786.6666666665, ans=10.0
2023-11-24 15:50:45,532 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2894920.0, ans=0.125
2023-11-24 15:50:51,224 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 15:50:51,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2894920.0, ans=0.1
2023-11-24 15:50:55,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2894986.6666666665, ans=0.2
2023-11-24 15:50:56,494 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434250
2023-11-24 15:50:59,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2894986.6666666665, ans=0.05
2023-11-24 15:51:06,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2894986.6666666665, ans=0.1
2023-11-24 15:51:08,231 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1400, loss[loss=0.07057, simple_loss=0.08579, pruned_loss=0.01656, audio_tagging_loss=0.01112, over 14305.00 frames. ], tot_loss[loss=0.06655, simple_loss=0.09004, pruned_loss=0.01266, audio_tagging_loss=0.008869, over 3047955.84 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:51:17,324 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.00 vs. limit=6.0
2023-11-24 15:51:32,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2895186.6666666665, ans=0.125
2023-11-24 15:51:33,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2895186.6666666665, ans=0.125
2023-11-24 15:51:34,243 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.14 vs. limit=10.0
2023-11-24 15:51:39,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2895186.6666666665, ans=0.0
2023-11-24 15:51:40,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2895186.6666666665, ans=0.0
2023-11-24 15:51:56,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=2895320.0, ans=0.125
2023-11-24 15:51:57,622 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434300
2023-11-24 15:51:58,409 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.14 vs. limit=12.0
2023-11-24 15:52:03,611 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2895320.0, ans=0.0
2023-11-24 15:52:10,340 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.31 vs. limit=8.0
2023-11-24 15:52:10,537 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1450, loss[loss=0.06899, simple_loss=0.09803, pruned_loss=0.009998, audio_tagging_loss=0.009977, over 14192.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.09018, pruned_loss=0.01281, audio_tagging_loss=0.00886, over 3044655.40 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:52:20,300 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.122e+01 8.571e+01 9.242e+01 1.000e+02 1.352e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 15:52:25,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2895453.3333333335, ans=0.125
2023-11-24 15:52:30,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2895453.3333333335, ans=0.2
2023-11-24 15:53:00,142 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434350
2023-11-24 15:53:05,040 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2895653.3333333335, ans=0.125
2023-11-24 15:53:06,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.36 vs. limit=15.0
2023-11-24 15:53:12,016 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1500, loss[loss=0.05617, simple_loss=0.06886, pruned_loss=0.01026, audio_tagging_loss=0.01148, over 14995.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09064, pruned_loss=0.0128, audio_tagging_loss=0.008968, over 3044434.49 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:53:13,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2895720.0, ans=0.0
2023-11-24 15:53:17,657 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2895720.0, ans=0.125
2023-11-24 15:53:26,182 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.67 vs. limit=15.0
2023-11-24 15:53:28,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2895786.6666666665, ans=0.125
2023-11-24 15:53:34,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2895786.6666666665, ans=0.2
2023-11-24 15:53:40,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=2895853.3333333335, ans=0.025
2023-11-24 15:53:41,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2895853.3333333335, ans=0.0
2023-11-24 15:53:49,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2895920.0, ans=0.2
2023-11-24 15:53:55,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2895920.0, ans=0.0
2023-11-24 15:54:01,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434400
2023-11-24 15:54:08,457 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:54:14,020 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1550, loss[loss=0.06355, simple_loss=0.08061, pruned_loss=0.0121, audio_tagging_loss=0.01114, over 13411.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09165, pruned_loss=0.01303, audio_tagging_loss=0.009047, over 3047644.75 frames. ], batch size: 54, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:54:17,037 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.99 vs. limit=15.0
2023-11-24 15:54:21,752 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.07 vs. limit=6.0
2023-11-24 15:54:23,505 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.499e+01 8.776e+01 9.350e+01 1.010e+02 1.250e+02, threshold=1.870e+02, percent-clipped=0.0
2023-11-24 15:54:43,258 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2896186.6666666665, ans=0.0
2023-11-24 15:54:53,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2896253.3333333335, ans=0.0
2023-11-24 15:55:03,749 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434450
2023-11-24 15:55:15,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2896386.6666666665, ans=0.125
2023-11-24 15:55:16,195 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1600, loss[loss=0.06896, simple_loss=0.09076, pruned_loss=0.01503, audio_tagging_loss=0.00855, over 15642.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09148, pruned_loss=0.01303, audio_tagging_loss=0.009049, over 3047664.63 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:55:58,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2896586.6666666665, ans=0.2
2023-11-24 15:55:59,472 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2896586.6666666665, ans=0.125
2023-11-24 15:56:05,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434500
2023-11-24 15:56:07,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2896653.3333333335, ans=0.1
2023-11-24 15:56:17,679 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=14.13 vs. limit=15.0
2023-11-24 15:56:18,322 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1650, loss[loss=0.05858, simple_loss=0.08126, pruned_loss=0.009167, audio_tagging_loss=0.008784, over 14722.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09114, pruned_loss=0.01288, audio_tagging_loss=0.009128, over 3047296.09 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 15:56:28,158 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.503e+01 8.575e+01 9.140e+01 1.003e+02 1.202e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 15:56:29,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2896786.6666666665, ans=0.125
2023-11-24 15:56:34,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2896786.6666666665, ans=0.125
2023-11-24 15:57:02,605 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.67 vs. limit=15.0
2023-11-24 15:57:08,130 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434550
2023-11-24 15:57:17,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2896986.6666666665, ans=0.1
2023-11-24 15:57:20,456 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1700, loss[loss=0.06188, simple_loss=0.07822, pruned_loss=0.01429, audio_tagging_loss=0.008482, over 13837.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09089, pruned_loss=0.01296, audio_tagging_loss=0.009206, over 3045614.44 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:57:38,761 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2897120.0, ans=0.1
2023-11-24 15:57:51,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2897186.6666666665, ans=0.0
2023-11-24 15:57:57,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2897253.3333333335, ans=0.125
2023-11-24 15:57:59,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2897253.3333333335, ans=0.125
2023-11-24 15:58:04,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2897253.3333333335, ans=0.125
2023-11-24 15:58:10,028 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434600
2023-11-24 15:58:12,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2897320.0, ans=0.0
2023-11-24 15:58:21,158 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 15:58:22,747 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1750, loss[loss=0.04889, simple_loss=0.0594, pruned_loss=0.004956, audio_tagging_loss=0.01423, over 14927.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09038, pruned_loss=0.01286, audio_tagging_loss=0.009195, over 3048489.63 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:58:22,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2897386.6666666665, ans=0.125
2023-11-24 15:58:33,877 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.758e+01 8.384e+01 9.014e+01 9.814e+01 1.863e+02, threshold=1.803e+02, percent-clipped=1.0
2023-11-24 15:58:34,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2897453.3333333335, ans=0.05
2023-11-24 15:58:38,352 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2897453.3333333335, ans=0.125
2023-11-24 15:58:45,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2897453.3333333335, ans=0.07
2023-11-24 15:58:56,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2897520.0, ans=0.125
2023-11-24 15:58:57,965 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2897520.0, ans=0.125
2023-11-24 15:59:01,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2897586.6666666665, ans=0.125
2023-11-24 15:59:04,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.54 vs. limit=15.0
2023-11-24 15:59:12,542 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434650
2023-11-24 15:59:20,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2897653.3333333335, ans=0.0
2023-11-24 15:59:24,849 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1800, loss[loss=0.05186, simple_loss=0.06789, pruned_loss=0.00919, audio_tagging_loss=0.00873, over 14842.00 frames. ], tot_loss[loss=0.06655, simple_loss=0.0897, pruned_loss=0.01268, audio_tagging_loss=0.009025, over 3046955.95 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 15:59:36,125 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.05 vs. limit=15.0
2023-11-24 15:59:58,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2897853.3333333335, ans=0.07
2023-11-24 16:00:04,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2897920.0, ans=0.125
2023-11-24 16:00:12,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2897920.0, ans=0.125
2023-11-24 16:00:15,188 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434700
2023-11-24 16:00:15,366 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2897986.6666666665, ans=10.0
2023-11-24 16:00:21,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2897986.6666666665, ans=0.07
2023-11-24 16:00:27,542 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1850, loss[loss=0.0858, simple_loss=0.1143, pruned_loss=0.02312, audio_tagging_loss=0.005514, over 14421.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09036, pruned_loss=0.01284, audio_tagging_loss=0.008989, over 3047159.92 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:00:38,265 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.639e+01 8.698e+01 9.278e+01 9.936e+01 1.415e+02, threshold=1.856e+02, percent-clipped=0.0
2023-11-24 16:00:42,508 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.65 vs. limit=12.0
2023-11-24 16:01:14,838 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.72 vs. limit=22.5
2023-11-24 16:01:17,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434750
2023-11-24 16:01:21,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2898320.0, ans=0.1
2023-11-24 16:01:25,406 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:01:29,994 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1900, loss[loss=0.06673, simple_loss=0.08974, pruned_loss=0.01204, audio_tagging_loss=0.009825, over 16235.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.0904, pruned_loss=0.01286, audio_tagging_loss=0.008856, over 3044074.80 frames. ], batch size: 62, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:01:36,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2898386.6666666665, ans=0.2
2023-11-24 16:02:20,234 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434800
2023-11-24 16:02:32,976 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 1950, loss[loss=0.05615, simple_loss=0.07785, pruned_loss=0.008653, audio_tagging_loss=0.008572, over 15595.00 frames. ], tot_loss[loss=0.0666, simple_loss=0.08971, pruned_loss=0.01279, audio_tagging_loss=0.008963, over 3047503.28 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:02:34,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2898720.0, ans=0.125
2023-11-24 16:02:35,931 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.50 vs. limit=6.0
2023-11-24 16:02:43,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2898720.0, ans=0.125
2023-11-24 16:02:44,687 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.590e+01 9.514e+01 1.026e+02 1.248e+02, threshold=1.903e+02, percent-clipped=0.0
2023-11-24 16:03:22,728 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434850
2023-11-24 16:03:35,537 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2000, loss[loss=0.04585, simple_loss=0.0584, pruned_loss=0.009217, audio_tagging_loss=0.007434, over 13953.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.08966, pruned_loss=0.01294, audio_tagging_loss=0.008932, over 3044631.35 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:03:47,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2899120.0, ans=0.125
2023-11-24 16:03:55,230 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:04:18,427 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.03 vs. limit=15.0
2023-11-24 16:04:21,121 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.74 vs. limit=15.0
2023-11-24 16:04:24,462 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:04:25,322 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434900
2023-11-24 16:04:29,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=2899320.0, ans=0.0
2023-11-24 16:04:36,760 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2050, loss[loss=0.07777, simple_loss=0.1059, pruned_loss=0.01742, audio_tagging_loss=0.0074, over 16021.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.08998, pruned_loss=0.013, audio_tagging_loss=0.008869, over 3045529.82 frames. ], batch size: 61, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:04:37,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2899386.6666666665, ans=0.0
2023-11-24 16:04:44,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2899386.6666666665, ans=0.125
2023-11-24 16:04:45,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2899386.6666666665, ans=0.07
2023-11-24 16:04:49,655 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.917e+01 8.636e+01 9.159e+01 9.953e+01 1.239e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 16:05:13,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2899520.0, ans=0.2
2023-11-24 16:05:17,855 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.40 vs. limit=15.0
2023-11-24 16:05:23,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2899586.6666666665, ans=0.125
2023-11-24 16:05:26,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2899653.3333333335, ans=0.09899494936611666
2023-11-24 16:05:27,065 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 434950
2023-11-24 16:05:39,840 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2100, loss[loss=0.05608, simple_loss=0.06941, pruned_loss=0.01111, audio_tagging_loss=0.01026, over 15042.00 frames. ], tot_loss[loss=0.06663, simple_loss=0.08992, pruned_loss=0.01283, audio_tagging_loss=0.008841, over 3052219.81 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:05:40,471 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.04 vs. limit=15.0
2023-11-24 16:05:47,137 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2899720.0, ans=0.0
2023-11-24 16:05:53,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2899786.6666666665, ans=0.0
2023-11-24 16:06:23,352 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:06:28,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2899986.6666666665, ans=0.0
2023-11-24 16:06:29,657 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435000
2023-11-24 16:06:40,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2899986.6666666665, ans=0.125
2023-11-24 16:06:42,426 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2150, loss[loss=0.06309, simple_loss=0.07652, pruned_loss=0.01254, audio_tagging_loss=0.0123, over 14974.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.09058, pruned_loss=0.01281, audio_tagging_loss=0.008751, over 3049954.79 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:06:46,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2900053.3333333335, ans=0.125
2023-11-24 16:06:50,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2900053.3333333335, ans=0.1
2023-11-24 16:06:54,769 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.082e+01 8.565e+01 9.008e+01 9.643e+01 1.315e+02, threshold=1.802e+02, percent-clipped=0.0
2023-11-24 16:06:55,033 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2900120.0, ans=0.1
2023-11-24 16:07:05,526 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.23 vs. limit=15.0
2023-11-24 16:07:19,839 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:07:32,307 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435050
2023-11-24 16:07:44,816 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2200, loss[loss=0.05747, simple_loss=0.07334, pruned_loss=0.01102, audio_tagging_loss=0.00978, over 15959.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09061, pruned_loss=0.01285, audio_tagging_loss=0.008878, over 3048716.36 frames. ], batch size: 60, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:07:48,937 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.42 vs. limit=12.0
2023-11-24 16:08:00,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2900453.3333333335, ans=0.0
2023-11-24 16:08:07,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2900453.3333333335, ans=0.0
2023-11-24 16:08:09,456 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.00 vs. limit=5.0
2023-11-24 16:08:22,810 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.32 vs. limit=6.0
2023-11-24 16:08:34,466 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435100
2023-11-24 16:08:34,652 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2900653.3333333335, ans=0.0
2023-11-24 16:08:43,494 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2900653.3333333335, ans=0.125
2023-11-24 16:08:47,288 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2250, loss[loss=0.06172, simple_loss=0.08321, pruned_loss=0.01209, audio_tagging_loss=0.008022, over 14632.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09116, pruned_loss=0.0131, audio_tagging_loss=0.008935, over 3045813.26 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:08:47,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2900720.0, ans=0.09899494936611666
2023-11-24 16:08:58,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2900786.6666666665, ans=0.125
2023-11-24 16:08:59,271 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.499e+01 9.314e+01 1.008e+02 1.259e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 16:09:14,007 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2900853.3333333335, ans=0.0
2023-11-24 16:09:23,670 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2900920.0, ans=0.1
2023-11-24 16:09:37,551 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435150
2023-11-24 16:09:38,264 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.68 vs. limit=10.0
2023-11-24 16:09:49,111 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2300, loss[loss=0.0578, simple_loss=0.07121, pruned_loss=0.01177, audio_tagging_loss=0.01042, over 15185.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.08964, pruned_loss=0.01287, audio_tagging_loss=0.009074, over 3041312.41 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:10:05,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2901120.0, ans=0.125
2023-11-24 16:10:39,089 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435200
2023-11-24 16:10:44,195 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:10:51,727 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2350, loss[loss=0.0828, simple_loss=0.1075, pruned_loss=0.01921, audio_tagging_loss=0.009827, over 15460.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09017, pruned_loss=0.01278, audio_tagging_loss=0.009103, over 3037313.33 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:10:54,721 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.51 vs. limit=15.0
2023-11-24 16:11:04,101 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.021e+01 8.491e+01 9.098e+01 9.735e+01 1.165e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 16:11:05,936 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.71 vs. limit=22.5
2023-11-24 16:11:15,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2901520.0, ans=0.125
2023-11-24 16:11:16,714 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2901520.0, ans=0.125
2023-11-24 16:11:22,788 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2901520.0, ans=0.1
2023-11-24 16:11:29,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2901586.6666666665, ans=0.1
2023-11-24 16:11:41,208 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435250
2023-11-24 16:11:44,300 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.93 vs. limit=15.0
2023-11-24 16:11:44,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2901653.3333333335, ans=0.125
2023-11-24 16:11:44,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2901653.3333333335, ans=0.125
2023-11-24 16:11:50,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2901653.3333333335, ans=0.0
2023-11-24 16:11:51,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.min_positive, batch_count=2901653.3333333335, ans=0.025
2023-11-24 16:11:52,535 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2901720.0, ans=0.2
2023-11-24 16:11:53,575 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2400, loss[loss=0.0709, simple_loss=0.09564, pruned_loss=0.01595, audio_tagging_loss=0.007125, over 16245.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09055, pruned_loss=0.0128, audio_tagging_loss=0.009129, over 3046701.85 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:12:13,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2901786.6666666665, ans=0.125
2023-11-24 16:12:20,166 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.02 vs. limit=15.0
2023-11-24 16:12:32,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2901920.0, ans=0.0
2023-11-24 16:12:44,419 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435300
2023-11-24 16:12:56,205 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2450, loss[loss=0.07812, simple_loss=0.1133, pruned_loss=0.01459, audio_tagging_loss=0.00688, over 16287.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09052, pruned_loss=0.01281, audio_tagging_loss=0.009159, over 3043413.75 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:13:09,834 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.445e+01 8.651e+01 9.072e+01 9.865e+01 1.248e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 16:13:19,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2902120.0, ans=0.2
2023-11-24 16:13:46,184 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435350
2023-11-24 16:13:58,429 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2500, loss[loss=0.06019, simple_loss=0.078, pruned_loss=0.01154, audio_tagging_loss=0.009643, over 15035.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.0909, pruned_loss=0.01279, audio_tagging_loss=0.009129, over 3044277.74 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:13:59,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=2902386.6666666665, ans=0.0
2023-11-24 16:14:08,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2902386.6666666665, ans=0.0
2023-11-24 16:14:19,865 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.25 vs. limit=10.0
2023-11-24 16:14:19,959 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.52 vs. limit=22.5
2023-11-24 16:14:25,466 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2902520.0, ans=0.125
2023-11-24 16:14:42,862 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2902586.6666666665, ans=15.0
2023-11-24 16:14:48,498 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435400
2023-11-24 16:15:01,963 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2550, loss[loss=0.05375, simple_loss=0.07057, pruned_loss=0.009247, audio_tagging_loss=0.009223, over 14443.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09153, pruned_loss=0.01295, audio_tagging_loss=0.008994, over 3037278.87 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:15:15,544 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.436e+01 8.712e+01 9.254e+01 9.868e+01 2.546e+02, threshold=1.851e+02, percent-clipped=1.0
2023-11-24 16:15:51,505 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435450
2023-11-24 16:15:54,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2902986.6666666665, ans=0.2
2023-11-24 16:16:03,757 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2600, loss[loss=0.06991, simple_loss=0.09682, pruned_loss=0.0153, audio_tagging_loss=0.006204, over 15346.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09099, pruned_loss=0.01287, audio_tagging_loss=0.008892, over 3040563.38 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:16:10,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=2903053.3333333335, ans=0.05
2023-11-24 16:16:11,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2903053.3333333335, ans=0.0
2023-11-24 16:16:12,587 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=2903053.3333333335, ans=0.125
2023-11-24 16:16:38,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2903186.6666666665, ans=0.125
2023-11-24 16:16:49,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2903253.3333333335, ans=0.125
2023-11-24 16:16:50,222 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.00 vs. limit=15.0
2023-11-24 16:16:53,233 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435500
2023-11-24 16:17:05,542 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2650, loss[loss=0.07056, simple_loss=0.09552, pruned_loss=0.01384, audio_tagging_loss=0.00895, over 14880.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.09092, pruned_loss=0.01286, audio_tagging_loss=0.008887, over 3044423.79 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:17:07,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2903386.6666666665, ans=0.125
2023-11-24 16:17:18,504 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.228e+01 8.439e+01 8.953e+01 1.003e+02 3.059e+02, threshold=1.791e+02, percent-clipped=1.0
2023-11-24 16:17:54,381 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435550
2023-11-24 16:17:56,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2903653.3333333335, ans=0.0
2023-11-24 16:18:06,104 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2700, loss[loss=0.07263, simple_loss=0.09417, pruned_loss=0.0126, audio_tagging_loss=0.01294, over 14014.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09047, pruned_loss=0.01287, audio_tagging_loss=0.008871, over 3038757.52 frames. ], batch size: 53, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:18:18,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2903786.6666666665, ans=0.125
2023-11-24 16:18:56,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435600
2023-11-24 16:19:09,900 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2750, loss[loss=0.0683, simple_loss=0.09071, pruned_loss=0.01487, audio_tagging_loss=0.008074, over 16411.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09035, pruned_loss=0.01283, audio_tagging_loss=0.008908, over 3041819.70 frames. ], batch size: 62, lr: 1.84e-03, grad_scale: 8.0
2023-11-24 16:19:12,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2904053.3333333335, ans=0.0
2023-11-24 16:19:24,583 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.485e+01 8.630e+01 9.216e+01 9.892e+01 1.188e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 16:19:28,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2904120.0, ans=0.0
2023-11-24 16:19:31,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2904120.0, ans=0.1
2023-11-24 16:19:34,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2904186.6666666665, ans=0.125
2023-11-24 16:19:50,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2904253.3333333335, ans=0.125
2023-11-24 16:19:59,730 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435650
2023-11-24 16:20:02,118 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:20:02,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2904320.0, ans=0.0
2023-11-24 16:20:10,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2904320.0, ans=0.125
2023-11-24 16:20:12,157 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2800, loss[loss=0.0637, simple_loss=0.08835, pruned_loss=0.01249, audio_tagging_loss=0.007037, over 13689.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09089, pruned_loss=0.01288, audio_tagging_loss=0.008789, over 3040856.16 frames. ], batch size: 52, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:20:16,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2904386.6666666665, ans=0.025
2023-11-24 16:20:23,204 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2904453.3333333335, ans=0.0
2023-11-24 16:20:34,449 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2904453.3333333335, ans=0.125
2023-11-24 16:20:54,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2904586.6666666665, ans=0.125
2023-11-24 16:21:00,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2904653.3333333335, ans=0.025
2023-11-24 16:21:01,508 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435700
2023-11-24 16:21:13,369 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2850, loss[loss=0.05511, simple_loss=0.07617, pruned_loss=0.006636, audio_tagging_loss=0.01039, over 15301.00 frames. ], tot_loss[loss=0.06657, simple_loss=0.09005, pruned_loss=0.01272, audio_tagging_loss=0.008823, over 3041705.18 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:21:19,118 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:21:29,328 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.073e+01 8.333e+01 9.111e+01 9.785e+01 2.308e+02, threshold=1.822e+02, percent-clipped=1.0
2023-11-24 16:21:35,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2904786.6666666665, ans=0.125
2023-11-24 16:21:38,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2904853.3333333335, ans=0.0
2023-11-24 16:21:43,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2904853.3333333335, ans=0.015
2023-11-24 16:21:49,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2904853.3333333335, ans=0.1
2023-11-24 16:21:49,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2904853.3333333335, ans=0.0
2023-11-24 16:21:54,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2904920.0, ans=0.125
2023-11-24 16:21:58,573 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2904920.0, ans=0.1
2023-11-24 16:22:03,263 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435750
2023-11-24 16:22:16,948 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2900, loss[loss=0.06219, simple_loss=0.08662, pruned_loss=0.01327, audio_tagging_loss=0.005609, over 14685.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.0902, pruned_loss=0.01291, audio_tagging_loss=0.008852, over 3039538.68 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:22:33,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2905120.0, ans=0.0
2023-11-24 16:22:43,110 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.35 vs. limit=15.0
2023-11-24 16:22:47,654 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.22 vs. limit=12.0
2023-11-24 16:22:59,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2905253.3333333335, ans=0.125
2023-11-24 16:22:59,460 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=17.89 vs. limit=22.5
2023-11-24 16:23:06,476 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435800
2023-11-24 16:23:08,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2905320.0, ans=0.125
2023-11-24 16:23:11,684 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=2905320.0, ans=0.05
2023-11-24 16:23:18,545 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=7.29 vs. limit=15.0
2023-11-24 16:23:19,024 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 2950, loss[loss=0.0584, simple_loss=0.07496, pruned_loss=0.01071, audio_tagging_loss=0.01022, over 14938.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09087, pruned_loss=0.01301, audio_tagging_loss=0.008895, over 3045867.87 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:23:33,479 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.071e+01 8.426e+01 9.166e+01 9.683e+01 1.219e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 16:23:38,059 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.88 vs. limit=15.0
2023-11-24 16:23:40,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2905453.3333333335, ans=0.125
2023-11-24 16:23:45,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=2905520.0, ans=0.05
2023-11-24 16:23:47,277 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.66 vs. limit=15.0
2023-11-24 16:23:53,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2905520.0, ans=0.2
2023-11-24 16:24:02,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2905586.6666666665, ans=0.125
2023-11-24 16:24:06,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2905586.6666666665, ans=0.0
2023-11-24 16:24:09,128 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435850
2023-11-24 16:24:15,429 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.86 vs. limit=15.0
2023-11-24 16:24:21,016 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3000, loss[loss=0.06396, simple_loss=0.08801, pruned_loss=0.01184, audio_tagging_loss=0.008115, over 15088.00 frames. ], tot_loss[loss=0.06764, simple_loss=0.09123, pruned_loss=0.01313, audio_tagging_loss=0.008895, over 3049179.67 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:24:21,019 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 16:25:01,967 INFO [train_asr.py:1253] (0/4) Epoch 37, validation: loss=0.05757, simple_loss=0.05085, pruned_loss=0.005185, audio_tagging_loss=0.02697, over 4681554.00 frames. 
2023-11-24 16:25:01,968 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 16:25:29,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2905853.3333333335, ans=0.95
2023-11-24 16:25:38,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2905920.0, ans=0.2
2023-11-24 16:25:49,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2905920.0, ans=0.2
2023-11-24 16:25:51,896 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435900
2023-11-24 16:26:03,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2906053.3333333335, ans=0.0
2023-11-24 16:26:04,090 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3050, loss[loss=0.05864, simple_loss=0.07762, pruned_loss=0.009335, audio_tagging_loss=0.0105, over 15114.00 frames. ], tot_loss[loss=0.06806, simple_loss=0.09151, pruned_loss=0.0133, audio_tagging_loss=0.009, over 3048802.48 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:26:15,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2906120.0, ans=0.0
2023-11-24 16:26:18,211 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.443e+01 9.096e+01 9.825e+01 1.321e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 16:26:36,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2906186.6666666665, ans=0.125
2023-11-24 16:26:40,241 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:26:53,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2906320.0, ans=0.125
2023-11-24 16:26:54,069 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 435950
2023-11-24 16:26:58,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2906320.0, ans=0.07
2023-11-24 16:27:03,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2906320.0, ans=0.1
2023-11-24 16:27:05,777 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3100, loss[loss=0.06829, simple_loss=0.09038, pruned_loss=0.01388, audio_tagging_loss=0.009224, over 14771.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09215, pruned_loss=0.01333, audio_tagging_loss=0.009028, over 3048215.22 frames. ], batch size: 55, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:27:27,236 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.49 vs. limit=12.0
2023-11-24 16:27:46,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2906586.6666666665, ans=0.125
2023-11-24 16:27:48,902 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2906586.6666666665, ans=0.09899494936611666
2023-11-24 16:27:51,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2906586.6666666665, ans=0.0
2023-11-24 16:27:55,669 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436000
2023-11-24 16:27:57,217 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-436000.pt
2023-11-24 16:28:12,032 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3150, loss[loss=0.04258, simple_loss=0.04657, pruned_loss=0.006716, audio_tagging_loss=0.01258, over 15892.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09168, pruned_loss=0.01315, audio_tagging_loss=0.009017, over 3042213.85 frames. ], batch size: 64, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:28:12,398 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2906720.0, ans=0.0
2023-11-24 16:28:16,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2906720.0, ans=0.2
2023-11-24 16:28:27,185 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.802e+01 8.617e+01 9.268e+01 9.907e+01 1.441e+02, threshold=1.854e+02, percent-clipped=0.0
2023-11-24 16:28:27,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2906786.6666666665, ans=0.2
2023-11-24 16:28:44,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2906853.3333333335, ans=0.1
2023-11-24 16:28:45,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.17 vs. limit=15.0
2023-11-24 16:28:46,475 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2906853.3333333335, ans=0.1
2023-11-24 16:28:52,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2906920.0, ans=0.125
2023-11-24 16:28:53,471 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=6.55 vs. limit=10.0
2023-11-24 16:28:55,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2906920.0, ans=0.0
2023-11-24 16:29:01,666 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436050
2023-11-24 16:29:14,529 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3200, loss[loss=0.07467, simple_loss=0.09944, pruned_loss=0.01571, audio_tagging_loss=0.009238, over 14696.00 frames. ], tot_loss[loss=0.06816, simple_loss=0.0919, pruned_loss=0.01315, audio_tagging_loss=0.009064, over 3047368.68 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:29:37,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2907186.6666666665, ans=0.0
2023-11-24 16:29:49,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2907186.6666666665, ans=0.1
2023-11-24 16:30:04,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436100
2023-11-24 16:30:05,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2907320.0, ans=0.125
2023-11-24 16:30:05,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2907320.0, ans=0.2
2023-11-24 16:30:15,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2907386.6666666665, ans=0.125
2023-11-24 16:30:16,187 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3250, loss[loss=0.061, simple_loss=0.07026, pruned_loss=0.01364, audio_tagging_loss=0.01222, over 15263.00 frames. ], tot_loss[loss=0.06826, simple_loss=0.09208, pruned_loss=0.01313, audio_tagging_loss=0.009093, over 3054408.66 frames. ], batch size: 57, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:30:28,573 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=4.16 vs. limit=12.0
2023-11-24 16:30:31,300 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.384e+01 8.491e+01 9.014e+01 9.810e+01 1.302e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 16:30:50,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2907520.0, ans=0.125
2023-11-24 16:31:05,885 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436150
2023-11-24 16:31:18,084 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3300, loss[loss=0.07415, simple_loss=0.1034, pruned_loss=0.01562, audio_tagging_loss=0.006809, over 15661.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09197, pruned_loss=0.01311, audio_tagging_loss=0.009064, over 3054506.96 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:31:23,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2907720.0, ans=0.0
2023-11-24 16:31:29,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2907786.6666666665, ans=0.0
2023-11-24 16:31:45,442 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2907853.3333333335, ans=0.125
2023-11-24 16:31:53,813 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:31:55,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2907920.0, ans=0.2
2023-11-24 16:31:59,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2907920.0, ans=0.125
2023-11-24 16:32:06,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2907986.6666666665, ans=0.0
2023-11-24 16:32:07,871 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436200
2023-11-24 16:32:21,580 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3350, loss[loss=0.06677, simple_loss=0.09628, pruned_loss=0.01139, audio_tagging_loss=0.007243, over 15734.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09214, pruned_loss=0.01318, audio_tagging_loss=0.008989, over 3052822.30 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:32:26,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2908053.3333333335, ans=0.125
2023-11-24 16:32:27,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2908053.3333333335, ans=0.05
2023-11-24 16:32:35,790 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.051e+01 8.766e+01 9.367e+01 1.008e+02 1.183e+02, threshold=1.873e+02, percent-clipped=0.0
2023-11-24 16:32:49,907 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:32:53,618 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.76 vs. limit=22.5
2023-11-24 16:33:07,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2908253.3333333335, ans=0.125
2023-11-24 16:33:11,484 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436250
2023-11-24 16:33:18,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2908320.0, ans=0.125
2023-11-24 16:33:22,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2908386.6666666665, ans=0.0
2023-11-24 16:33:23,274 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3400, loss[loss=0.07245, simple_loss=0.0983, pruned_loss=0.01147, audio_tagging_loss=0.01183, over 15161.00 frames. ], tot_loss[loss=0.06817, simple_loss=0.09209, pruned_loss=0.01316, audio_tagging_loss=0.008962, over 3053236.95 frames. ], batch size: 56, lr: 1.84e-03, grad_scale: 32.0
2023-11-24 16:33:28,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2908386.6666666665, ans=0.2
2023-11-24 16:33:53,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2908520.0, ans=0.2
2023-11-24 16:34:04,911 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2908586.6666666665, ans=0.0
2023-11-24 16:34:06,331 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.99 vs. limit=10.0
2023-11-24 16:34:13,206 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436300
2023-11-24 16:34:15,292 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.98 vs. limit=22.5
2023-11-24 16:34:15,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.max_positive, batch_count=2908653.3333333335, ans=0.95
2023-11-24 16:34:18,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2908653.3333333335, ans=0.1
2023-11-24 16:34:26,322 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3450, loss[loss=0.08915, simple_loss=0.1214, pruned_loss=0.02272, audio_tagging_loss=0.005712, over 15964.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09087, pruned_loss=0.0129, audio_tagging_loss=0.008804, over 3043976.25 frames. ], batch size: 58, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:34:42,909 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.858e+01 8.653e+01 9.241e+01 9.942e+01 2.012e+02, threshold=1.848e+02, percent-clipped=1.0
2023-11-24 16:34:44,942 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.98 vs. limit=15.0
2023-11-24 16:34:52,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2908853.3333333335, ans=0.125
2023-11-24 16:35:16,732 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436350
2023-11-24 16:35:22,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2908986.6666666665, ans=0.125
2023-11-24 16:35:29,774 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3500, loss[loss=0.04237, simple_loss=0.0564, pruned_loss=0.005868, audio_tagging_loss=0.008305, over 15343.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.09062, pruned_loss=0.01281, audio_tagging_loss=0.008775, over 3047857.49 frames. ], batch size: 59, lr: 1.84e-03, grad_scale: 16.0
2023-11-24 16:35:52,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2909120.0, ans=0.125
2023-11-24 16:36:00,998 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:36:03,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2909186.6666666665, ans=0.125
2023-11-24 16:36:20,166 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436400
2023-11-24 16:36:33,056 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3550, loss[loss=0.06872, simple_loss=0.0906, pruned_loss=0.01468, audio_tagging_loss=0.008744, over 15153.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.09062, pruned_loss=0.01268, audio_tagging_loss=0.008723, over 3048557.22 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:36:39,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2909386.6666666665, ans=0.125
2023-11-24 16:36:49,076 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.431e+01 8.760e+01 9.470e+01 1.011e+02 1.264e+02, threshold=1.894e+02, percent-clipped=0.0
2023-11-24 16:36:53,551 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.17 vs. limit=22.5
2023-11-24 16:36:58,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2909520.0, ans=0.1
2023-11-24 16:37:19,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2909586.6666666665, ans=0.125
2023-11-24 16:37:22,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436450
2023-11-24 16:37:35,208 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3600, loss[loss=0.06356, simple_loss=0.08174, pruned_loss=0.01275, audio_tagging_loss=0.009948, over 15903.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.09069, pruned_loss=0.01262, audio_tagging_loss=0.008717, over 3048551.07 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:38:11,560 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=14.68 vs. limit=15.0
2023-11-24 16:38:25,046 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436500
2023-11-24 16:38:37,893 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3650, loss[loss=0.06914, simple_loss=0.09726, pruned_loss=0.00981, audio_tagging_loss=0.01069, over 14495.00 frames. ], tot_loss[loss=0.06656, simple_loss=0.09037, pruned_loss=0.01266, audio_tagging_loss=0.008725, over 3049702.28 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:38:54,064 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.268e+01 8.295e+01 9.068e+01 9.649e+01 1.086e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 16:38:54,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2910120.0, ans=0.0
2023-11-24 16:39:04,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2910186.6666666665, ans=0.0
2023-11-24 16:39:27,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436550
2023-11-24 16:39:39,415 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3700, loss[loss=0.07162, simple_loss=0.1021, pruned_loss=0.01165, audio_tagging_loss=0.008933, over 15783.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.09078, pruned_loss=0.01279, audio_tagging_loss=0.008714, over 3052092.39 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:39:47,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2910386.6666666665, ans=0.125
2023-11-24 16:39:48,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2910386.6666666665, ans=0.125
2023-11-24 16:40:07,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2910520.0, ans=0.2
2023-11-24 16:40:29,926 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436600
2023-11-24 16:40:36,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2910653.3333333335, ans=0.0
2023-11-24 16:40:43,391 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3750, loss[loss=0.06884, simple_loss=0.09899, pruned_loss=0.009856, audio_tagging_loss=0.009488, over 14632.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09148, pruned_loss=0.01303, audio_tagging_loss=0.008714, over 3057225.08 frames. ], batch size: 52, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:40:45,393 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.75 vs. limit=12.0
2023-11-24 16:40:46,131 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2910720.0, ans=0.125
2023-11-24 16:40:46,414 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=9.02 vs. limit=15.0
2023-11-24 16:40:53,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2910720.0, ans=0.125
2023-11-24 16:40:54,892 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:41:01,145 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.275e+01 8.745e+01 9.267e+01 9.947e+01 1.281e+02, threshold=1.853e+02, percent-clipped=0.0
2023-11-24 16:41:01,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2910786.6666666665, ans=0.0
2023-11-24 16:41:07,709 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.67 vs. limit=6.0
2023-11-24 16:41:25,468 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:41:29,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2910920.0, ans=0.1
2023-11-24 16:41:33,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436650
2023-11-24 16:41:41,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2910986.6666666665, ans=0.1
2023-11-24 16:41:45,770 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3800, loss[loss=0.06431, simple_loss=0.08224, pruned_loss=0.01534, audio_tagging_loss=0.007842, over 14448.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09156, pruned_loss=0.01304, audio_tagging_loss=0.008646, over 3055175.67 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:42:00,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2911120.0, ans=0.125
2023-11-24 16:42:28,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2911253.3333333335, ans=0.125
2023-11-24 16:42:36,222 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436700
2023-11-24 16:42:37,818 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.99 vs. limit=15.0
2023-11-24 16:42:38,902 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 16:42:48,431 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3850, loss[loss=0.03664, simple_loss=0.04212, pruned_loss=0.003482, audio_tagging_loss=0.0121, over 14595.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09127, pruned_loss=0.01305, audio_tagging_loss=0.008759, over 3054117.57 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:43:03,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2911453.3333333335, ans=0.2
2023-11-24 16:43:03,633 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2911453.3333333335, ans=0.125
2023-11-24 16:43:05,596 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.06 vs. limit=10.0
2023-11-24 16:43:06,191 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.966e+01 8.511e+01 9.199e+01 9.867e+01 1.160e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-24 16:43:15,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.96 vs. limit=12.0
2023-11-24 16:43:21,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2911520.0, ans=0.125
2023-11-24 16:43:27,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2911586.6666666665, ans=0.05
2023-11-24 16:43:38,389 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436750
2023-11-24 16:43:38,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2911653.3333333335, ans=0.0
2023-11-24 16:43:43,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2911653.3333333335, ans=0.125
2023-11-24 16:43:50,758 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3900, loss[loss=0.06028, simple_loss=0.07747, pruned_loss=0.01019, audio_tagging_loss=0.01135, over 14353.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09102, pruned_loss=0.01304, audio_tagging_loss=0.008845, over 3053895.13 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:44:03,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2911786.6666666665, ans=0.1
2023-11-24 16:44:17,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2911853.3333333335, ans=0.1
2023-11-24 16:44:18,216 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.66 vs. limit=6.0
2023-11-24 16:44:38,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2911920.0, ans=0.125
2023-11-24 16:44:41,519 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436800
2023-11-24 16:44:44,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2911986.6666666665, ans=0.125
2023-11-24 16:44:44,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2911986.6666666665, ans=0.125
2023-11-24 16:44:48,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2911986.6666666665, ans=0.0
2023-11-24 16:44:54,256 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 3950, loss[loss=0.08898, simple_loss=0.1155, pruned_loss=0.01951, audio_tagging_loss=0.01175, over 15893.00 frames. ], tot_loss[loss=0.06721, simple_loss=0.09047, pruned_loss=0.01292, audio_tagging_loss=0.009052, over 3056388.46 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:45:11,315 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.836e+01 8.639e+01 9.043e+01 9.863e+01 1.208e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 16:45:41,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2912253.3333333335, ans=0.0
2023-11-24 16:45:44,223 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436850
2023-11-24 16:45:47,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2912320.0, ans=0.07
2023-11-24 16:45:53,361 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2912320.0, ans=0.125
2023-11-24 16:45:56,545 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4000, loss[loss=0.05563, simple_loss=0.07452, pruned_loss=0.008785, audio_tagging_loss=0.009581, over 16843.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09144, pruned_loss=0.01303, audio_tagging_loss=0.009071, over 3061391.82 frames. ], batch size: 66, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:46:05,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2912386.6666666665, ans=0.125
2023-11-24 16:46:09,915 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2912453.3333333335, ans=0.0
2023-11-24 16:46:12,118 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2912453.3333333335, ans=0.1
2023-11-24 16:46:13,726 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.65 vs. limit=15.0
2023-11-24 16:46:18,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2912453.3333333335, ans=10.0
2023-11-24 16:46:22,202 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.53 vs. limit=15.0
2023-11-24 16:46:45,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2912653.3333333335, ans=0.0
2023-11-24 16:46:46,516 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436900
2023-11-24 16:46:46,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2912653.3333333335, ans=0.1
2023-11-24 16:46:50,309 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2912653.3333333335, ans=0.0
2023-11-24 16:46:58,172 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4050, loss[loss=0.04336, simple_loss=0.05451, pruned_loss=0.005044, audio_tagging_loss=0.01106, over 14868.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09157, pruned_loss=0.01307, audio_tagging_loss=0.009127, over 3056905.90 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:47:00,523 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:47:16,513 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.242e+01 8.690e+01 9.276e+01 1.004e+02 1.184e+02, threshold=1.855e+02, percent-clipped=0.0
2023-11-24 16:47:23,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2912853.3333333335, ans=0.0
2023-11-24 16:47:36,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2912920.0, ans=0.125
2023-11-24 16:47:47,928 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 436950
2023-11-24 16:47:49,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2912986.6666666665, ans=0.2
2023-11-24 16:47:56,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2912986.6666666665, ans=0.0
2023-11-24 16:47:56,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2912986.6666666665, ans=0.2
2023-11-24 16:48:01,369 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4100, loss[loss=0.06489, simple_loss=0.09385, pruned_loss=0.009209, audio_tagging_loss=0.008754, over 15771.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09222, pruned_loss=0.01301, audio_tagging_loss=0.00907, over 3059601.80 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 16:48:27,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=10.29 vs. limit=15.0
2023-11-24 16:48:45,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2913253.3333333335, ans=0.0
2023-11-24 16:48:51,499 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437000
2023-11-24 16:49:04,132 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4150, loss[loss=0.08024, simple_loss=0.1076, pruned_loss=0.01802, audio_tagging_loss=0.008434, over 15045.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09227, pruned_loss=0.01315, audio_tagging_loss=0.008907, over 3054651.16 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:49:17,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2913453.3333333335, ans=0.125
2023-11-24 16:49:22,042 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.962e+01 8.740e+01 9.379e+01 9.988e+01 1.190e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 16:49:32,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2913520.0, ans=0.125
2023-11-24 16:49:41,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2913586.6666666665, ans=0.1
2023-11-24 16:49:42,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2913586.6666666665, ans=0.125
2023-11-24 16:49:44,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2913586.6666666665, ans=0.125
2023-11-24 16:49:45,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2913586.6666666665, ans=0.0
2023-11-24 16:49:47,801 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:49:53,883 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437050
2023-11-24 16:49:56,903 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.31 vs. limit=15.0
2023-11-24 16:50:01,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2913653.3333333335, ans=0.0
2023-11-24 16:50:04,834 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2913720.0, ans=0.2
2023-11-24 16:50:05,833 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4200, loss[loss=0.06326, simple_loss=0.07998, pruned_loss=0.01262, audio_tagging_loss=0.01065, over 14781.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09175, pruned_loss=0.01295, audio_tagging_loss=0.008763, over 3049118.27 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:50:24,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2913786.6666666665, ans=0.125
2023-11-24 16:50:45,643 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2913920.0, ans=0.2
2023-11-24 16:50:56,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437100
2023-11-24 16:51:01,785 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2913986.6666666665, ans=0.5
2023-11-24 16:51:07,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2914053.3333333335, ans=0.0
2023-11-24 16:51:08,513 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4250, loss[loss=0.06455, simple_loss=0.08876, pruned_loss=0.01114, audio_tagging_loss=0.00903, over 14392.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09134, pruned_loss=0.01297, audio_tagging_loss=0.00882, over 3044184.67 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:51:08,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2914053.3333333335, ans=0.2
2023-11-24 16:51:27,414 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.332e+01 8.707e+01 9.618e+01 1.026e+02 1.343e+02, threshold=1.924e+02, percent-clipped=0.0
2023-11-24 16:51:45,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2914253.3333333335, ans=0.125
2023-11-24 16:51:57,712 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437150
2023-11-24 16:52:03,348 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.59 vs. limit=15.0
2023-11-24 16:52:10,481 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4300, loss[loss=0.08676, simple_loss=0.1213, pruned_loss=0.0171, audio_tagging_loss=0.008986, over 15851.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09244, pruned_loss=0.01317, audio_tagging_loss=0.008658, over 3048417.99 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:52:10,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2914386.6666666665, ans=0.2
2023-11-24 16:52:23,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2914453.3333333335, ans=0.1
2023-11-24 16:52:31,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2914453.3333333335, ans=0.125
2023-11-24 16:52:45,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2914520.0, ans=0.125
2023-11-24 16:53:00,522 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437200
2023-11-24 16:53:05,977 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=13.10 vs. limit=22.5
2023-11-24 16:53:06,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2914653.3333333335, ans=0.125
2023-11-24 16:53:11,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2914720.0, ans=0.125
2023-11-24 16:53:12,547 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4350, loss[loss=0.07044, simple_loss=0.1011, pruned_loss=0.01151, audio_tagging_loss=0.008353, over 15098.00 frames. ], tot_loss[loss=0.06827, simple_loss=0.09271, pruned_loss=0.01322, audio_tagging_loss=0.008687, over 3049477.01 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:53:23,124 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.40 vs. limit=22.5
2023-11-24 16:53:32,388 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.693e+01 9.378e+01 1.028e+02 1.311e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 16:53:35,712 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.54 vs. limit=15.0
2023-11-24 16:53:47,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2914853.3333333335, ans=0.125
2023-11-24 16:53:55,433 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.19 vs. limit=15.0
2023-11-24 16:53:57,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2914920.0, ans=0.04949747468305833
2023-11-24 16:54:01,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437250
2023-11-24 16:54:11,823 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2914986.6666666665, ans=0.0
2023-11-24 16:54:13,960 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4400, loss[loss=0.09309, simple_loss=0.1364, pruned_loss=0.01834, audio_tagging_loss=0.006558, over 16760.00 frames. ], tot_loss[loss=0.06836, simple_loss=0.09284, pruned_loss=0.0133, audio_tagging_loss=0.008639, over 3051959.41 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 16:54:16,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2915053.3333333335, ans=0.125
2023-11-24 16:54:24,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2915053.3333333335, ans=0.125
2023-11-24 16:54:29,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=2915120.0, ans=0.05
2023-11-24 16:54:37,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2915120.0, ans=0.0
2023-11-24 16:55:02,845 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.74 vs. limit=15.0
2023-11-24 16:55:03,466 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437300
2023-11-24 16:55:16,833 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4450, loss[loss=0.06731, simple_loss=0.09896, pruned_loss=0.009002, audio_tagging_loss=0.008825, over 15072.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09262, pruned_loss=0.01325, audio_tagging_loss=0.008666, over 3055925.34 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:55:36,773 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.463e+01 9.107e+01 9.644e+01 1.325e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 16:56:06,659 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437350
2023-11-24 16:56:18,388 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4500, loss[loss=0.09962, simple_loss=0.1342, pruned_loss=0.02296, audio_tagging_loss=0.009568, over 15869.00 frames. ], tot_loss[loss=0.06835, simple_loss=0.09272, pruned_loss=0.01331, audio_tagging_loss=0.008683, over 3046871.32 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:56:22,505 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.81 vs. limit=22.5
2023-11-24 16:56:23,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.34 vs. limit=15.0
2023-11-24 16:56:32,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2915786.6666666665, ans=0.2
2023-11-24 16:56:34,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2915786.6666666665, ans=0.0
2023-11-24 16:57:04,312 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2915920.0, ans=0.0
2023-11-24 16:57:07,664 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437400
2023-11-24 16:57:11,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2915986.6666666665, ans=0.1
2023-11-24 16:57:12,138 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.96 vs. limit=15.0
2023-11-24 16:57:20,279 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4550, loss[loss=0.06608, simple_loss=0.08312, pruned_loss=0.01319, audio_tagging_loss=0.01133, over 15102.00 frames. ], tot_loss[loss=0.06825, simple_loss=0.09268, pruned_loss=0.01317, audio_tagging_loss=0.008741, over 3047694.35 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 4.0
2023-11-24 16:57:29,852 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=20.79 vs. limit=22.5
2023-11-24 16:57:43,340 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.438e+01 8.634e+01 9.216e+01 9.790e+01 1.251e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 16:57:47,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2916186.6666666665, ans=0.95
2023-11-24 16:58:06,780 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 16:58:10,358 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437450
2023-11-24 16:58:10,935 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.16 vs. limit=12.0
2023-11-24 16:58:17,394 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0
2023-11-24 16:58:23,349 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4600, loss[loss=0.06685, simple_loss=0.08277, pruned_loss=0.01202, audio_tagging_loss=0.01345, over 14889.00 frames. ], tot_loss[loss=0.06795, simple_loss=0.09199, pruned_loss=0.01313, audio_tagging_loss=0.008821, over 3046325.62 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:58:34,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2916453.3333333335, ans=0.125
2023-11-24 16:58:51,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2916520.0, ans=0.2
2023-11-24 16:58:54,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2916520.0, ans=0.1
2023-11-24 16:58:57,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2916520.0, ans=0.0
2023-11-24 16:59:12,972 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437500
2023-11-24 16:59:13,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2916653.3333333335, ans=0.0
2023-11-24 16:59:18,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1.whitening_limit, batch_count=2916653.3333333335, ans=10.0
2023-11-24 16:59:23,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2916653.3333333335, ans=0.0
2023-11-24 16:59:25,201 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4650, loss[loss=0.06564, simple_loss=0.08725, pruned_loss=0.01249, audio_tagging_loss=0.009525, over 14868.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09162, pruned_loss=0.01305, audio_tagging_loss=0.00885, over 3046872.19 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 16:59:41,190 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2916786.6666666665, ans=0.125
2023-11-24 16:59:46,885 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.369e+01 8.291e+01 9.050e+01 9.890e+01 1.291e+02, threshold=1.810e+02, percent-clipped=0.0
2023-11-24 16:59:51,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2916853.3333333335, ans=0.125
2023-11-24 17:00:14,709 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437550
2023-11-24 17:00:24,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2916986.6666666665, ans=0.0
2023-11-24 17:00:27,367 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4700, loss[loss=0.05383, simple_loss=0.06526, pruned_loss=0.01053, audio_tagging_loss=0.01068, over 15058.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.0915, pruned_loss=0.01307, audio_tagging_loss=0.008989, over 3047301.72 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:00:38,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2917120.0, ans=0.0
2023-11-24 17:00:46,995 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.94 vs. limit=15.0
2023-11-24 17:00:53,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2917186.6666666665, ans=0.2
2023-11-24 17:01:14,572 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.74 vs. limit=15.0
2023-11-24 17:01:17,432 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437600
2023-11-24 17:01:30,268 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4750, loss[loss=0.06359, simple_loss=0.07181, pruned_loss=0.01505, audio_tagging_loss=0.01263, over 14072.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09075, pruned_loss=0.01283, audio_tagging_loss=0.009048, over 3048041.01 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:01:40,933 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.16 vs. limit=15.0
2023-11-24 17:01:52,694 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.496e+01 8.751e+01 9.417e+01 1.046e+02 1.255e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-24 17:01:53,108 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2917453.3333333335, ans=0.125
2023-11-24 17:01:54,194 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:01:57,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2917520.0, ans=0.125
2023-11-24 17:02:04,152 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2917520.0, ans=0.0
2023-11-24 17:02:06,700 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2917586.6666666665, ans=0.1
2023-11-24 17:02:14,544 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.62 vs. limit=15.0
2023-11-24 17:02:19,599 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.38 vs. limit=15.0
2023-11-24 17:02:20,688 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437650
2023-11-24 17:02:32,325 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4800, loss[loss=0.077, simple_loss=0.09606, pruned_loss=0.01847, audio_tagging_loss=0.01051, over 16465.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09108, pruned_loss=0.01303, audio_tagging_loss=0.009129, over 3053978.90 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:02:36,939 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=12.0
2023-11-24 17:02:57,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2917853.3333333335, ans=0.125
2023-11-24 17:03:00,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.23 vs. limit=15.0
2023-11-24 17:03:19,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2917920.0, ans=0.125
2023-11-24 17:03:21,886 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437700
2023-11-24 17:03:34,102 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4850, loss[loss=0.07937, simple_loss=0.1175, pruned_loss=0.01287, audio_tagging_loss=0.00774, over 14470.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09099, pruned_loss=0.0129, audio_tagging_loss=0.009234, over 3044090.32 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:03:35,827 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.00 vs. limit=6.0
2023-11-24 17:03:58,341 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.433e+01 8.659e+01 9.301e+01 9.915e+01 1.482e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-24 17:04:18,822 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.22 vs. limit=15.0
2023-11-24 17:04:24,711 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437750
2023-11-24 17:04:36,857 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4900, loss[loss=0.06725, simple_loss=0.09601, pruned_loss=0.01269, audio_tagging_loss=0.006561, over 15779.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.0914, pruned_loss=0.01295, audio_tagging_loss=0.009157, over 3044157.58 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:04:37,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2918386.6666666665, ans=0.125
2023-11-24 17:04:45,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2918386.6666666665, ans=0.0
2023-11-24 17:04:49,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2918453.3333333335, ans=0.2
2023-11-24 17:04:57,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2918453.3333333335, ans=0.1
2023-11-24 17:04:57,626 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.58 vs. limit=12.0
2023-11-24 17:05:09,878 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:05:18,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2918586.6666666665, ans=0.0
2023-11-24 17:05:21,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2918586.6666666665, ans=0.125
2023-11-24 17:05:27,735 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437800
2023-11-24 17:05:37,864 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2918653.3333333335, ans=0.0
2023-11-24 17:05:39,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2918720.0, ans=0.0
2023-11-24 17:05:39,983 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 4950, loss[loss=0.08377, simple_loss=0.1057, pruned_loss=0.02319, audio_tagging_loss=0.007707, over 14589.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.0911, pruned_loss=0.01287, audio_tagging_loss=0.009086, over 3043196.74 frames. ], batch size: 52, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:05:41,984 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=9.18 vs. limit=15.0
2023-11-24 17:05:45,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2918720.0, ans=0.125
2023-11-24 17:05:58,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2918786.6666666665, ans=0.125
2023-11-24 17:06:04,044 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.080e+01 8.424e+01 9.084e+01 9.605e+01 1.394e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 17:06:14,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2918853.3333333335, ans=0.125
2023-11-24 17:06:18,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2918920.0, ans=0.125
2023-11-24 17:06:22,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2918920.0, ans=0.0
2023-11-24 17:06:28,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2918920.0, ans=0.0
2023-11-24 17:06:30,241 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437850
2023-11-24 17:06:38,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2918986.6666666665, ans=0.125
2023-11-24 17:06:42,661 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5000, loss[loss=0.05791, simple_loss=0.08083, pruned_loss=0.009205, audio_tagging_loss=0.008291, over 15398.00 frames. ], tot_loss[loss=0.06673, simple_loss=0.09015, pruned_loss=0.01273, audio_tagging_loss=0.008925, over 3048243.60 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:06:51,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2919053.3333333335, ans=0.1
2023-11-24 17:06:55,896 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.75 vs. limit=15.0
2023-11-24 17:07:13,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2919186.6666666665, ans=0.1
2023-11-24 17:07:24,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2919253.3333333335, ans=0.0
2023-11-24 17:07:32,496 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437900
2023-11-24 17:07:45,507 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5050, loss[loss=0.06745, simple_loss=0.09465, pruned_loss=0.01205, audio_tagging_loss=0.008073, over 14435.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09079, pruned_loss=0.01278, audio_tagging_loss=0.008833, over 3051080.74 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:08:00,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2919453.3333333335, ans=0.125
2023-11-24 17:08:01,438 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:08:06,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2919453.3333333335, ans=0.125
2023-11-24 17:08:06,672 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.60 vs. limit=10.0
2023-11-24 17:08:08,405 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.782e+01 8.595e+01 9.107e+01 9.818e+01 1.374e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 17:08:12,675 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.30 vs. limit=22.5
2023-11-24 17:08:13,477 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=2919520.0, ans=0.0
2023-11-24 17:08:31,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2919586.6666666665, ans=0.0
2023-11-24 17:08:35,315 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 437950
2023-11-24 17:08:43,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2919653.3333333335, ans=0.0
2023-11-24 17:08:47,805 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5100, loss[loss=0.04445, simple_loss=0.05713, pruned_loss=0.005944, audio_tagging_loss=0.009937, over 14435.00 frames. ], tot_loss[loss=0.06634, simple_loss=0.08992, pruned_loss=0.01255, audio_tagging_loss=0.008831, over 3045172.28 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:08:51,807 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.03 vs. limit=15.0
2023-11-24 17:09:37,355 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438000
2023-11-24 17:09:37,831 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.63 vs. limit=10.0
2023-11-24 17:09:39,995 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2919986.6666666665, ans=0.125
2023-11-24 17:09:49,267 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5150, loss[loss=0.06044, simple_loss=0.08047, pruned_loss=0.01293, audio_tagging_loss=0.00727, over 15218.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.09051, pruned_loss=0.01271, audio_tagging_loss=0.008863, over 3041010.77 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:10:13,399 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.241e+01 8.605e+01 9.201e+01 1.005e+02 1.322e+02, threshold=1.840e+02, percent-clipped=0.0
2023-11-24 17:10:19,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2920186.6666666665, ans=0.2
2023-11-24 17:10:23,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2920186.6666666665, ans=0.125
2023-11-24 17:10:25,623 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.02 vs. limit=15.0
2023-11-24 17:10:33,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2920253.3333333335, ans=0.125
2023-11-24 17:10:39,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438050
2023-11-24 17:10:52,002 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5200, loss[loss=0.06974, simple_loss=0.1067, pruned_loss=0.008051, audio_tagging_loss=0.008342, over 15303.00 frames. ], tot_loss[loss=0.06676, simple_loss=0.09021, pruned_loss=0.01274, audio_tagging_loss=0.008919, over 3042522.81 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:11:04,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2920453.3333333335, ans=0.0
2023-11-24 17:11:08,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2920453.3333333335, ans=0.125
2023-11-24 17:11:27,853 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2920586.6666666665, ans=0.125
2023-11-24 17:11:42,539 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438100
2023-11-24 17:11:49,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2920653.3333333335, ans=0.0
2023-11-24 17:11:55,062 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5250, loss[loss=0.06662, simple_loss=0.09107, pruned_loss=0.01283, audio_tagging_loss=0.008261, over 15458.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09068, pruned_loss=0.01291, audio_tagging_loss=0.00881, over 3039633.83 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:12:06,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2920786.6666666665, ans=0.1
2023-11-24 17:12:06,172 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.39 vs. limit=10.0
2023-11-24 17:12:17,931 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.875e+01 8.449e+01 8.931e+01 9.765e+01 1.225e+02, threshold=1.786e+02, percent-clipped=0.0
2023-11-24 17:12:45,681 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438150
2023-11-24 17:12:57,276 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5300, loss[loss=0.07268, simple_loss=0.1059, pruned_loss=0.01323, audio_tagging_loss=0.006517, over 15956.00 frames. ], tot_loss[loss=0.0675, simple_loss=0.09136, pruned_loss=0.01305, audio_tagging_loss=0.008763, over 3038334.29 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:13:00,157 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=4.85 vs. limit=10.0
2023-11-24 17:13:28,521 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2921186.6666666665, ans=0.0
2023-11-24 17:13:30,809 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2921186.6666666665, ans=0.125
2023-11-24 17:13:34,430 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2921253.3333333335, ans=0.0
2023-11-24 17:13:36,230 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.00 vs. limit=15.0
2023-11-24 17:13:38,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2921253.3333333335, ans=0.1
2023-11-24 17:13:44,156 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2921253.3333333335, ans=0.125
2023-11-24 17:13:47,411 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438200
2023-11-24 17:13:50,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=2921320.0, ans=0.09899494936611666
2023-11-24 17:13:52,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2921320.0, ans=0.2
2023-11-24 17:14:00,118 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5350, loss[loss=0.05189, simple_loss=0.0641, pruned_loss=0.01039, audio_tagging_loss=0.009451, over 14300.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09157, pruned_loss=0.01318, audio_tagging_loss=0.008748, over 3032508.02 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:14:12,285 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:14:24,405 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.013e+01 8.487e+01 9.193e+01 9.991e+01 1.472e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 17:14:24,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.min_positive, batch_count=2921520.0, ans=0.05
2023-11-24 17:14:27,390 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.80 vs. limit=15.0
2023-11-24 17:14:43,001 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.18 vs. limit=12.0
2023-11-24 17:14:47,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2921586.6666666665, ans=0.0
2023-11-24 17:14:49,463 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.79 vs. limit=22.5
2023-11-24 17:14:49,949 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438250
2023-11-24 17:14:50,220 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2921653.3333333335, ans=0.125
2023-11-24 17:14:58,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2921653.3333333335, ans=0.125
2023-11-24 17:15:03,525 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5400, loss[loss=0.07736, simple_loss=0.1032, pruned_loss=0.01684, audio_tagging_loss=0.008914, over 14669.00 frames. ], tot_loss[loss=0.06803, simple_loss=0.0919, pruned_loss=0.01331, audio_tagging_loss=0.008771, over 3034385.95 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:15:20,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2921786.6666666665, ans=0.1
2023-11-24 17:15:25,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2921786.6666666665, ans=0.125
2023-11-24 17:15:42,260 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2921920.0, ans=0.125
2023-11-24 17:15:47,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2921920.0, ans=0.0
2023-11-24 17:15:53,249 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438300
2023-11-24 17:16:04,906 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5450, loss[loss=0.07763, simple_loss=0.105, pruned_loss=0.01414, audio_tagging_loss=0.01097, over 15059.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.0913, pruned_loss=0.01325, audio_tagging_loss=0.008799, over 3034427.14 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:16:13,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2922053.3333333335, ans=0.015
2023-11-24 17:16:20,420 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2922120.0, ans=0.0
2023-11-24 17:16:25,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2922120.0, ans=0.125
2023-11-24 17:16:28,923 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.120e+01 8.393e+01 9.258e+01 1.002e+02 1.486e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 17:16:55,978 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438350
2023-11-24 17:17:08,723 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5500, loss[loss=0.05078, simple_loss=0.06396, pruned_loss=0.01007, audio_tagging_loss=0.008732, over 15725.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09151, pruned_loss=0.01338, audio_tagging_loss=0.008864, over 3030316.35 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:17:42,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2922520.0, ans=0.125
2023-11-24 17:17:55,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2922586.6666666665, ans=0.125
2023-11-24 17:17:58,573 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438400
2023-11-24 17:18:11,628 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5550, loss[loss=0.08464, simple_loss=0.1215, pruned_loss=0.01707, audio_tagging_loss=0.006832, over 15802.00 frames. ], tot_loss[loss=0.06822, simple_loss=0.09193, pruned_loss=0.01336, audio_tagging_loss=0.008899, over 3037821.30 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:18:33,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2922786.6666666665, ans=0.125
2023-11-24 17:18:34,540 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.462e+01 8.681e+01 9.449e+01 1.002e+02 1.118e+02, threshold=1.890e+02, percent-clipped=0.0
2023-11-24 17:18:39,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2922853.3333333335, ans=0.125
2023-11-24 17:19:01,984 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438450
2023-11-24 17:19:03,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2922986.6666666665, ans=0.0
2023-11-24 17:19:03,934 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=16.14 vs. limit=15.0
2023-11-24 17:19:09,111 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2922986.6666666665, ans=0.0
2023-11-24 17:19:13,620 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5600, loss[loss=0.07063, simple_loss=0.09371, pruned_loss=0.01441, audio_tagging_loss=0.009361, over 14830.00 frames. ], tot_loss[loss=0.06818, simple_loss=0.09168, pruned_loss=0.01329, audio_tagging_loss=0.009045, over 3042257.51 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:19:17,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2923053.3333333335, ans=0.125
2023-11-24 17:19:25,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.55 vs. limit=15.0
2023-11-24 17:19:41,292 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.41 vs. limit=15.0
2023-11-24 17:19:50,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2923253.3333333335, ans=0.1
2023-11-24 17:19:51,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2923253.3333333335, ans=0.125
2023-11-24 17:19:54,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2923253.3333333335, ans=0.125
2023-11-24 17:19:57,411 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 17:20:03,523 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438500
2023-11-24 17:20:03,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2923320.0, ans=0.0
2023-11-24 17:20:05,487 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.98 vs. limit=15.0
2023-11-24 17:20:10,842 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:20:15,350 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5650, loss[loss=0.04919, simple_loss=0.0618, pruned_loss=0.009848, audio_tagging_loss=0.008446, over 13931.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09152, pruned_loss=0.01341, audio_tagging_loss=0.009143, over 3041937.99 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:20:40,181 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.520e+01 8.503e+01 9.097e+01 9.762e+01 1.252e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 17:20:47,811 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=2923520.0, ans=0.2
2023-11-24 17:20:48,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2923520.0, ans=0.2
2023-11-24 17:20:57,239 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2923586.6666666665, ans=0.125
2023-11-24 17:21:03,453 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.13 vs. limit=15.0
2023-11-24 17:21:05,877 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438550
2023-11-24 17:21:16,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2923653.3333333335, ans=0.1
2023-11-24 17:21:18,241 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5700, loss[loss=0.08804, simple_loss=0.1187, pruned_loss=0.02159, audio_tagging_loss=0.007083, over 15142.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09098, pruned_loss=0.01339, audio_tagging_loss=0.00906, over 3036438.49 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:21:19,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2923720.0, ans=0.2
2023-11-24 17:21:29,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2923720.0, ans=0.125
2023-11-24 17:21:30,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2923786.6666666665, ans=0.1
2023-11-24 17:21:39,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2923786.6666666665, ans=0.0
2023-11-24 17:21:39,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2923786.6666666665, ans=0.125
2023-11-24 17:21:41,117 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2923786.6666666665, ans=0.2
2023-11-24 17:21:43,593 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2923853.3333333335, ans=0.0
2023-11-24 17:21:48,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=2923853.3333333335, ans=0.125
2023-11-24 17:21:56,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2923920.0, ans=0.125
2023-11-24 17:22:08,468 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438600
2023-11-24 17:22:11,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2923986.6666666665, ans=0.0
2023-11-24 17:22:21,773 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5750, loss[loss=0.07599, simple_loss=0.1015, pruned_loss=0.01792, audio_tagging_loss=0.0073, over 15704.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09064, pruned_loss=0.01331, audio_tagging_loss=0.008947, over 3041960.30 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:22:24,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=2924053.3333333335, ans=0.2
2023-11-24 17:22:26,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2924053.3333333335, ans=0.025
2023-11-24 17:22:28,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2924053.3333333335, ans=0.125
2023-11-24 17:22:36,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2924120.0, ans=0.2
2023-11-24 17:22:44,765 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.490e+01 8.615e+01 9.261e+01 9.857e+01 1.309e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 17:22:51,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2924186.6666666665, ans=0.0
2023-11-24 17:22:52,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2924186.6666666665, ans=0.2
2023-11-24 17:22:56,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2924186.6666666665, ans=0.1
2023-11-24 17:22:56,201 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:23:11,360 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438650
2023-11-24 17:23:22,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2924386.6666666665, ans=0.125
2023-11-24 17:23:23,082 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5800, loss[loss=0.04847, simple_loss=0.05775, pruned_loss=0.007742, audio_tagging_loss=0.01185, over 14862.00 frames. ], tot_loss[loss=0.06749, simple_loss=0.09096, pruned_loss=0.01318, audio_tagging_loss=0.008826, over 3045180.57 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:23:38,636 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2924453.3333333335, ans=0.125
2023-11-24 17:23:39,089 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=6.80 vs. limit=15.0
2023-11-24 17:23:46,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2924453.3333333335, ans=0.0
2023-11-24 17:24:02,649 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=14.39 vs. limit=15.0
2023-11-24 17:24:04,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2924586.6666666665, ans=0.0
2023-11-24 17:24:11,519 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2924653.3333333335, ans=0.0
2023-11-24 17:24:12,471 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438700
2023-11-24 17:24:14,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2924653.3333333335, ans=0.2
2023-11-24 17:24:22,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2924653.3333333335, ans=0.0
2023-11-24 17:24:24,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=2924720.0, ans=0.0
2023-11-24 17:24:25,410 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5850, loss[loss=0.06441, simple_loss=0.08333, pruned_loss=0.01282, audio_tagging_loss=0.009923, over 14810.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09036, pruned_loss=0.013, audio_tagging_loss=0.008775, over 3042841.63 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:24:30,508 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=6.37 vs. limit=12.0
2023-11-24 17:24:37,849 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2924786.6666666665, ans=0.0
2023-11-24 17:24:45,664 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2924786.6666666665, ans=0.0
2023-11-24 17:24:46,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2924786.6666666665, ans=0.2
2023-11-24 17:24:49,574 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.93 vs. limit=22.5
2023-11-24 17:24:50,128 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.433e+01 8.501e+01 8.938e+01 9.729e+01 1.237e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 17:24:51,850 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.70 vs. limit=10.0
2023-11-24 17:24:57,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.20 vs. limit=15.0
2023-11-24 17:25:10,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2924920.0, ans=0.2
2023-11-24 17:25:15,083 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438750
2023-11-24 17:25:20,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2924986.6666666665, ans=0.125
2023-11-24 17:25:27,324 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5900, loss[loss=0.06992, simple_loss=0.09869, pruned_loss=0.01284, audio_tagging_loss=0.00773, over 15058.00 frames. ], tot_loss[loss=0.067, simple_loss=0.09043, pruned_loss=0.01303, audio_tagging_loss=0.008758, over 3042194.62 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:25:42,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2925120.0, ans=0.125
2023-11-24 17:25:54,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2925186.6666666665, ans=0.1
2023-11-24 17:25:58,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2925186.6666666665, ans=0.125
2023-11-24 17:26:06,061 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.08 vs. limit=15.0
2023-11-24 17:26:15,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2925320.0, ans=0.125
2023-11-24 17:26:16,682 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438800
2023-11-24 17:26:19,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2925320.0, ans=0.125
2023-11-24 17:26:29,249 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 5950, loss[loss=0.04927, simple_loss=0.06595, pruned_loss=0.005704, audio_tagging_loss=0.01059, over 14853.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09086, pruned_loss=0.01295, audio_tagging_loss=0.00865, over 3040315.05 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:26:34,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2925386.6666666665, ans=0.0
2023-11-24 17:26:43,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2925453.3333333335, ans=0.125
2023-11-24 17:26:54,497 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.448e+01 8.647e+01 9.220e+01 9.867e+01 1.210e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 17:27:01,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2925520.0, ans=0.1
2023-11-24 17:27:03,780 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.96 vs. limit=8.0
2023-11-24 17:27:18,871 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438850
2023-11-24 17:27:18,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2925653.3333333335, ans=0.125
2023-11-24 17:27:31,684 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6000, loss[loss=0.06679, simple_loss=0.09425, pruned_loss=0.009964, audio_tagging_loss=0.009698, over 14964.00 frames. ], tot_loss[loss=0.0665, simple_loss=0.0901, pruned_loss=0.01274, audio_tagging_loss=0.008713, over 3032414.68 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:27:31,687 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 17:28:14,004 INFO [train_asr.py:1253] (0/4) Epoch 37, validation: loss=0.05829, simple_loss=0.05083, pruned_loss=0.00526, audio_tagging_loss=0.02761, over 4681554.00 frames. 
2023-11-24 17:28:14,004 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 17:28:15,418 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2925720.0, ans=0.5
2023-11-24 17:28:35,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2925786.6666666665, ans=0.125
2023-11-24 17:28:47,399 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=512, metric=18.16 vs. limit=22.5
2023-11-24 17:28:49,537 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.66 vs. limit=10.0
2023-11-24 17:28:59,045 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 17:29:01,550 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2925920.0, ans=0.125
2023-11-24 17:29:03,785 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438900
2023-11-24 17:29:16,134 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6050, loss[loss=0.07115, simple_loss=0.1018, pruned_loss=0.01233, audio_tagging_loss=0.007904, over 15279.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09105, pruned_loss=0.01295, audio_tagging_loss=0.008636, over 3038298.78 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:29:27,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2926120.0, ans=0.125
2023-11-24 17:29:32,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2926120.0, ans=0.2
2023-11-24 17:29:41,480 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.423e+01 8.485e+01 9.097e+01 9.869e+01 1.305e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 17:29:46,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2926186.6666666665, ans=0.1
2023-11-24 17:30:00,145 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2926253.3333333335, ans=0.2
2023-11-24 17:30:06,422 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 438950
2023-11-24 17:30:06,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2926320.0, ans=0.125
2023-11-24 17:30:08,427 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.85 vs. limit=15.0
2023-11-24 17:30:18,685 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6100, loss[loss=0.05791, simple_loss=0.07957, pruned_loss=0.009917, audio_tagging_loss=0.008205, over 15282.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09163, pruned_loss=0.01306, audio_tagging_loss=0.008593, over 3041350.80 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:30:56,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2926586.6666666665, ans=0.0
2023-11-24 17:31:06,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2926586.6666666665, ans=0.125
2023-11-24 17:31:08,403 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439000
2023-11-24 17:31:10,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2926653.3333333335, ans=0.125
2023-11-24 17:31:21,182 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6150, loss[loss=0.04829, simple_loss=0.06494, pruned_loss=0.006583, audio_tagging_loss=0.009239, over 15396.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09038, pruned_loss=0.01302, audio_tagging_loss=0.008669, over 3033858.34 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:31:26,415 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2926720.0, ans=0.2
2023-11-24 17:31:27,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2926720.0, ans=0.1
2023-11-24 17:31:45,660 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.51 vs. limit=10.0
2023-11-24 17:31:46,191 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.974e+01 8.702e+01 9.525e+01 1.018e+02 1.166e+02, threshold=1.905e+02, percent-clipped=0.0
2023-11-24 17:31:53,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2926853.3333333335, ans=0.0
2023-11-24 17:32:08,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2926920.0, ans=0.1
2023-11-24 17:32:10,850 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439050
2023-11-24 17:32:16,806 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2926986.6666666665, ans=0.125
2023-11-24 17:32:19,788 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.71 vs. limit=15.0
2023-11-24 17:32:22,654 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6200, loss[loss=0.06465, simple_loss=0.08586, pruned_loss=0.01439, audio_tagging_loss=0.007336, over 15718.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09078, pruned_loss=0.01305, audio_tagging_loss=0.00879, over 3041059.90 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:32:26,993 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:32:42,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=2927120.0, ans=0.04949747468305833
2023-11-24 17:32:54,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2927186.6666666665, ans=0.125
2023-11-24 17:32:55,987 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.40 vs. limit=15.0
2023-11-24 17:33:12,676 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439100
2023-11-24 17:33:25,879 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6250, loss[loss=0.05988, simple_loss=0.08247, pruned_loss=0.007706, audio_tagging_loss=0.01094, over 16009.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09016, pruned_loss=0.01288, audio_tagging_loss=0.008845, over 3053499.38 frames. ], batch size: 62, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 17:33:46,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2927453.3333333335, ans=0.125
2023-11-24 17:33:50,405 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.139e+01 8.621e+01 9.212e+01 1.010e+02 1.296e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 17:33:56,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2927520.0, ans=0.0
2023-11-24 17:34:15,157 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439150
2023-11-24 17:34:22,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2927653.3333333335, ans=0.95
2023-11-24 17:34:26,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2927720.0, ans=0.1
2023-11-24 17:34:27,449 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6300, loss[loss=0.07478, simple_loss=0.1091, pruned_loss=0.01262, audio_tagging_loss=0.007595, over 15487.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09105, pruned_loss=0.01298, audio_tagging_loss=0.008968, over 3053686.68 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:34:42,088 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.04 vs. limit=15.0
2023-11-24 17:34:53,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2927853.3333333335, ans=0.0
2023-11-24 17:35:06,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2927920.0, ans=0.2
2023-11-24 17:35:11,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2927920.0, ans=0.125
2023-11-24 17:35:17,212 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439200
2023-11-24 17:35:29,268 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6350, loss[loss=0.05889, simple_loss=0.07499, pruned_loss=0.01086, audio_tagging_loss=0.01053, over 16645.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09139, pruned_loss=0.01315, audio_tagging_loss=0.009082, over 3048686.52 frames. ], batch size: 65, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:35:40,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2928053.3333333335, ans=0.125
2023-11-24 17:35:42,838 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.whiten, num_groups=1, num_channels=512, metric=8.61 vs. limit=12.0
2023-11-24 17:35:52,281 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=2928120.0, ans=0.1
2023-11-24 17:35:56,868 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.284e+01 8.531e+01 9.107e+01 9.829e+01 2.915e+02, threshold=1.821e+02, percent-clipped=1.0
2023-11-24 17:35:58,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2928186.6666666665, ans=0.125
2023-11-24 17:35:58,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2928186.6666666665, ans=0.0
2023-11-24 17:36:17,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2928320.0, ans=0.125
2023-11-24 17:36:18,733 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439250
2023-11-24 17:36:26,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=2928320.0, ans=0.0
2023-11-24 17:36:29,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2928320.0, ans=0.09899494936611666
2023-11-24 17:36:31,545 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6400, loss[loss=0.07113, simple_loss=0.09758, pruned_loss=0.01011, audio_tagging_loss=0.01224, over 15634.00 frames. ], tot_loss[loss=0.06849, simple_loss=0.09221, pruned_loss=0.01321, audio_tagging_loss=0.009172, over 3046590.75 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:36:48,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2928453.3333333335, ans=0.125
2023-11-24 17:36:49,605 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.01 vs. limit=15.0
2023-11-24 17:36:53,224 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.81 vs. limit=15.0
2023-11-24 17:36:58,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.99 vs. limit=6.0
2023-11-24 17:37:03,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=2928520.0, ans=0.2
2023-11-24 17:37:13,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.00 vs. limit=10.0
2023-11-24 17:37:17,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2928586.6666666665, ans=0.0
2023-11-24 17:37:21,568 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439300
2023-11-24 17:37:21,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2928653.3333333335, ans=0.0
2023-11-24 17:37:33,832 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6450, loss[loss=0.06036, simple_loss=0.07366, pruned_loss=0.01247, audio_tagging_loss=0.01106, over 14775.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09094, pruned_loss=0.01298, audio_tagging_loss=0.00926, over 3039802.01 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:37:35,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2928720.0, ans=0.025
2023-11-24 17:37:43,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2928720.0, ans=0.125
2023-11-24 17:37:44,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2928786.6666666665, ans=0.0
2023-11-24 17:38:00,168 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.738e+01 8.603e+01 9.218e+01 1.012e+02 1.215e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 17:38:00,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2928853.3333333335, ans=0.125
2023-11-24 17:38:06,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2928853.3333333335, ans=0.0
2023-11-24 17:38:19,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2928920.0, ans=0.2
2023-11-24 17:38:20,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2928920.0, ans=0.2
2023-11-24 17:38:22,516 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439350
2023-11-24 17:38:29,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2928986.6666666665, ans=0.0
2023-11-24 17:38:34,349 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6500, loss[loss=0.05875, simple_loss=0.08118, pruned_loss=0.008323, audio_tagging_loss=0.009844, over 16514.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09035, pruned_loss=0.01282, audio_tagging_loss=0.009187, over 3046912.20 frames. ], batch size: 63, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:38:48,268 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2929120.0, ans=0.1
2023-11-24 17:38:50,017 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.84 vs. limit=15.0
2023-11-24 17:39:06,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2929186.6666666665, ans=0.0
2023-11-24 17:39:18,772 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.85 vs. limit=10.0
2023-11-24 17:39:24,217 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439400
2023-11-24 17:39:25,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2929320.0, ans=0.125
2023-11-24 17:39:36,750 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6550, loss[loss=0.06121, simple_loss=0.079, pruned_loss=0.01134, audio_tagging_loss=0.01037, over 15372.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.0905, pruned_loss=0.0131, audio_tagging_loss=0.009035, over 3046972.24 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:39:51,292 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:39:52,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2929453.3333333335, ans=0.1
2023-11-24 17:39:55,728 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.11 vs. limit=22.5
2023-11-24 17:40:05,707 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.142e+01 8.772e+01 9.362e+01 9.895e+01 1.833e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-24 17:40:11,077 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.98 vs. limit=15.0
2023-11-24 17:40:26,640 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439450
2023-11-24 17:40:37,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2929653.3333333335, ans=0.1
2023-11-24 17:40:39,584 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6600, loss[loss=0.06414, simple_loss=0.08487, pruned_loss=0.01167, audio_tagging_loss=0.01003, over 16508.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.0911, pruned_loss=0.01323, audio_tagging_loss=0.008869, over 3044914.05 frames. ], batch size: 62, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:40:43,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2929720.0, ans=0.125
2023-11-24 17:40:50,286 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.93 vs. limit=22.5
2023-11-24 17:40:54,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2929786.6666666665, ans=0.0
2023-11-24 17:40:56,037 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.02 vs. limit=12.0
2023-11-24 17:41:22,373 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2929920.0, ans=0.125
2023-11-24 17:41:29,662 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439500
2023-11-24 17:41:38,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2929986.6666666665, ans=0.0
2023-11-24 17:41:41,330 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6650, loss[loss=0.05722, simple_loss=0.07971, pruned_loss=0.009398, audio_tagging_loss=0.007963, over 15525.00 frames. ], tot_loss[loss=0.06716, simple_loss=0.09024, pruned_loss=0.01312, audio_tagging_loss=0.008919, over 3041608.01 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:41:42,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2930053.3333333335, ans=0.125
2023-11-24 17:41:47,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2930053.3333333335, ans=0.125
2023-11-24 17:42:10,023 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.316e+01 8.532e+01 9.137e+01 1.001e+02 1.205e+02, threshold=1.827e+02, percent-clipped=0.0
2023-11-24 17:42:31,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439550
2023-11-24 17:42:36,691 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2930320.0, ans=0.125
2023-11-24 17:42:38,973 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2930320.0, ans=0.2
2023-11-24 17:42:40,742 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2930320.0, ans=0.0
2023-11-24 17:42:44,160 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6700, loss[loss=0.06593, simple_loss=0.09343, pruned_loss=0.01283, audio_tagging_loss=0.006387, over 14635.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09086, pruned_loss=0.01324, audio_tagging_loss=0.008778, over 3045347.08 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:42:44,818 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.77 vs. limit=15.0
2023-11-24 17:43:13,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2930520.0, ans=0.125
2023-11-24 17:43:19,736 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.99 vs. limit=10.0
2023-11-24 17:43:24,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2930586.6666666665, ans=0.125
2023-11-24 17:43:33,927 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439600
2023-11-24 17:43:34,249 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:43:45,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0
2023-11-24 17:43:46,507 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6750, loss[loss=0.06625, simple_loss=0.09081, pruned_loss=0.01113, audio_tagging_loss=0.009722, over 14592.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09053, pruned_loss=0.01301, audio_tagging_loss=0.008833, over 3037652.49 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 8.0
2023-11-24 17:44:10,843 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.72 vs. limit=15.0
2023-11-24 17:44:15,654 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.778e+01 8.260e+01 8.901e+01 9.551e+01 1.159e+02, threshold=1.780e+02, percent-clipped=0.0
2023-11-24 17:44:15,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2930853.3333333335, ans=0.07
2023-11-24 17:44:23,332 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2930920.0, ans=0.2
2023-11-24 17:44:26,778 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.71 vs. limit=15.0
2023-11-24 17:44:35,216 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.12 vs. limit=15.0
2023-11-24 17:44:37,017 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439650
2023-11-24 17:44:49,445 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6800, loss[loss=0.06983, simple_loss=0.08977, pruned_loss=0.01309, audio_tagging_loss=0.01185, over 15859.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09013, pruned_loss=0.01291, audio_tagging_loss=0.008834, over 3040203.51 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:45:08,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2931120.0, ans=0.125
2023-11-24 17:45:14,999 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.76 vs. limit=6.0
2023-11-24 17:45:38,622 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2931320.0, ans=0.125
2023-11-24 17:45:39,655 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439700
2023-11-24 17:45:50,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2931386.6666666665, ans=0.035
2023-11-24 17:45:51,544 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.02 vs. limit=15.0
2023-11-24 17:45:51,922 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6850, loss[loss=0.04985, simple_loss=0.06618, pruned_loss=0.008626, audio_tagging_loss=0.008132, over 15733.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09032, pruned_loss=0.01294, audio_tagging_loss=0.008834, over 3041294.94 frames. ], batch size: 63, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:46:15,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2931453.3333333335, ans=0.125
2023-11-24 17:46:20,460 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2931520.0, ans=0.2
2023-11-24 17:46:21,181 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.185e+01 8.495e+01 8.934e+01 9.864e+01 1.145e+02, threshold=1.787e+02, percent-clipped=0.0
2023-11-24 17:46:42,316 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439750
2023-11-24 17:46:43,740 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2931653.3333333335, ans=0.125
2023-11-24 17:46:55,045 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6900, loss[loss=0.06229, simple_loss=0.08756, pruned_loss=0.009463, audio_tagging_loss=0.009047, over 16285.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.09001, pruned_loss=0.0128, audio_tagging_loss=0.008875, over 3043752.45 frames. ], batch size: 61, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:47:07,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2931786.6666666665, ans=0.2
2023-11-24 17:47:11,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2931786.6666666665, ans=0.0
2023-11-24 17:47:14,947 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2931786.6666666665, ans=0.0
2023-11-24 17:47:19,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2931853.3333333335, ans=0.07
2023-11-24 17:47:20,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2931853.3333333335, ans=0.0
2023-11-24 17:47:33,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2931920.0, ans=0.1
2023-11-24 17:47:40,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2931920.0, ans=0.125
2023-11-24 17:47:42,856 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 17:47:45,325 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439800
2023-11-24 17:47:45,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2931986.6666666665, ans=0.1
2023-11-24 17:47:51,167 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2931986.6666666665, ans=0.1
2023-11-24 17:47:56,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2932053.3333333335, ans=0.0
2023-11-24 17:47:58,609 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 6950, loss[loss=0.07813, simple_loss=0.1041, pruned_loss=0.01797, audio_tagging_loss=0.008114, over 15227.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.0904, pruned_loss=0.01288, audio_tagging_loss=0.008816, over 3043685.39 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:48:01,243 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2932053.3333333335, ans=0.0
2023-11-24 17:48:01,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2932053.3333333335, ans=0.125
2023-11-24 17:48:13,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=13.46 vs. limit=15.0
2023-11-24 17:48:18,526 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2932120.0, ans=0.125
2023-11-24 17:48:27,211 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.428e+01 8.692e+01 9.069e+01 1.003e+02 1.264e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 17:48:30,058 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.72 vs. limit=10.0
2023-11-24 17:48:44,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2932253.3333333335, ans=0.0
2023-11-24 17:48:48,752 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439850
2023-11-24 17:48:51,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2932320.0, ans=0.125
2023-11-24 17:49:00,507 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7000, loss[loss=0.07052, simple_loss=0.09802, pruned_loss=0.01242, audio_tagging_loss=0.00909, over 13761.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.09003, pruned_loss=0.01272, audio_tagging_loss=0.008853, over 3043758.23 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:49:18,451 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2932453.3333333335, ans=0.0
2023-11-24 17:49:45,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2932586.6666666665, ans=0.2
2023-11-24 17:49:47,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.19 vs. limit=22.5
2023-11-24 17:49:51,113 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439900
2023-11-24 17:49:53,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2932653.3333333335, ans=0.0
2023-11-24 17:49:57,395 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=2.523e-03
2023-11-24 17:50:03,454 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7050, loss[loss=0.05453, simple_loss=0.06474, pruned_loss=0.0118, audio_tagging_loss=0.01036, over 14253.00 frames. ], tot_loss[loss=0.06677, simple_loss=0.09038, pruned_loss=0.01272, audio_tagging_loss=0.008865, over 3040727.09 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:50:11,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2932720.0, ans=0.0
2023-11-24 17:50:16,788 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.44 vs. limit=22.5
2023-11-24 17:50:25,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2932786.6666666665, ans=0.125
2023-11-24 17:50:31,914 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.508e+01 9.071e+01 9.763e+01 1.227e+02, threshold=1.814e+02, percent-clipped=0.0
2023-11-24 17:50:32,260 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:50:42,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2932920.0, ans=0.2
2023-11-24 17:50:49,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2932920.0, ans=0.1
2023-11-24 17:50:50,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.79 vs. limit=15.0
2023-11-24 17:50:52,824 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.42 vs. limit=15.0
2023-11-24 17:50:53,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 439950
2023-11-24 17:50:53,891 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.06 vs. limit=15.0
2023-11-24 17:51:05,589 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7100, loss[loss=0.06065, simple_loss=0.08122, pruned_loss=0.01082, audio_tagging_loss=0.009215, over 15172.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.09027, pruned_loss=0.01269, audio_tagging_loss=0.008869, over 3039155.24 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:51:11,379 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.07 vs. limit=15.0
2023-11-24 17:51:14,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2933053.3333333335, ans=0.125
2023-11-24 17:51:34,764 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2933186.6666666665, ans=0.125
2023-11-24 17:51:35,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2933186.6666666665, ans=0.0
2023-11-24 17:51:45,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2933253.3333333335, ans=0.0
2023-11-24 17:51:55,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440000
2023-11-24 17:51:57,074 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-440000.pt
2023-11-24 17:52:05,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2933320.0, ans=0.125
2023-11-24 17:52:12,088 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7150, loss[loss=0.0695, simple_loss=0.09582, pruned_loss=0.01099, audio_tagging_loss=0.0106, over 14946.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09053, pruned_loss=0.01282, audio_tagging_loss=0.008954, over 3040967.57 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:52:21,172 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_abs, batch_count=2933386.6666666665, ans=0.5
2023-11-24 17:52:40,900 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.362e+01 8.675e+01 9.215e+01 9.966e+01 1.271e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 17:52:43,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2933520.0, ans=0.5
2023-11-24 17:52:47,691 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.38 vs. limit=15.0
2023-11-24 17:52:58,727 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.72 vs. limit=15.0
2023-11-24 17:53:01,805 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440050
2023-11-24 17:53:07,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2933653.3333333335, ans=0.0
2023-11-24 17:53:14,521 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7200, loss[loss=0.05772, simple_loss=0.07198, pruned_loss=0.01114, audio_tagging_loss=0.01059, over 14892.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.09046, pruned_loss=0.01292, audio_tagging_loss=0.008937, over 3044139.00 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:53:14,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2933720.0, ans=0.0
2023-11-24 17:53:15,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2933720.0, ans=0.1
2023-11-24 17:53:24,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2933720.0, ans=0.07
2023-11-24 17:53:34,463 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2933786.6666666665, ans=0.125
2023-11-24 17:53:49,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2933853.3333333335, ans=0.0
2023-11-24 17:53:50,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2933920.0, ans=0.0
2023-11-24 17:54:03,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2933986.6666666665, ans=0.1
2023-11-24 17:54:04,232 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440100
2023-11-24 17:54:10,907 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=10.28 vs. limit=12.0
2023-11-24 17:54:16,753 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7250, loss[loss=0.04879, simple_loss=0.06831, pruned_loss=0.004197, audio_tagging_loss=0.01044, over 15921.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.08989, pruned_loss=0.0128, audio_tagging_loss=0.008974, over 3049174.91 frames. ], batch size: 60, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:54:47,444 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.141e+01 8.488e+01 9.107e+01 9.916e+01 1.399e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 17:55:06,894 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440150
2023-11-24 17:55:18,553 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7300, loss[loss=0.05372, simple_loss=0.07402, pruned_loss=0.008392, audio_tagging_loss=0.008322, over 15166.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09079, pruned_loss=0.01299, audio_tagging_loss=0.008936, over 3052978.48 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:55:27,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2934386.6666666665, ans=0.1
2023-11-24 17:55:27,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2934386.6666666665, ans=0.125
2023-11-24 17:55:29,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2934386.6666666665, ans=0.07
2023-11-24 17:55:33,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2934453.3333333335, ans=0.0
2023-11-24 17:56:01,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2934586.6666666665, ans=0.125
2023-11-24 17:56:09,191 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440200
2023-11-24 17:56:22,501 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7350, loss[loss=0.04754, simple_loss=0.06008, pruned_loss=0.008468, audio_tagging_loss=0.009032, over 15236.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09074, pruned_loss=0.01294, audio_tagging_loss=0.008876, over 3051134.25 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:56:26,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2934720.0, ans=0.125
2023-11-24 17:56:26,208 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=2934720.0, ans=10.0
2023-11-24 17:56:51,151 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.056e+01 8.582e+01 8.969e+01 9.550e+01 1.265e+02, threshold=1.794e+02, percent-clipped=0.0
2023-11-24 17:57:04,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2934920.0, ans=0.125
2023-11-24 17:57:11,770 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440250
2023-11-24 17:57:24,027 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7400, loss[loss=0.05917, simple_loss=0.07569, pruned_loss=0.01161, audio_tagging_loss=0.009714, over 15891.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09062, pruned_loss=0.01286, audio_tagging_loss=0.008844, over 3044364.28 frames. ], batch size: 62, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:57:32,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2935053.3333333335, ans=0.1
2023-11-24 17:57:33,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2935053.3333333335, ans=0.125
2023-11-24 17:57:35,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2935120.0, ans=0.95
2023-11-24 17:57:43,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2935120.0, ans=0.0
2023-11-24 17:57:54,940 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2935186.6666666665, ans=0.0
2023-11-24 17:57:58,952 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2935186.6666666665, ans=0.2
2023-11-24 17:58:14,557 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440300
2023-11-24 17:58:26,490 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7450, loss[loss=0.07308, simple_loss=0.1063, pruned_loss=0.01322, audio_tagging_loss=0.006716, over 16418.00 frames. ], tot_loss[loss=0.06665, simple_loss=0.09021, pruned_loss=0.01283, audio_tagging_loss=0.008714, over 3051051.23 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:58:55,993 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 17:58:56,703 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.405e+01 8.534e+01 9.145e+01 9.758e+01 1.407e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 17:59:06,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2935586.6666666665, ans=0.05
2023-11-24 17:59:11,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2935586.6666666665, ans=0.125
2023-11-24 17:59:12,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2935586.6666666665, ans=0.125
2023-11-24 17:59:16,384 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440350
2023-11-24 17:59:17,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2935653.3333333335, ans=0.125
2023-11-24 17:59:20,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2935653.3333333335, ans=0.125
2023-11-24 17:59:28,791 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7500, loss[loss=0.07324, simple_loss=0.1064, pruned_loss=0.01301, audio_tagging_loss=0.007022, over 15081.00 frames. ], tot_loss[loss=0.06632, simple_loss=0.08972, pruned_loss=0.01279, audio_tagging_loss=0.008679, over 3050716.20 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 17:59:32,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2935720.0, ans=0.0
2023-11-24 17:59:34,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2935720.0, ans=0.125
2023-11-24 17:59:44,265 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.46 vs. limit=6.0
2023-11-24 17:59:50,949 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=16.31 vs. limit=15.0
2023-11-24 18:00:01,082 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:00:15,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2935920.0, ans=0.1
2023-11-24 18:00:18,936 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440400
2023-11-24 18:00:31,114 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7550, loss[loss=0.04577, simple_loss=0.05446, pruned_loss=0.008118, audio_tagging_loss=0.01042, over 16117.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.09012, pruned_loss=0.01283, audio_tagging_loss=0.008729, over 3053982.62 frames. ], batch size: 63, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:00:32,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2936053.3333333335, ans=0.07
2023-11-24 18:00:35,474 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2936053.3333333335, ans=0.125
2023-11-24 18:00:52,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2936120.0, ans=0.0
2023-11-24 18:00:57,777 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2936186.6666666665, ans=0.0
2023-11-24 18:01:00,296 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.354e+01 8.524e+01 9.162e+01 9.767e+01 1.233e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 18:01:09,285 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.77 vs. limit=15.0
2023-11-24 18:01:13,934 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.17 vs. limit=15.0
2023-11-24 18:01:15,990 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2936253.3333333335, ans=0.125
2023-11-24 18:01:21,336 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440450
2023-11-24 18:01:24,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2936320.0, ans=0.0
2023-11-24 18:01:32,056 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2936386.6666666665, ans=0.2
2023-11-24 18:01:32,931 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7600, loss[loss=0.0807, simple_loss=0.1097, pruned_loss=0.01666, audio_tagging_loss=0.009193, over 15232.00 frames. ], tot_loss[loss=0.06651, simple_loss=0.0898, pruned_loss=0.01287, audio_tagging_loss=0.008738, over 3049527.65 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:01:42,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2936386.6666666665, ans=0.2
2023-11-24 18:02:18,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2936586.6666666665, ans=0.125
2023-11-24 18:02:23,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440500
2023-11-24 18:02:24,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2936653.3333333335, ans=0.125
2023-11-24 18:02:29,376 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2936653.3333333335, ans=0.0
2023-11-24 18:02:30,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2936653.3333333335, ans=0.125
2023-11-24 18:02:35,612 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7650, loss[loss=0.07422, simple_loss=0.1086, pruned_loss=0.01395, audio_tagging_loss=0.005963, over 14858.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.08975, pruned_loss=0.01287, audio_tagging_loss=0.008702, over 3044277.53 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:02:36,234 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.34 vs. limit=15.0
2023-11-24 18:02:41,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2936720.0, ans=0.125
2023-11-24 18:02:54,770 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2936786.6666666665, ans=0.2
2023-11-24 18:02:58,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2936786.6666666665, ans=0.0
2023-11-24 18:03:05,430 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.945e+01 8.749e+01 9.173e+01 9.808e+01 1.366e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-24 18:03:19,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2936920.0, ans=0.5
2023-11-24 18:03:23,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=2936920.0, ans=0.025
2023-11-24 18:03:25,537 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440550
2023-11-24 18:03:32,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2936986.6666666665, ans=10.0
2023-11-24 18:03:35,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2936986.6666666665, ans=0.0
2023-11-24 18:03:37,741 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7700, loss[loss=0.09997, simple_loss=0.1339, pruned_loss=0.02458, audio_tagging_loss=0.008412, over 15004.00 frames. ], tot_loss[loss=0.06622, simple_loss=0.08925, pruned_loss=0.01284, audio_tagging_loss=0.008752, over 3046414.25 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:03:53,892 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2937120.0, ans=0.0
2023-11-24 18:04:12,300 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2937186.6666666665, ans=0.125
2023-11-24 18:04:16,730 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2937253.3333333335, ans=0.2
2023-11-24 18:04:27,259 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440600
2023-11-24 18:04:27,517 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2937320.0, ans=0.125
2023-11-24 18:04:39,838 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7750, loss[loss=0.07335, simple_loss=0.1036, pruned_loss=0.0143, audio_tagging_loss=0.007268, over 14530.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.09014, pruned_loss=0.01301, audio_tagging_loss=0.008777, over 3043497.49 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:04:41,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2937386.6666666665, ans=0.125
2023-11-24 18:04:54,505 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2937453.3333333335, ans=0.2
2023-11-24 18:05:01,094 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2937453.3333333335, ans=0.125
2023-11-24 18:05:09,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2937520.0, ans=0.1
2023-11-24 18:05:10,506 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.466e+01 8.710e+01 9.479e+01 9.926e+01 1.240e+02, threshold=1.896e+02, percent-clipped=0.0
2023-11-24 18:05:30,279 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440650
2023-11-24 18:05:34,024 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2937653.3333333335, ans=0.2
2023-11-24 18:05:37,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2937653.3333333335, ans=0.1
2023-11-24 18:05:42,085 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7800, loss[loss=0.04765, simple_loss=0.06464, pruned_loss=0.004058, audio_tagging_loss=0.01127, over 14948.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09117, pruned_loss=0.01304, audio_tagging_loss=0.008733, over 3041445.23 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:05:46,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2937720.0, ans=0.125
2023-11-24 18:05:48,230 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-24 18:05:55,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2937786.6666666665, ans=0.0
2023-11-24 18:05:57,701 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.70 vs. limit=5.0
2023-11-24 18:06:03,352 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.57 vs. limit=6.0
2023-11-24 18:06:16,538 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.63 vs. limit=15.0
2023-11-24 18:06:17,394 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2937853.3333333335, ans=0.1
2023-11-24 18:06:17,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.skip_rate, batch_count=2937853.3333333335, ans=0.07
2023-11-24 18:06:22,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2937920.0, ans=0.1
2023-11-24 18:06:32,231 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440700
2023-11-24 18:06:37,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2937986.6666666665, ans=0.125
2023-11-24 18:06:42,456 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2937986.6666666665, ans=0.0
2023-11-24 18:06:45,223 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7850, loss[loss=0.07676, simple_loss=0.105, pruned_loss=0.01575, audio_tagging_loss=0.008521, over 14457.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09125, pruned_loss=0.01315, audio_tagging_loss=0.008776, over 3042491.21 frames. ], batch size: 56, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:06:49,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2938053.3333333335, ans=0.0
2023-11-24 18:07:00,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2938120.0, ans=0.125
2023-11-24 18:07:14,183 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.363e+01 8.824e+01 9.632e+01 1.042e+02 1.245e+02, threshold=1.926e+02, percent-clipped=0.0
2023-11-24 18:07:26,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2938253.3333333335, ans=0.1
2023-11-24 18:07:34,888 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440750
2023-11-24 18:07:40,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.20 vs. limit=6.0
2023-11-24 18:07:44,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten.whitening_limit, batch_count=2938320.0, ans=15.0
2023-11-24 18:07:45,192 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer2.prob, batch_count=2938320.0, ans=0.125
2023-11-24 18:07:47,158 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7900, loss[loss=0.07899, simple_loss=0.1035, pruned_loss=0.01707, audio_tagging_loss=0.01017, over 15366.00 frames. ], tot_loss[loss=0.06729, simple_loss=0.09069, pruned_loss=0.01304, audio_tagging_loss=0.008904, over 3051803.35 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:07:49,991 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2938386.6666666665, ans=0.125
2023-11-24 18:07:58,107 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2938453.3333333335, ans=0.125
2023-11-24 18:08:14,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2938520.0, ans=0.125
2023-11-24 18:08:16,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=2938520.0, ans=6.0
2023-11-24 18:08:16,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2938520.0, ans=0.125
2023-11-24 18:08:37,013 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440800
2023-11-24 18:08:48,998 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 7950, loss[loss=0.08599, simple_loss=0.1155, pruned_loss=0.01728, audio_tagging_loss=0.01094, over 15946.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09077, pruned_loss=0.01294, audio_tagging_loss=0.009023, over 3051861.85 frames. ], batch size: 58, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:09:04,171 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:09:06,711 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2938786.6666666665, ans=0.1
2023-11-24 18:09:20,729 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.553e+01 8.522e+01 9.138e+01 9.780e+01 1.184e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 18:09:38,597 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440850
2023-11-24 18:09:45,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2938986.6666666665, ans=0.125
2023-11-24 18:09:49,350 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2938986.6666666665, ans=0.0
2023-11-24 18:09:51,418 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8000, loss[loss=0.07448, simple_loss=0.09875, pruned_loss=0.01682, audio_tagging_loss=0.008295, over 14134.00 frames. ], tot_loss[loss=0.06688, simple_loss=0.0896, pruned_loss=0.01292, audio_tagging_loss=0.009158, over 3039675.59 frames. ], batch size: 55, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:09:52,075 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.18 vs. limit=15.0
2023-11-24 18:10:16,718 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=2939186.6666666665, ans=0.09899494936611666
2023-11-24 18:10:35,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.88 vs. limit=15.0
2023-11-24 18:10:41,051 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440900
2023-11-24 18:10:44,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2939320.0, ans=0.0
2023-11-24 18:10:49,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2939320.0, ans=0.125
2023-11-24 18:10:53,882 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8050, loss[loss=0.07279, simple_loss=0.1028, pruned_loss=0.01436, audio_tagging_loss=0.007026, over 16049.00 frames. ], tot_loss[loss=0.06709, simple_loss=0.08986, pruned_loss=0.013, audio_tagging_loss=0.009154, over 3040662.93 frames. ], batch size: 57, lr: 1.83e-03, grad_scale: 32.0
2023-11-24 18:10:59,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2939386.6666666665, ans=0.125
2023-11-24 18:11:18,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2939520.0, ans=0.1
2023-11-24 18:11:19,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2939520.0, ans=0.1
2023-11-24 18:11:23,338 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2939520.0, ans=0.0
2023-11-24 18:11:25,411 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.434e+01 8.492e+01 9.262e+01 1.003e+02 1.241e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 18:11:25,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=2939520.0, ans=0.09899494936611666
2023-11-24 18:11:26,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=2939520.0, ans=0.025
2023-11-24 18:11:28,306 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.86 vs. limit=6.0
2023-11-24 18:11:37,728 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=13.19 vs. limit=15.0
2023-11-24 18:11:43,195 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 440950
2023-11-24 18:11:54,750 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8100, loss[loss=0.07358, simple_loss=0.1052, pruned_loss=0.01412, audio_tagging_loss=0.006879, over 16473.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09038, pruned_loss=0.01314, audio_tagging_loss=0.009199, over 3049074.00 frames. ], batch size: 62, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:11:55,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2939720.0, ans=0.0
2023-11-24 18:12:06,076 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2939786.6666666665, ans=0.125
2023-11-24 18:12:07,822 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=2939786.6666666665, ans=0.125
2023-11-24 18:12:15,825 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2939786.6666666665, ans=0.07
2023-11-24 18:12:17,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2939786.6666666665, ans=0.125
2023-11-24 18:12:34,805 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:12:44,272 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441000
2023-11-24 18:12:46,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2939986.6666666665, ans=0.0
2023-11-24 18:12:57,406 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8150, loss[loss=0.07565, simple_loss=0.1, pruned_loss=0.01837, audio_tagging_loss=0.007266, over 14666.00 frames. ], tot_loss[loss=0.06784, simple_loss=0.09128, pruned_loss=0.01324, audio_tagging_loss=0.008963, over 3053994.21 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:12:59,133 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.85 vs. limit=15.0
2023-11-24 18:13:05,152 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.91 vs. limit=15.0
2023-11-24 18:13:18,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2940120.0, ans=0.125
2023-11-24 18:13:29,020 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.802e+01 9.331e+01 9.892e+01 1.682e+02, threshold=1.866e+02, percent-clipped=0.0
2023-11-24 18:13:40,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2940253.3333333335, ans=0.0
2023-11-24 18:13:46,739 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441050
2023-11-24 18:13:59,373 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8200, loss[loss=0.0623, simple_loss=0.0842, pruned_loss=0.01012, audio_tagging_loss=0.01009, over 15571.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09073, pruned_loss=0.01304, audio_tagging_loss=0.008935, over 3055665.17 frames. ], batch size: 59, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:13:59,410 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:14:04,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2940386.6666666665, ans=0.125
2023-11-24 18:14:07,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2940386.6666666665, ans=0.125
2023-11-24 18:14:11,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2940453.3333333335, ans=0.2
2023-11-24 18:14:34,369 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2940520.0, ans=0.0
2023-11-24 18:14:48,901 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441100
2023-11-24 18:14:56,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2940653.3333333335, ans=0.0
2023-11-24 18:14:59,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=2940653.3333333335, ans=0.125
2023-11-24 18:15:01,163 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8250, loss[loss=0.07406, simple_loss=0.1056, pruned_loss=0.01489, audio_tagging_loss=0.006365, over 14842.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09054, pruned_loss=0.01302, audio_tagging_loss=0.008904, over 3040343.43 frames. ], batch size: 53, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:15:01,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2940720.0, ans=0.2
2023-11-24 18:15:33,535 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.363e+01 8.417e+01 9.119e+01 9.803e+01 1.778e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 18:15:50,720 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441150
2023-11-24 18:15:52,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2940986.6666666665, ans=0.125
2023-11-24 18:16:03,870 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8300, loss[loss=0.05741, simple_loss=0.07439, pruned_loss=0.01018, audio_tagging_loss=0.01002, over 13774.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.09012, pruned_loss=0.0129, audio_tagging_loss=0.00886, over 3035046.61 frames. ], batch size: 54, lr: 1.83e-03, grad_scale: 16.0
2023-11-24 18:16:09,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2941053.3333333335, ans=0.125
2023-11-24 18:16:29,102 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.47 vs. limit=15.0
2023-11-24 18:16:29,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2941186.6666666665, ans=0.05
2023-11-24 18:16:38,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2941186.6666666665, ans=0.1
2023-11-24 18:16:54,696 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441200
2023-11-24 18:16:54,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=2941320.0, ans=0.125
2023-11-24 18:17:07,486 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8350, loss[loss=0.0708, simple_loss=0.1048, pruned_loss=0.009732, audio_tagging_loss=0.00869, over 15411.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.09028, pruned_loss=0.01293, audio_tagging_loss=0.008757, over 3042053.62 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:17:07,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2941386.6666666665, ans=0.0
2023-11-24 18:17:09,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2941386.6666666665, ans=0.125
2023-11-24 18:17:15,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2941386.6666666665, ans=0.0
2023-11-24 18:17:22,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2941453.3333333335, ans=0.125
2023-11-24 18:17:32,769 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2941520.0, ans=0.0
2023-11-24 18:17:37,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=2941520.0, ans=15.0
2023-11-24 18:17:40,294 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.269e+01 8.581e+01 9.277e+01 1.007e+02 1.908e+02, threshold=1.855e+02, percent-clipped=1.0
2023-11-24 18:17:57,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441250
2023-11-24 18:18:05,994 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2941653.3333333335, ans=0.125
2023-11-24 18:18:06,311 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.82 vs. limit=10.0
2023-11-24 18:18:08,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2941720.0, ans=0.1
2023-11-24 18:18:09,942 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8400, loss[loss=0.05325, simple_loss=0.06906, pruned_loss=0.01043, audio_tagging_loss=0.008291, over 16021.00 frames. ], tot_loss[loss=0.06649, simple_loss=0.08988, pruned_loss=0.01283, audio_tagging_loss=0.00872, over 3037186.12 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:18:12,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=2941720.0, ans=0.125
2023-11-24 18:18:36,069 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2941853.3333333335, ans=0.2
2023-11-24 18:18:36,314 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=8.36 vs. limit=15.0
2023-11-24 18:18:41,617 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.58 vs. limit=15.0
2023-11-24 18:18:59,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441300
2023-11-24 18:19:11,673 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8450, loss[loss=0.05386, simple_loss=0.07115, pruned_loss=0.01148, audio_tagging_loss=0.006811, over 14625.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.09051, pruned_loss=0.01286, audio_tagging_loss=0.00872, over 3044900.06 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:19:28,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2942120.0, ans=0.2
2023-11-24 18:19:28,492 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=2942120.0, ans=0.125
2023-11-24 18:19:43,713 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.476e+01 8.739e+01 9.324e+01 1.024e+02 1.265e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 18:20:02,306 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441350
2023-11-24 18:20:13,962 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8500, loss[loss=0.07371, simple_loss=0.101, pruned_loss=0.01471, audio_tagging_loss=0.008486, over 14152.00 frames. ], tot_loss[loss=0.06674, simple_loss=0.09033, pruned_loss=0.0128, audio_tagging_loss=0.008774, over 3046839.92 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:20:17,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2942386.6666666665, ans=0.2
2023-11-24 18:20:19,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2942386.6666666665, ans=0.125
2023-11-24 18:20:27,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2942453.3333333335, ans=0.0
2023-11-24 18:20:37,201 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2942453.3333333335, ans=0.125
2023-11-24 18:20:49,861 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2942520.0, ans=0.2
2023-11-24 18:20:55,681 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.17 vs. limit=15.0
2023-11-24 18:21:05,370 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441400
2023-11-24 18:21:07,229 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2942653.3333333335, ans=0.2
2023-11-24 18:21:17,646 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8550, loss[loss=0.06169, simple_loss=0.08187, pruned_loss=0.01186, audio_tagging_loss=0.008899, over 13794.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09055, pruned_loss=0.01285, audio_tagging_loss=0.008764, over 3039334.32 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:21:20,833 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=14.41 vs. limit=15.0
2023-11-24 18:21:31,354 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.09 vs. limit=22.5
2023-11-24 18:21:34,543 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2942786.6666666665, ans=0.125
2023-11-24 18:21:35,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=7.21 vs. limit=8.0
2023-11-24 18:21:52,034 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.190e+01 8.580e+01 9.059e+01 9.638e+01 1.247e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 18:22:07,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441450
2023-11-24 18:22:21,128 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8600, loss[loss=0.08394, simple_loss=0.1115, pruned_loss=0.01583, audio_tagging_loss=0.01234, over 15510.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09044, pruned_loss=0.01269, audio_tagging_loss=0.008902, over 3038357.02 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:22:41,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2943120.0, ans=0.125
2023-11-24 18:22:52,193 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2943186.6666666665, ans=0.2
2023-11-24 18:23:09,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2943253.3333333335, ans=0.125
2023-11-24 18:23:11,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.min_positive, batch_count=2943320.0, ans=0.025
2023-11-24 18:23:12,945 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441500
2023-11-24 18:23:25,580 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8650, loss[loss=0.0608, simple_loss=0.07836, pruned_loss=0.01356, audio_tagging_loss=0.008063, over 15386.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09191, pruned_loss=0.01282, audio_tagging_loss=0.008906, over 3050658.00 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:23:38,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2943453.3333333335, ans=0.0
2023-11-24 18:23:41,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2943453.3333333335, ans=0.1
2023-11-24 18:23:59,836 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.252e+01 8.635e+01 9.102e+01 9.726e+01 1.317e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 18:24:10,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2943586.6666666665, ans=0.0
2023-11-24 18:24:16,897 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441550
2023-11-24 18:24:26,723 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:24:27,821 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2943720.0, ans=0.125
2023-11-24 18:24:28,907 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8700, loss[loss=0.07958, simple_loss=0.1095, pruned_loss=0.01567, audio_tagging_loss=0.009185, over 14501.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09162, pruned_loss=0.01274, audio_tagging_loss=0.008891, over 3049384.69 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:24:51,346 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.81 vs. limit=15.0
2023-11-24 18:25:11,133 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2943920.0, ans=0.125
2023-11-24 18:25:19,327 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441600
2023-11-24 18:25:31,364 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8750, loss[loss=0.06603, simple_loss=0.0845, pruned_loss=0.01422, audio_tagging_loss=0.009554, over 16547.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09144, pruned_loss=0.0128, audio_tagging_loss=0.009016, over 3050433.13 frames. ], batch size: 62, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:25:38,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2944053.3333333335, ans=0.0
2023-11-24 18:25:45,452 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=2944120.0, ans=0.2
2023-11-24 18:26:05,772 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.362e+01 8.931e+01 9.559e+01 1.034e+02 1.434e+02, threshold=1.912e+02, percent-clipped=0.0
2023-11-24 18:26:08,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2944253.3333333335, ans=0.125
2023-11-24 18:26:17,421 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=8.65 vs. limit=15.0
2023-11-24 18:26:22,248 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441650
2023-11-24 18:26:27,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2944320.0, ans=0.125
2023-11-24 18:26:34,977 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8800, loss[loss=0.06646, simple_loss=0.09696, pruned_loss=0.0114, audio_tagging_loss=0.006588, over 15099.00 frames. ], tot_loss[loss=0.06759, simple_loss=0.09139, pruned_loss=0.01282, audio_tagging_loss=0.009079, over 3041983.05 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:26:50,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2944453.3333333335, ans=0.1
2023-11-24 18:26:58,889 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2944520.0, ans=0.125
2023-11-24 18:27:12,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=2944586.6666666665, ans=0.0
2023-11-24 18:27:17,533 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2944586.6666666665, ans=0.125
2023-11-24 18:27:20,066 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2944586.6666666665, ans=0.0
2023-11-24 18:27:25,837 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441700
2023-11-24 18:27:28,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2944653.3333333335, ans=0.125
2023-11-24 18:27:37,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2944720.0, ans=0.0
2023-11-24 18:27:38,744 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8850, loss[loss=0.06408, simple_loss=0.09447, pruned_loss=0.01123, audio_tagging_loss=0.005609, over 15412.00 frames. ], tot_loss[loss=0.06746, simple_loss=0.09111, pruned_loss=0.01281, audio_tagging_loss=0.0091, over 3048645.95 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:27:38,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2944720.0, ans=0.0
2023-11-24 18:27:42,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2944720.0, ans=0.125
2023-11-24 18:27:48,753 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2944720.0, ans=0.125
2023-11-24 18:27:49,704 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:27:51,681 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.21 vs. limit=15.0
2023-11-24 18:28:01,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2944786.6666666665, ans=0.0
2023-11-24 18:28:12,406 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.411e+01 8.596e+01 9.206e+01 9.875e+01 1.365e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 18:28:16,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2944920.0, ans=0.0
2023-11-24 18:28:28,869 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441750
2023-11-24 18:28:35,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2944986.6666666665, ans=0.2
2023-11-24 18:28:35,189 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2944986.6666666665, ans=0.2
2023-11-24 18:28:40,890 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8900, loss[loss=0.06057, simple_loss=0.07561, pruned_loss=0.01069, audio_tagging_loss=0.01208, over 14754.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09143, pruned_loss=0.01288, audio_tagging_loss=0.008969, over 3051201.94 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:29:20,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2945253.3333333335, ans=0.2
2023-11-24 18:29:31,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2945320.0, ans=0.0
2023-11-24 18:29:32,502 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441800
2023-11-24 18:29:38,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2945320.0, ans=0.0
2023-11-24 18:29:40,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2945320.0, ans=0.0
2023-11-24 18:29:46,781 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 8950, loss[loss=0.05163, simple_loss=0.06876, pruned_loss=0.00967, audio_tagging_loss=0.007585, over 14577.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09189, pruned_loss=0.01291, audio_tagging_loss=0.008859, over 3056753.96 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:29:47,112 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2945386.6666666665, ans=0.125
2023-11-24 18:29:51,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2945386.6666666665, ans=0.125
2023-11-24 18:30:04,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=2945453.3333333335, ans=0.0
2023-11-24 18:30:07,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=2945453.3333333335, ans=0.0
2023-11-24 18:30:18,351 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.98 vs. limit=22.5
2023-11-24 18:30:20,104 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.368e+01 8.707e+01 9.494e+01 1.019e+02 1.254e+02, threshold=1.899e+02, percent-clipped=0.0
2023-11-24 18:30:37,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441850
2023-11-24 18:30:37,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2945653.3333333335, ans=0.125
2023-11-24 18:30:40,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2945653.3333333335, ans=0.125
2023-11-24 18:30:50,104 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9000, loss[loss=0.0793, simple_loss=0.1017, pruned_loss=0.02144, audio_tagging_loss=0.007002, over 15485.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09133, pruned_loss=0.0129, audio_tagging_loss=0.00877, over 3051999.87 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:30:50,107 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 18:31:31,483 INFO [train_asr.py:1253] (0/4) Epoch 37, validation: loss=0.05871, simple_loss=0.05072, pruned_loss=0.005135, audio_tagging_loss=0.02821, over 4681554.00 frames. 
2023-11-24 18:31:31,484 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 18:31:39,913 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.36 vs. limit=15.0
2023-11-24 18:31:49,489 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.45 vs. limit=15.0
2023-11-24 18:31:53,211 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2945786.6666666665, ans=0.1
2023-11-24 18:31:54,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2945786.6666666665, ans=0.125
2023-11-24 18:32:02,006 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.06 vs. limit=10.0
2023-11-24 18:32:07,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2945853.3333333335, ans=0.125
2023-11-24 18:32:14,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2945920.0, ans=0.0
2023-11-24 18:32:22,551 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441900
2023-11-24 18:32:25,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2945986.6666666665, ans=0.1
2023-11-24 18:32:28,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2945986.6666666665, ans=0.125
2023-11-24 18:32:32,302 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.77 vs. limit=10.0
2023-11-24 18:32:34,940 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9050, loss[loss=0.05057, simple_loss=0.06832, pruned_loss=0.006746, audio_tagging_loss=0.009665, over 15295.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09092, pruned_loss=0.01285, audio_tagging_loss=0.008749, over 3050754.62 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:32:45,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=2946053.3333333335, ans=0.04949747468305833
2023-11-24 18:32:45,888 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:33:02,041 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2946186.6666666665, ans=0.0
2023-11-24 18:33:09,599 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.312e+01 8.396e+01 9.037e+01 9.861e+01 1.283e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 18:33:12,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2946253.3333333335, ans=0.07
2023-11-24 18:33:25,125 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 441950
2023-11-24 18:33:37,244 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.50 vs. limit=12.0
2023-11-24 18:33:37,588 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9100, loss[loss=0.08053, simple_loss=0.09868, pruned_loss=0.02006, audio_tagging_loss=0.01112, over 15391.00 frames. ], tot_loss[loss=0.06737, simple_loss=0.09138, pruned_loss=0.01294, audio_tagging_loss=0.008743, over 3054204.94 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:33:39,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2946386.6666666665, ans=0.0
2023-11-24 18:33:40,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2946386.6666666665, ans=0.0
2023-11-24 18:33:52,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2946453.3333333335, ans=0.0
2023-11-24 18:33:56,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2946453.3333333335, ans=0.2
2023-11-24 18:34:27,586 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442000
2023-11-24 18:34:31,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2946653.3333333335, ans=0.0
2023-11-24 18:34:34,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2946653.3333333335, ans=0.05
2023-11-24 18:34:38,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2946720.0, ans=0.0
2023-11-24 18:34:38,776 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2946720.0, ans=0.125
2023-11-24 18:34:39,639 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9150, loss[loss=0.08762, simple_loss=0.1192, pruned_loss=0.02113, audio_tagging_loss=0.006888, over 14130.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09148, pruned_loss=0.01286, audio_tagging_loss=0.008633, over 3051447.59 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:34:59,222 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2946786.6666666665, ans=0.125
2023-11-24 18:35:03,802 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.69 vs. limit=15.0
2023-11-24 18:35:15,206 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.679e+01 8.510e+01 9.159e+01 9.829e+01 1.251e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 18:35:20,531 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=15.0
2023-11-24 18:35:23,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2946920.0, ans=0.2
2023-11-24 18:35:30,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442050
2023-11-24 18:35:42,494 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9200, loss[loss=0.08557, simple_loss=0.1199, pruned_loss=0.01821, audio_tagging_loss=0.007434, over 15899.00 frames. ], tot_loss[loss=0.06674, simple_loss=0.09062, pruned_loss=0.01275, audio_tagging_loss=0.008679, over 3052364.67 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:35:57,420 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.07 vs. limit=6.0
2023-11-24 18:36:32,590 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442100
2023-11-24 18:36:37,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2947320.0, ans=0.125
2023-11-24 18:36:45,620 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9250, loss[loss=0.07176, simple_loss=0.09087, pruned_loss=0.0163, audio_tagging_loss=0.01003, over 16199.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09159, pruned_loss=0.0129, audio_tagging_loss=0.008685, over 3057988.75 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:36:51,832 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=2947386.6666666665, ans=0.125
2023-11-24 18:36:54,333 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=2947386.6666666665, ans=15.0
2023-11-24 18:37:07,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2947453.3333333335, ans=0.125
2023-11-24 18:37:19,952 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.271e+01 8.726e+01 9.340e+01 9.852e+01 1.345e+02, threshold=1.868e+02, percent-clipped=0.0
2023-11-24 18:37:35,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442150
2023-11-24 18:37:36,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2947653.3333333335, ans=0.1
2023-11-24 18:37:47,426 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9300, loss[loss=0.0692, simple_loss=0.09597, pruned_loss=0.01277, audio_tagging_loss=0.008443, over 14356.00 frames. ], tot_loss[loss=0.06769, simple_loss=0.09204, pruned_loss=0.01293, audio_tagging_loss=0.008743, over 3059642.46 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:37:48,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2947720.0, ans=0.2
2023-11-24 18:37:56,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2947720.0, ans=0.125
2023-11-24 18:37:58,018 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2947720.0, ans=0.07
2023-11-24 18:38:08,287 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.95 vs. limit=15.0
2023-11-24 18:38:17,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=2947853.3333333335, ans=0.125
2023-11-24 18:38:37,550 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442200
2023-11-24 18:38:37,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=2947986.6666666665, ans=0.95
2023-11-24 18:38:40,474 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0
2023-11-24 18:38:51,068 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9350, loss[loss=0.0804, simple_loss=0.09666, pruned_loss=0.0194, audio_tagging_loss=0.01267, over 15705.00 frames. ], tot_loss[loss=0.06815, simple_loss=0.09259, pruned_loss=0.01313, audio_tagging_loss=0.00873, over 3060972.60 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:38:52,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2948053.3333333335, ans=0.1
2023-11-24 18:38:55,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=2948053.3333333335, ans=0.0
2023-11-24 18:38:59,883 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.75 vs. limit=10.0
2023-11-24 18:39:10,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.max_positive, batch_count=2948120.0, ans=0.95
2023-11-24 18:39:26,936 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.309e+01 8.625e+01 9.145e+01 9.875e+01 1.374e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 18:39:32,665 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.90 vs. limit=15.0
2023-11-24 18:39:35,114 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.20 vs. limit=15.0
2023-11-24 18:39:40,962 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442250
2023-11-24 18:39:42,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2948320.0, ans=0.125
2023-11-24 18:39:52,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.85 vs. limit=15.0
2023-11-24 18:39:53,306 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9400, loss[loss=0.06707, simple_loss=0.09892, pruned_loss=0.008459, audio_tagging_loss=0.009152, over 15496.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09203, pruned_loss=0.01311, audio_tagging_loss=0.008876, over 3053909.48 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:40:00,381 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.29 vs. limit=12.0
2023-11-24 18:40:11,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2948453.3333333335, ans=0.1
2023-11-24 18:40:13,779 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2948453.3333333335, ans=0.125
2023-11-24 18:40:34,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2948586.6666666665, ans=0.125
2023-11-24 18:40:34,465 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2948586.6666666665, ans=0.2
2023-11-24 18:40:44,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442300
2023-11-24 18:40:48,655 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2948653.3333333335, ans=0.2
2023-11-24 18:40:54,226 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:40:56,584 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9450, loss[loss=0.08474, simple_loss=0.1119, pruned_loss=0.01876, audio_tagging_loss=0.01005, over 15240.00 frames. ], tot_loss[loss=0.0682, simple_loss=0.09213, pruned_loss=0.01323, audio_tagging_loss=0.008908, over 3050291.11 frames. ], batch size: 55, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:41:05,671 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2948720.0, ans=0.125
2023-11-24 18:41:08,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2948786.6666666665, ans=0.125
2023-11-24 18:41:09,256 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2948786.6666666665, ans=0.125
2023-11-24 18:41:14,755 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2948786.6666666665, ans=0.2
2023-11-24 18:41:33,026 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.255e+01 8.857e+01 9.505e+01 1.052e+02 1.346e+02, threshold=1.901e+02, percent-clipped=0.0
2023-11-24 18:41:47,133 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442350
2023-11-24 18:41:51,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2948986.6666666665, ans=0.125
2023-11-24 18:41:59,894 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9500, loss[loss=0.06039, simple_loss=0.08512, pruned_loss=0.008952, audio_tagging_loss=0.008882, over 15017.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09112, pruned_loss=0.01308, audio_tagging_loss=0.00898, over 3047296.71 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:42:33,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2949186.6666666665, ans=0.0
2023-11-24 18:42:42,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2949253.3333333335, ans=0.0
2023-11-24 18:42:44,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2949253.3333333335, ans=0.0
2023-11-24 18:42:45,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2949253.3333333335, ans=0.0
2023-11-24 18:42:49,693 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442400
2023-11-24 18:42:52,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2949320.0, ans=0.2
2023-11-24 18:42:53,838 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=2949320.0, ans=0.0
2023-11-24 18:43:02,623 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9550, loss[loss=0.07323, simple_loss=0.1043, pruned_loss=0.01376, audio_tagging_loss=0.00733, over 15175.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09095, pruned_loss=0.01299, audio_tagging_loss=0.009086, over 3044644.95 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:43:05,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2949386.6666666665, ans=0.1
2023-11-24 18:43:05,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass_mid.scale_min, batch_count=2949386.6666666665, ans=0.2
2023-11-24 18:43:07,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2949386.6666666665, ans=0.125
2023-11-24 18:43:32,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=2949520.0, ans=0.5
2023-11-24 18:43:33,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2949520.0, ans=0.1
2023-11-24 18:43:38,729 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.868e+01 8.675e+01 9.207e+01 9.929e+01 1.581e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 18:43:42,558 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2949586.6666666665, ans=0.0
2023-11-24 18:43:43,897 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2949586.6666666665, ans=0.0
2023-11-24 18:43:44,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.43 vs. limit=12.0
2023-11-24 18:43:50,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2949586.6666666665, ans=0.0
2023-11-24 18:43:52,572 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442450
2023-11-24 18:43:56,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2949653.3333333335, ans=0.125
2023-11-24 18:44:00,294 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.23 vs. limit=15.0
2023-11-24 18:44:04,424 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9600, loss[loss=0.0625, simple_loss=0.08561, pruned_loss=0.009176, audio_tagging_loss=0.01052, over 15654.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.0906, pruned_loss=0.01296, audio_tagging_loss=0.009184, over 3046761.43 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:44:36,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2949853.3333333335, ans=0.2
2023-11-24 18:44:47,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2949920.0, ans=0.2
2023-11-24 18:44:54,663 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442500
2023-11-24 18:44:54,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2949986.6666666665, ans=0.0
2023-11-24 18:45:07,136 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9650, loss[loss=0.04438, simple_loss=0.06062, pruned_loss=0.005252, audio_tagging_loss=0.008818, over 15609.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09036, pruned_loss=0.01287, audio_tagging_loss=0.009152, over 3042384.77 frames. ], batch size: 62, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:45:07,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2950053.3333333335, ans=0.125
2023-11-24 18:45:17,791 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.24 vs. limit=10.0
2023-11-24 18:45:20,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2950120.0, ans=0.2
2023-11-24 18:45:26,196 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:45:43,084 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.493e+01 8.425e+01 9.108e+01 9.681e+01 1.366e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 18:45:44,086 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=7.91 vs. limit=10.0
2023-11-24 18:45:57,330 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442550
2023-11-24 18:46:09,221 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9700, loss[loss=0.07351, simple_loss=0.102, pruned_loss=0.01613, audio_tagging_loss=0.006393, over 15787.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09023, pruned_loss=0.01293, audio_tagging_loss=0.009033, over 3036668.50 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:46:20,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2950453.3333333335, ans=0.0
2023-11-24 18:46:24,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2950453.3333333335, ans=0.125
2023-11-24 18:46:33,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=2950520.0, ans=0.05
2023-11-24 18:46:39,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff3_skip_rate, batch_count=2950520.0, ans=0.0
2023-11-24 18:46:44,187 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.05 vs. limit=15.0
2023-11-24 18:47:00,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442600
2023-11-24 18:47:12,415 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9750, loss[loss=0.04975, simple_loss=0.07041, pruned_loss=0.006397, audio_tagging_loss=0.008148, over 14449.00 frames. ], tot_loss[loss=0.06672, simple_loss=0.09015, pruned_loss=0.01284, audio_tagging_loss=0.008808, over 3031494.43 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:47:48,959 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.947e+01 8.439e+01 9.144e+01 9.970e+01 1.220e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 18:48:02,052 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442650
2023-11-24 18:48:05,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=2950986.6666666665, ans=0.125
2023-11-24 18:48:14,355 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9800, loss[loss=0.06113, simple_loss=0.08477, pruned_loss=0.01246, audio_tagging_loss=0.006284, over 15901.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09097, pruned_loss=0.01302, audio_tagging_loss=0.008715, over 3037359.71 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:48:36,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2951120.0, ans=0.0
2023-11-24 18:48:40,839 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.39 vs. limit=6.0
2023-11-24 18:48:52,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2951253.3333333335, ans=0.125
2023-11-24 18:49:03,864 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442700
2023-11-24 18:49:09,072 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:49:16,070 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9850, loss[loss=0.05486, simple_loss=0.07118, pruned_loss=0.01175, audio_tagging_loss=0.007519, over 14329.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09107, pruned_loss=0.01305, audio_tagging_loss=0.008676, over 3041994.94 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:49:20,567 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2951386.6666666665, ans=0.125
2023-11-24 18:49:53,632 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.250e+01 8.535e+01 9.041e+01 9.752e+01 1.279e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 18:50:04,480 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2951653.3333333335, ans=0.125
2023-11-24 18:50:05,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442750
2023-11-24 18:50:09,020 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.12 vs. limit=6.0
2023-11-24 18:50:15,070 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.49 vs. limit=15.0
2023-11-24 18:50:17,760 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9900, loss[loss=0.06567, simple_loss=0.08517, pruned_loss=0.0129, audio_tagging_loss=0.01019, over 15608.00 frames. ], tot_loss[loss=0.06696, simple_loss=0.09073, pruned_loss=0.01297, audio_tagging_loss=0.008624, over 3047078.33 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:50:19,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2951720.0, ans=0.125
2023-11-24 18:50:44,007 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=8.98 vs. limit=15.0
2023-11-24 18:50:44,017 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.83 vs. limit=12.0
2023-11-24 18:50:49,646 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2951853.3333333335, ans=0.125
2023-11-24 18:50:52,030 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2951853.3333333335, ans=0.0
2023-11-24 18:51:00,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=2951920.0, ans=0.0
2023-11-24 18:51:07,638 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442800
2023-11-24 18:51:13,148 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn2.whiten, num_groups=1, num_channels=512, metric=16.73 vs. limit=22.5
2023-11-24 18:51:14,349 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=9.08 vs. limit=15.0
2023-11-24 18:51:19,572 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 9950, loss[loss=0.08597, simple_loss=0.1124, pruned_loss=0.02201, audio_tagging_loss=0.007754, over 16851.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.09032, pruned_loss=0.01302, audio_tagging_loss=0.008676, over 3052774.99 frames. ], batch size: 62, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:51:33,371 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2952120.0, ans=0.125
2023-11-24 18:51:35,752 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2952120.0, ans=0.1
2023-11-24 18:51:50,620 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2952186.6666666665, ans=0.1
2023-11-24 18:51:57,277 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.411e+01 8.463e+01 9.019e+01 9.662e+01 1.211e+02, threshold=1.804e+02, percent-clipped=0.0
2023-11-24 18:51:58,909 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2952253.3333333335, ans=0.1
2023-11-24 18:52:04,066 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=9.14 vs. limit=12.0
2023-11-24 18:52:10,132 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442850
2023-11-24 18:52:23,032 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10000, loss[loss=0.05414, simple_loss=0.06308, pruned_loss=0.01176, audio_tagging_loss=0.01084, over 14900.00 frames. ], tot_loss[loss=0.06681, simple_loss=0.09046, pruned_loss=0.01294, audio_tagging_loss=0.008643, over 3049788.34 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 18:52:33,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2952386.6666666665, ans=0.125
2023-11-24 18:52:33,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2952386.6666666665, ans=0.0
2023-11-24 18:52:34,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2952453.3333333335, ans=0.125
2023-11-24 18:52:42,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2952453.3333333335, ans=0.125
2023-11-24 18:52:43,879 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2952453.3333333335, ans=0.95
2023-11-24 18:52:53,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2952520.0, ans=0.0
2023-11-24 18:53:11,979 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442900
2023-11-24 18:53:12,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2952653.3333333335, ans=0.2
2023-11-24 18:53:24,198 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10050, loss[loss=0.04924, simple_loss=0.06565, pruned_loss=0.006358, audio_tagging_loss=0.01006, over 14677.00 frames. ], tot_loss[loss=0.06666, simple_loss=0.09032, pruned_loss=0.01282, audio_tagging_loss=0.008687, over 3053137.23 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:53:38,491 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2952786.6666666665, ans=0.125
2023-11-24 18:53:44,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2952786.6666666665, ans=0.125
2023-11-24 18:54:02,841 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.473e+01 8.442e+01 9.039e+01 9.729e+01 1.134e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 18:54:13,537 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 442950
2023-11-24 18:54:14,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2952986.6666666665, ans=0.0
2023-11-24 18:54:15,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2952986.6666666665, ans=0.125
2023-11-24 18:54:25,237 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10100, loss[loss=0.06702, simple_loss=0.08032, pruned_loss=0.01695, audio_tagging_loss=0.009909, over 14680.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09122, pruned_loss=0.01308, audio_tagging_loss=0.008696, over 3057391.02 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:54:41,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2953120.0, ans=0.95
2023-11-24 18:55:13,619 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2953320.0, ans=0.2
2023-11-24 18:55:14,624 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:55:14,700 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443000
2023-11-24 18:55:16,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2953320.0, ans=0.125
2023-11-24 18:55:23,343 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:55:28,594 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10150, loss[loss=0.07145, simple_loss=0.09923, pruned_loss=0.01393, audio_tagging_loss=0.007904, over 15099.00 frames. ], tot_loss[loss=0.06682, simple_loss=0.09032, pruned_loss=0.01284, audio_tagging_loss=0.00883, over 3053875.98 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:55:56,454 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:56:06,392 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.827e+01 8.628e+01 9.087e+01 9.762e+01 1.339e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 18:56:18,462 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443050
2023-11-24 18:56:24,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2953653.3333333335, ans=0.015
2023-11-24 18:56:30,720 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10200, loss[loss=0.06159, simple_loss=0.07694, pruned_loss=0.01152, audio_tagging_loss=0.01161, over 15255.00 frames. ], tot_loss[loss=0.06711, simple_loss=0.09048, pruned_loss=0.01299, audio_tagging_loss=0.008882, over 3052343.69 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:56:44,102 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2953786.6666666665, ans=0.0
2023-11-24 18:56:52,734 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 18:57:04,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2953853.3333333335, ans=0.125
2023-11-24 18:57:20,405 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443100
2023-11-24 18:57:27,736 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:57:32,188 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10250, loss[loss=0.06991, simple_loss=0.08259, pruned_loss=0.01656, audio_tagging_loss=0.01205, over 13863.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09069, pruned_loss=0.01317, audio_tagging_loss=0.008899, over 3047842.42 frames. ], batch size: 53, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:57:41,237 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2954053.3333333335, ans=0.1
2023-11-24 18:57:47,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2954120.0, ans=0.1
2023-11-24 18:57:49,269 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 18:57:50,841 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.51 vs. limit=22.5
2023-11-24 18:57:55,786 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2954120.0, ans=0.2
2023-11-24 18:58:11,697 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.337e+01 8.466e+01 9.132e+01 9.893e+01 1.266e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 18:58:11,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2954253.3333333335, ans=0.0
2023-11-24 18:58:22,706 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443150
2023-11-24 18:58:35,795 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10300, loss[loss=0.07836, simple_loss=0.118, pruned_loss=0.01295, audio_tagging_loss=0.006408, over 14574.00 frames. ], tot_loss[loss=0.06797, simple_loss=0.09153, pruned_loss=0.01321, audio_tagging_loss=0.008992, over 3052539.99 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:58:42,207 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.68 vs. limit=6.0
2023-11-24 18:58:49,503 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.71 vs. limit=12.0
2023-11-24 18:58:52,032 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.14 vs. limit=6.0
2023-11-24 18:58:57,909 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.89 vs. limit=15.0
2023-11-24 18:59:17,439 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2954586.6666666665, ans=0.0
2023-11-24 18:59:22,419 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2954586.6666666665, ans=0.125
2023-11-24 18:59:25,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443200
2023-11-24 18:59:34,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2954653.3333333335, ans=0.125
2023-11-24 18:59:39,373 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10350, loss[loss=0.07215, simple_loss=0.09514, pruned_loss=0.01315, audio_tagging_loss=0.01143, over 15089.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09073, pruned_loss=0.01303, audio_tagging_loss=0.009168, over 3050706.57 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 18:59:50,291 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=2954786.6666666665, ans=10.0
2023-11-24 18:59:51,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2954786.6666666665, ans=0.0
2023-11-24 18:59:57,602 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.91 vs. limit=15.0
2023-11-24 19:00:01,925 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.91 vs. limit=15.0
2023-11-24 19:00:03,802 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:00:17,086 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.601e+01 8.606e+01 9.110e+01 9.960e+01 1.328e+02, threshold=1.822e+02, percent-clipped=0.0
2023-11-24 19:00:28,633 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443250
2023-11-24 19:00:36,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=2954986.6666666665, ans=0.125
2023-11-24 19:00:36,999 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2954986.6666666665, ans=0.2
2023-11-24 19:00:40,319 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10400, loss[loss=0.07356, simple_loss=0.09958, pruned_loss=0.01592, audio_tagging_loss=0.00785, over 15421.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09096, pruned_loss=0.01322, audio_tagging_loss=0.009211, over 3047223.72 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:00:51,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2955120.0, ans=0.125
2023-11-24 19:00:53,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2955120.0, ans=0.2
2023-11-24 19:01:05,389 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2955186.6666666665, ans=0.125
2023-11-24 19:01:23,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2955253.3333333335, ans=0.1
2023-11-24 19:01:29,334 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2955320.0, ans=0.125
2023-11-24 19:01:29,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.40 vs. limit=12.0
2023-11-24 19:01:30,236 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443300
2023-11-24 19:01:43,039 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10450, loss[loss=0.06803, simple_loss=0.09215, pruned_loss=0.01352, audio_tagging_loss=0.008434, over 15353.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09121, pruned_loss=0.01327, audio_tagging_loss=0.009162, over 3043463.22 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:01:44,927 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.67 vs. limit=15.0
2023-11-24 19:01:45,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2955386.6666666665, ans=0.125
2023-11-24 19:01:46,754 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2955386.6666666665, ans=0.1
2023-11-24 19:01:48,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2955386.6666666665, ans=0.5
2023-11-24 19:01:56,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2955453.3333333335, ans=0.125
2023-11-24 19:02:11,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=2955520.0, ans=0.05
2023-11-24 19:02:12,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2955520.0, ans=0.0
2023-11-24 19:02:18,904 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2955586.6666666665, ans=0.2
2023-11-24 19:02:22,865 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.090e+01 8.670e+01 9.376e+01 1.004e+02 1.489e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 19:02:33,010 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443350
2023-11-24 19:02:34,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=2955653.3333333335, ans=0.2
2023-11-24 19:02:45,217 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10500, loss[loss=0.0723, simple_loss=0.1086, pruned_loss=0.01188, audio_tagging_loss=0.006117, over 15518.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09058, pruned_loss=0.01306, audio_tagging_loss=0.008971, over 3040823.06 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:02:49,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2955720.0, ans=0.125
2023-11-24 19:03:14,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2955853.3333333335, ans=0.125
2023-11-24 19:03:27,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2955920.0, ans=0.1
2023-11-24 19:03:35,034 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443400
2023-11-24 19:03:45,887 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2955986.6666666665, ans=0.1
2023-11-24 19:03:48,027 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10550, loss[loss=0.05969, simple_loss=0.07975, pruned_loss=0.008602, audio_tagging_loss=0.01121, over 14940.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.08975, pruned_loss=0.01289, audio_tagging_loss=0.008903, over 3036195.90 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:03:49,757 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.44 vs. limit=15.0
2023-11-24 19:03:58,178 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=2956053.3333333335, ans=0.0
2023-11-24 19:03:58,637 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.45 vs. limit=22.5
2023-11-24 19:04:08,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=2956120.0, ans=0.2
2023-11-24 19:04:15,615 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=14.01 vs. limit=22.5
2023-11-24 19:04:19,524 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2956186.6666666665, ans=0.1
2023-11-24 19:04:29,169 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.227e+01 8.503e+01 9.037e+01 9.861e+01 1.136e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 19:04:29,397 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=2956253.3333333335, ans=0.05
2023-11-24 19:04:29,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2956253.3333333335, ans=0.125
2023-11-24 19:04:31,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2956253.3333333335, ans=0.07
2023-11-24 19:04:37,555 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443450
2023-11-24 19:04:49,852 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10600, loss[loss=0.04622, simple_loss=0.05201, pruned_loss=0.01037, audio_tagging_loss=0.009847, over 16222.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.09014, pruned_loss=0.01277, audio_tagging_loss=0.008857, over 3038107.76 frames. ], batch size: 64, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:04:54,307 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=2956386.6666666665, ans=0.0
2023-11-24 19:05:03,709 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.10 vs. limit=12.0
2023-11-24 19:05:11,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2956453.3333333335, ans=0.1
2023-11-24 19:05:19,975 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=2956520.0, ans=0.035
2023-11-24 19:05:21,166 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2956520.0, ans=0.125
2023-11-24 19:05:39,466 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443500
2023-11-24 19:05:48,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=2956653.3333333335, ans=0.2
2023-11-24 19:05:51,797 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10650, loss[loss=0.07212, simple_loss=0.09523, pruned_loss=0.01738, audio_tagging_loss=0.007131, over 15272.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09059, pruned_loss=0.01289, audio_tagging_loss=0.008798, over 3037607.13 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:06:03,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2956786.6666666665, ans=0.125
2023-11-24 19:06:06,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2956786.6666666665, ans=0.0
2023-11-24 19:06:06,891 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.21 vs. limit=6.0
2023-11-24 19:06:26,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2956853.3333333335, ans=0.125
2023-11-24 19:06:30,397 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.22 vs. limit=6.0
2023-11-24 19:06:33,393 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.594e+01 8.793e+01 9.512e+01 1.031e+02 1.281e+02, threshold=1.902e+02, percent-clipped=0.0
2023-11-24 19:06:38,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2956920.0, ans=0.1
2023-11-24 19:06:39,814 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:06:42,728 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443550
2023-11-24 19:06:42,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2956986.6666666665, ans=0.125
2023-11-24 19:06:45,623 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.21 vs. limit=10.0
2023-11-24 19:06:49,325 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.33 vs. limit=15.0
2023-11-24 19:06:54,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2957053.3333333335, ans=0.125
2023-11-24 19:06:55,220 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10700, loss[loss=0.07489, simple_loss=0.104, pruned_loss=0.01388, audio_tagging_loss=0.009009, over 13961.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.09051, pruned_loss=0.01286, audio_tagging_loss=0.00874, over 3032304.17 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:07:10,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2957120.0, ans=0.1
2023-11-24 19:07:12,768 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2957120.0, ans=0.125
2023-11-24 19:07:14,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2957120.0, ans=0.125
2023-11-24 19:07:21,796 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=9.90 vs. limit=15.0
2023-11-24 19:07:31,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2957253.3333333335, ans=0.0
2023-11-24 19:07:34,137 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.72 vs. limit=15.0
2023-11-24 19:07:34,976 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2957253.3333333335, ans=0.125
2023-11-24 19:07:45,003 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443600
2023-11-24 19:07:57,792 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10750, loss[loss=0.05512, simple_loss=0.06614, pruned_loss=0.01305, audio_tagging_loss=0.009003, over 15521.00 frames. ], tot_loss[loss=0.06636, simple_loss=0.08972, pruned_loss=0.01278, audio_tagging_loss=0.008718, over 3035325.23 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 8.0
2023-11-24 19:08:11,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2957453.3333333335, ans=0.1
2023-11-24 19:08:23,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2957520.0, ans=0.1
2023-11-24 19:08:34,905 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=2957586.6666666665, ans=0.125
2023-11-24 19:08:37,963 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.899e+01 8.444e+01 9.135e+01 9.725e+01 3.439e+02, threshold=1.827e+02, percent-clipped=1.0
2023-11-24 19:08:46,388 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0
2023-11-24 19:08:46,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443650
2023-11-24 19:08:49,447 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2957653.3333333335, ans=0.1
2023-11-24 19:08:51,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2957653.3333333335, ans=0.0
2023-11-24 19:08:54,617 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2957653.3333333335, ans=0.2
2023-11-24 19:08:59,158 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10800, loss[loss=0.07279, simple_loss=0.0921, pruned_loss=0.0173, audio_tagging_loss=0.009432, over 15288.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.09011, pruned_loss=0.01293, audio_tagging_loss=0.008689, over 3035019.65 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:09:03,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2957720.0, ans=0.0
2023-11-24 19:09:07,234 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2957720.0, ans=0.125
2023-11-24 19:09:15,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2957786.6666666665, ans=0.0
2023-11-24 19:09:20,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2957786.6666666665, ans=0.0
2023-11-24 19:09:48,875 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443700
2023-11-24 19:09:56,616 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2957986.6666666665, ans=0.1
2023-11-24 19:10:01,071 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10850, loss[loss=0.05749, simple_loss=0.07915, pruned_loss=0.00619, audio_tagging_loss=0.01173, over 15273.00 frames. ], tot_loss[loss=0.06654, simple_loss=0.08974, pruned_loss=0.01285, audio_tagging_loss=0.008817, over 3035255.55 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:10:06,116 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer1.prob, batch_count=2958053.3333333335, ans=0.125
2023-11-24 19:10:15,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2958120.0, ans=0.05
2023-11-24 19:10:20,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2958120.0, ans=0.0
2023-11-24 19:10:21,200 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.46 vs. limit=6.0
2023-11-24 19:10:43,176 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.404e+01 9.026e+01 9.467e+01 1.240e+02, threshold=1.805e+02, percent-clipped=0.0
2023-11-24 19:10:43,978 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.82 vs. limit=15.0
2023-11-24 19:10:45,747 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2958253.3333333335, ans=0.0
2023-11-24 19:10:51,560 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443750
2023-11-24 19:10:59,090 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:11:04,359 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10900, loss[loss=0.0469, simple_loss=0.06106, pruned_loss=0.005992, audio_tagging_loss=0.01038, over 15510.00 frames. ], tot_loss[loss=0.0664, simple_loss=0.08973, pruned_loss=0.01271, audio_tagging_loss=0.00883, over 3035160.44 frames. ], batch size: 60, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:11:32,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=2958520.0, ans=0.2
2023-11-24 19:11:38,105 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.17 vs. limit=15.0
2023-11-24 19:11:41,445 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2958586.6666666665, ans=0.1
2023-11-24 19:11:52,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2958586.6666666665, ans=0.125
2023-11-24 19:11:54,392 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443800
2023-11-24 19:12:05,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.21 vs. limit=12.0
2023-11-24 19:12:06,979 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 10950, loss[loss=0.07338, simple_loss=0.1056, pruned_loss=0.01297, audio_tagging_loss=0.007629, over 15132.00 frames. ], tot_loss[loss=0.06689, simple_loss=0.09027, pruned_loss=0.01291, audio_tagging_loss=0.008845, over 3036557.32 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:12:11,446 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=10.37 vs. limit=15.0
2023-11-24 19:12:48,393 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.026e+01 8.471e+01 9.149e+01 9.860e+01 1.271e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 19:12:51,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2958920.0, ans=0.125
2023-11-24 19:12:56,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443850
2023-11-24 19:13:01,493 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.30 vs. limit=12.0
2023-11-24 19:13:08,963 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11000, loss[loss=0.06876, simple_loss=0.09286, pruned_loss=0.01389, audio_tagging_loss=0.008439, over 16068.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09061, pruned_loss=0.01303, audio_tagging_loss=0.008896, over 3039731.95 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:13:17,573 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:13:53,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2959253.3333333335, ans=0.0
2023-11-24 19:13:57,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.20 vs. limit=6.0
2023-11-24 19:13:58,857 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443900
2023-11-24 19:13:59,038 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2959320.0, ans=0.125
2023-11-24 19:14:09,598 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2959386.6666666665, ans=0.0
2023-11-24 19:14:10,683 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11050, loss[loss=0.07013, simple_loss=0.09356, pruned_loss=0.01465, audio_tagging_loss=0.008703, over 14604.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.08999, pruned_loss=0.01297, audio_tagging_loss=0.009022, over 3041615.39 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:14:20,022 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.74 vs. limit=15.0
2023-11-24 19:14:50,888 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.22 vs. limit=12.0
2023-11-24 19:14:52,692 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.475e+01 8.793e+01 9.326e+01 1.000e+02 1.252e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 19:14:54,191 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2959586.6666666665, ans=0.1
2023-11-24 19:15:01,776 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 443950
2023-11-24 19:15:14,609 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11100, loss[loss=0.04068, simple_loss=0.0407, pruned_loss=0.006711, audio_tagging_loss=0.01362, over 16002.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.08979, pruned_loss=0.01289, audio_tagging_loss=0.009122, over 3040459.66 frames. ], batch size: 63, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:15:19,678 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2959720.0, ans=0.1
2023-11-24 19:15:27,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.12 vs. limit=12.0
2023-11-24 19:15:41,730 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=4.02 vs. limit=15.0
2023-11-24 19:16:04,352 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444000
2023-11-24 19:16:05,807 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-444000.pt
2023-11-24 19:16:17,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2959986.6666666665, ans=0.0
2023-11-24 19:16:18,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2959986.6666666665, ans=0.1
2023-11-24 19:16:21,578 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11150, loss[loss=0.0729, simple_loss=0.09817, pruned_loss=0.01356, audio_tagging_loss=0.01026, over 16824.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09008, pruned_loss=0.01293, audio_tagging_loss=0.009209, over 3046488.09 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:16:44,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2960120.0, ans=0.0
2023-11-24 19:16:59,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2960253.3333333335, ans=0.1
2023-11-24 19:17:02,996 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.194e+01 8.502e+01 9.253e+01 1.001e+02 1.275e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 19:17:04,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2960253.3333333335, ans=0.125
2023-11-24 19:17:11,609 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444050
2023-11-24 19:17:23,209 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11200, loss[loss=0.0629, simple_loss=0.08058, pruned_loss=0.01269, audio_tagging_loss=0.009913, over 15978.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.08963, pruned_loss=0.01296, audio_tagging_loss=0.009283, over 3049921.99 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:17:26,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2960386.6666666665, ans=0.125
2023-11-24 19:17:34,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer2.prob, batch_count=2960386.6666666665, ans=0.125
2023-11-24 19:17:59,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.whiten.whitening_limit, batch_count=2960520.0, ans=12.0
2023-11-24 19:18:05,208 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=23.32 vs. limit=22.5
2023-11-24 19:18:09,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2960586.6666666665, ans=0.1
2023-11-24 19:18:13,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444100
2023-11-24 19:18:26,582 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11250, loss[loss=0.08109, simple_loss=0.1023, pruned_loss=0.02133, audio_tagging_loss=0.008629, over 14467.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.08921, pruned_loss=0.01285, audio_tagging_loss=0.00922, over 3046529.16 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:18:26,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2960720.0, ans=0.125
2023-11-24 19:18:42,189 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.03 vs. limit=22.5
2023-11-24 19:19:01,119 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2960853.3333333335, ans=0.125
2023-11-24 19:19:01,583 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.87 vs. limit=15.0
2023-11-24 19:19:08,007 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.311e+01 8.458e+01 9.183e+01 1.013e+02 1.310e+02, threshold=1.837e+02, percent-clipped=0.0
2023-11-24 19:19:16,065 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444150
2023-11-24 19:19:18,682 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2960986.6666666665, ans=0.125
2023-11-24 19:19:28,398 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11300, loss[loss=0.07, simple_loss=0.106, pruned_loss=0.0102, audio_tagging_loss=0.006822, over 15825.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.08998, pruned_loss=0.01303, audio_tagging_loss=0.009047, over 3050440.17 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:19:36,021 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.12 vs. limit=15.0
2023-11-24 19:19:44,443 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.52 vs. limit=10.0
2023-11-24 19:20:05,562 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.35 vs. limit=22.5
2023-11-24 19:20:18,489 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444200
2023-11-24 19:20:30,541 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11350, loss[loss=0.08068, simple_loss=0.1172, pruned_loss=0.01341, audio_tagging_loss=0.008666, over 15106.00 frames. ], tot_loss[loss=0.06757, simple_loss=0.09106, pruned_loss=0.01314, audio_tagging_loss=0.008903, over 3049674.56 frames. ], batch size: 54, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:20:34,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=2961386.6666666665, ans=0.2
2023-11-24 19:20:47,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2961453.3333333335, ans=0.0
2023-11-24 19:21:02,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2961520.0, ans=0.125
2023-11-24 19:21:12,886 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.997e+01 8.844e+01 9.278e+01 1.039e+02 2.071e+02, threshold=1.856e+02, percent-clipped=1.0
2023-11-24 19:21:13,558 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.82 vs. limit=15.0
2023-11-24 19:21:16,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2961586.6666666665, ans=0.125
2023-11-24 19:21:20,333 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444250
2023-11-24 19:21:26,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=2961653.3333333335, ans=0.95
2023-11-24 19:21:30,642 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2961653.3333333335, ans=0.125
2023-11-24 19:21:32,754 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11400, loss[loss=0.06213, simple_loss=0.08371, pruned_loss=0.01054, audio_tagging_loss=0.009741, over 14915.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.0908, pruned_loss=0.01303, audio_tagging_loss=0.008903, over 3042227.84 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:21:39,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2961720.0, ans=0.0
2023-11-24 19:22:09,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2961920.0, ans=0.0
2023-11-24 19:22:09,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2961920.0, ans=0.07
2023-11-24 19:22:23,254 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444300
2023-11-24 19:22:24,687 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2961986.6666666665, ans=0.125
2023-11-24 19:22:34,727 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2962053.3333333335, ans=0.125
2023-11-24 19:22:36,161 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11450, loss[loss=0.06564, simple_loss=0.08334, pruned_loss=0.0165, audio_tagging_loss=0.007466, over 15123.00 frames. ], tot_loss[loss=0.06728, simple_loss=0.09066, pruned_loss=0.01311, audio_tagging_loss=0.008841, over 3045858.24 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:22:44,901 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2962053.3333333335, ans=0.125
2023-11-24 19:22:51,813 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.balancer.prob, batch_count=2962120.0, ans=0.125
2023-11-24 19:23:18,420 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.568e+01 8.582e+01 9.161e+01 1.011e+02 1.144e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 19:23:26,176 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444350
2023-11-24 19:23:35,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=2962320.0, ans=0.0
2023-11-24 19:23:38,060 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11500, loss[loss=0.07937, simple_loss=0.1042, pruned_loss=0.01561, audio_tagging_loss=0.01166, over 16409.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.09, pruned_loss=0.01284, audio_tagging_loss=0.008843, over 3046920.02 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:23:38,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2962386.6666666665, ans=0.1
2023-11-24 19:24:12,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=2962520.0, ans=0.05
2023-11-24 19:24:14,004 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2962520.0, ans=0.125
2023-11-24 19:24:28,144 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444400
2023-11-24 19:24:40,764 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11550, loss[loss=0.05131, simple_loss=0.06253, pruned_loss=0.008651, audio_tagging_loss=0.01139, over 14910.00 frames. ], tot_loss[loss=0.06684, simple_loss=0.09048, pruned_loss=0.01281, audio_tagging_loss=0.00879, over 3053319.81 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:24:42,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2962720.0, ans=0.0
2023-11-24 19:24:49,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2962720.0, ans=0.0
2023-11-24 19:24:55,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=2962786.6666666665, ans=0.5
2023-11-24 19:25:17,127 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:25:23,487 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.651e+01 8.528e+01 9.073e+01 9.782e+01 1.310e+02, threshold=1.815e+02, percent-clipped=0.0
2023-11-24 19:25:23,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2962920.0, ans=0.2
2023-11-24 19:25:31,221 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444450
2023-11-24 19:25:37,129 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2962986.6666666665, ans=0.1
2023-11-24 19:25:38,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2962986.6666666665, ans=0.1
2023-11-24 19:25:40,651 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2962986.6666666665, ans=0.2
2023-11-24 19:25:43,450 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11600, loss[loss=0.07369, simple_loss=0.09649, pruned_loss=0.01592, audio_tagging_loss=0.009527, over 15353.00 frames. ], tot_loss[loss=0.06658, simple_loss=0.09001, pruned_loss=0.0128, audio_tagging_loss=0.008769, over 3055991.89 frames. ], batch size: 56, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:25:44,210 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=19.42 vs. limit=22.5
2023-11-24 19:25:44,833 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2963053.3333333335, ans=0.2
2023-11-24 19:25:52,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2963053.3333333335, ans=0.0
2023-11-24 19:26:01,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2963120.0, ans=0.1
2023-11-24 19:26:03,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.29 vs. limit=15.0
2023-11-24 19:26:15,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2963186.6666666665, ans=0.0
2023-11-24 19:26:33,565 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444500
2023-11-24 19:26:45,182 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11650, loss[loss=0.07047, simple_loss=0.09645, pruned_loss=0.01379, audio_tagging_loss=0.008455, over 15451.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.0911, pruned_loss=0.01293, audio_tagging_loss=0.008752, over 3055957.05 frames. ], batch size: 58, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:26:49,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=2963386.6666666665, ans=22.5
2023-11-24 19:26:54,914 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=2963386.6666666665, ans=0.025
2023-11-24 19:26:58,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=2963453.3333333335, ans=0.125
2023-11-24 19:26:59,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2963453.3333333335, ans=0.0
2023-11-24 19:27:09,017 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=2963520.0, ans=0.07
2023-11-24 19:27:28,346 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.973e+01 8.230e+01 8.998e+01 9.783e+01 1.223e+02, threshold=1.800e+02, percent-clipped=0.0
2023-11-24 19:27:34,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444550
2023-11-24 19:27:45,120 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=2.97 vs. limit=15.0
2023-11-24 19:27:46,808 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11700, loss[loss=0.06928, simple_loss=0.09113, pruned_loss=0.01563, audio_tagging_loss=0.008083, over 15157.00 frames. ], tot_loss[loss=0.06707, simple_loss=0.09061, pruned_loss=0.01286, audio_tagging_loss=0.008906, over 3055397.05 frames. ], batch size: 57, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:28:02,844 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2963786.6666666665, ans=0.125
2023-11-24 19:28:08,173 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.61 vs. limit=12.0
2023-11-24 19:28:36,663 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444600
2023-11-24 19:28:41,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2963986.6666666665, ans=0.125
2023-11-24 19:28:49,227 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11750, loss[loss=0.07686, simple_loss=0.117, pruned_loss=0.0119, audio_tagging_loss=0.006478, over 14157.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09044, pruned_loss=0.01272, audio_tagging_loss=0.008926, over 3057268.55 frames. ], batch size: 53, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:29:27,421 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2964253.3333333335, ans=0.125
2023-11-24 19:29:30,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2964253.3333333335, ans=0.125
2023-11-24 19:29:33,001 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.133e+01 8.583e+01 9.295e+01 1.006e+02 1.151e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 19:29:38,952 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444650
2023-11-24 19:29:49,881 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2964320.0, ans=0.125
2023-11-24 19:29:52,057 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11800, loss[loss=0.06876, simple_loss=0.09257, pruned_loss=0.01292, audio_tagging_loss=0.009556, over 13797.00 frames. ], tot_loss[loss=0.06693, simple_loss=0.09027, pruned_loss=0.01281, audio_tagging_loss=0.008977, over 3053162.01 frames. ], batch size: 52, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:29:54,699 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2964386.6666666665, ans=0.125
2023-11-24 19:29:57,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2964386.6666666665, ans=0.125
2023-11-24 19:30:00,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2964386.6666666665, ans=0.0
2023-11-24 19:30:21,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2964520.0, ans=0.125
2023-11-24 19:30:42,152 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444700
2023-11-24 19:30:54,560 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11850, loss[loss=0.05997, simple_loss=0.07765, pruned_loss=0.01017, audio_tagging_loss=0.01098, over 16251.00 frames. ], tot_loss[loss=0.06744, simple_loss=0.09117, pruned_loss=0.01298, audio_tagging_loss=0.008886, over 3053888.75 frames. ], batch size: 61, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:31:05,346 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=8.05 vs. limit=12.0
2023-11-24 19:31:07,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2964786.6666666665, ans=0.0
2023-11-24 19:31:27,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2964853.3333333335, ans=0.2
2023-11-24 19:31:37,855 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.162e+01 8.569e+01 9.195e+01 9.795e+01 1.501e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 19:31:43,490 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=2964986.6666666665, ans=0.07
2023-11-24 19:31:44,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444750
2023-11-24 19:31:56,654 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11900, loss[loss=0.05565, simple_loss=0.08217, pruned_loss=0.00795, audio_tagging_loss=0.006609, over 14215.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09099, pruned_loss=0.0128, audio_tagging_loss=0.008953, over 3053580.04 frames. ], batch size: 53, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:32:08,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.30 vs. limit=15.0
2023-11-24 19:32:13,958 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.39 vs. limit=22.5
2023-11-24 19:32:17,319 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.21 vs. limit=15.0
2023-11-24 19:32:35,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2965253.3333333335, ans=0.09899494936611666
2023-11-24 19:32:46,409 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444800
2023-11-24 19:32:59,216 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 11950, loss[loss=0.06171, simple_loss=0.07841, pruned_loss=0.01263, audio_tagging_loss=0.009877, over 15607.00 frames. ], tot_loss[loss=0.06667, simple_loss=0.0898, pruned_loss=0.0126, audio_tagging_loss=0.009172, over 3051477.83 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 16.0
2023-11-24 19:33:09,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2965386.6666666665, ans=0.0
2023-11-24 19:33:09,555 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2965386.6666666665, ans=0.125
2023-11-24 19:33:11,932 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=2965453.3333333335, ans=0.0
2023-11-24 19:33:12,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2965453.3333333335, ans=0.0
2023-11-24 19:33:22,000 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=2965453.3333333335, ans=0.125
2023-11-24 19:33:24,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2965520.0, ans=0.1
2023-11-24 19:33:25,921 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=2965520.0, ans=0.125
2023-11-24 19:33:26,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2965520.0, ans=0.1
2023-11-24 19:33:28,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2965520.0, ans=0.07
2023-11-24 19:33:41,978 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.648e+01 8.644e+01 9.247e+01 9.996e+01 1.290e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 19:33:47,788 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444850
2023-11-24 19:33:59,363 INFO [train_asr.py:1221] (0/4) Epoch 37, batch 12000, loss[loss=0.07608, simple_loss=0.09397, pruned_loss=0.01914, audio_tagging_loss=0.009955, over 16191.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09001, pruned_loss=0.01264, audio_tagging_loss=0.009291, over 3048173.98 frames. ], batch size: 59, lr: 1.82e-03, grad_scale: 32.0
2023-11-24 19:33:59,366 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 19:34:23,766 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.0941, 4.6320, 5.1761, 4.7812], device='cuda:0')
2023-11-24 19:34:41,838 INFO [train_asr.py:1253] (0/4) Epoch 37, validation: loss=0.058, simple_loss=0.05081, pruned_loss=0.005169, audio_tagging_loss=0.02743, over 4681554.00 frames. 
2023-11-24 19:34:41,839 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 19:34:45,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2965720.0, ans=0.125
2023-11-24 19:34:49,926 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=2965720.0, ans=0.025
2023-11-24 19:34:53,948 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.49 vs. limit=15.0
2023-11-24 19:34:56,618 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2965786.6666666665, ans=0.125
2023-11-24 19:35:09,419 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-37.pt
2023-11-24 19:35:40,410 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 0, loss[loss=0.08496, simple_loss=0.1024, pruned_loss=0.01129, audio_tagging_loss=0.02246, over 15879.00 frames. ], tot_loss[loss=0.08496, simple_loss=0.1024, pruned_loss=0.01129, audio_tagging_loss=0.02246, over 15879.00 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:35:40,413 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 19:36:04,031 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.3.encoder.layers.3.self_attn_weights, attn_weights_entropy = tensor([3.9550, 3.1793, 2.8862, 3.1399, 3.3649, 2.8234, 3.3864, 2.5917],
       device='cuda:0')
2023-11-24 19:36:08,454 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3443, 5.0235, 4.6849, 5.1864], device='cuda:0')
2023-11-24 19:36:09,811 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.4.encoder.layers.2.self_attn_weights, attn_weights_entropy = tensor([3.1422, 3.9564, 3.6798, 3.1156], device='cuda:0')
2023-11-24 19:36:10,483 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.8020, 4.9466, 5.0592, 4.8872], device='cuda:0')
2023-11-24 19:36:12,483 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([5.3296, 4.9957, 4.5977, 5.1454], device='cuda:0')
2023-11-24 19:36:13,280 INFO [zipformer.py:1873] (0/4) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.7947, 4.9420, 5.0744, 4.8974], device='cuda:0')
2023-11-24 19:36:16,570 INFO [train_asr.py:1253] (0/4) Epoch 38, validation: loss=0.05758, simple_loss=0.05072, pruned_loss=0.005057, audio_tagging_loss=0.02716, over 4681554.00 frames. 
2023-11-24 19:36:16,571 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 19:36:20,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2965873.3333333335, ans=0.2
2023-11-24 19:36:32,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2965940.0, ans=0.0
2023-11-24 19:36:37,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444900
2023-11-24 19:36:53,673 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2966073.3333333335, ans=0.0
2023-11-24 19:37:18,860 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 50, loss[loss=0.04665, simple_loss=0.05029, pruned_loss=0.005533, audio_tagging_loss=0.01597, over 14643.00 frames. ], tot_loss[loss=0.07301, simple_loss=0.08697, pruned_loss=0.012, audio_tagging_loss=0.01752, over 689308.53 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:37:18,966 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=2966206.6666666665, ans=0.125
2023-11-24 19:37:20,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2966206.6666666665, ans=0.0
2023-11-24 19:37:20,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=2966206.6666666665, ans=0.2
2023-11-24 19:37:30,282 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.55 vs. limit=22.5
2023-11-24 19:37:34,636 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:37:35,484 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 8.245e+01 9.441e+01 1.025e+02 1.118e+02 1.388e+02, threshold=2.050e+02, percent-clipped=0.0
2023-11-24 19:37:40,872 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 444950
2023-11-24 19:38:17,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2966473.3333333335, ans=0.2
2023-11-24 19:38:17,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=2966473.3333333335, ans=0.2
2023-11-24 19:38:21,601 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 100, loss[loss=0.07402, simple_loss=0.09095, pruned_loss=0.01088, audio_tagging_loss=0.01767, over 15851.00 frames. ], tot_loss[loss=0.07408, simple_loss=0.08984, pruned_loss=0.01268, audio_tagging_loss=0.01648, over 1211059.49 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:38:43,043 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445000
2023-11-24 19:38:48,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2966673.3333333335, ans=0.0
2023-11-24 19:38:51,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2966673.3333333335, ans=0.0
2023-11-24 19:38:59,557 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.97 vs. limit=15.0
2023-11-24 19:39:09,434 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2966740.0, ans=0.0
2023-11-24 19:39:14,603 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2966806.6666666665, ans=0.125
2023-11-24 19:39:18,378 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.15 vs. limit=15.0
2023-11-24 19:39:24,357 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 150, loss[loss=0.07014, simple_loss=0.08951, pruned_loss=0.01346, audio_tagging_loss=0.01192, over 15903.00 frames. ], tot_loss[loss=0.07327, simple_loss=0.09173, pruned_loss=0.0129, audio_tagging_loss=0.01451, over 1623998.07 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:39:35,203 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2966940.0, ans=0.0
2023-11-24 19:39:38,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2966940.0, ans=0.125
2023-11-24 19:39:39,715 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.190e+01 9.070e+01 9.567e+01 1.041e+02 1.259e+02, threshold=1.913e+02, percent-clipped=0.0
2023-11-24 19:39:44,639 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445050
2023-11-24 19:39:45,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=2966940.0, ans=0.2
2023-11-24 19:40:03,390 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2967073.3333333335, ans=0.125
2023-11-24 19:40:20,900 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.55 vs. limit=15.0
2023-11-24 19:40:22,017 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.06 vs. limit=22.5
2023-11-24 19:40:26,144 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 200, loss[loss=0.09926, simple_loss=0.1363, pruned_loss=0.02358, audio_tagging_loss=0.007501, over 15123.00 frames. ], tot_loss[loss=0.07202, simple_loss=0.09235, pruned_loss=0.01317, audio_tagging_loss=0.01268, over 1945627.04 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:40:29,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2967206.6666666665, ans=0.2
2023-11-24 19:40:48,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445100
2023-11-24 19:40:51,087 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2967340.0, ans=0.0
2023-11-24 19:40:51,481 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.28 vs. limit=15.0
2023-11-24 19:40:53,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2967340.0, ans=0.125
2023-11-24 19:41:28,679 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 250, loss[loss=0.05964, simple_loss=0.0782, pruned_loss=0.01033, audio_tagging_loss=0.0102, over 14732.00 frames. ], tot_loss[loss=0.07052, simple_loss=0.09207, pruned_loss=0.01301, audio_tagging_loss=0.01147, over 2188311.77 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:41:45,194 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.968e+01 8.613e+01 9.234e+01 1.003e+02 1.300e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 19:41:50,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445150
2023-11-24 19:41:50,866 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2967606.6666666665, ans=0.1
2023-11-24 19:42:24,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2967806.6666666665, ans=0.1
2023-11-24 19:42:31,605 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 300, loss[loss=0.07001, simple_loss=0.09766, pruned_loss=0.01394, audio_tagging_loss=0.007241, over 14911.00 frames. ], tot_loss[loss=0.06937, simple_loss=0.09162, pruned_loss=0.01284, audio_tagging_loss=0.01072, over 2381410.45 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:42:41,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2967873.3333333335, ans=0.125
2023-11-24 19:42:52,497 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445200
2023-11-24 19:43:34,422 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 350, loss[loss=0.06786, simple_loss=0.09043, pruned_loss=0.01127, audio_tagging_loss=0.01137, over 14790.00 frames. ], tot_loss[loss=0.06962, simple_loss=0.09257, pruned_loss=0.0132, audio_tagging_loss=0.01013, over 2533937.60 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:43:49,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2968273.3333333335, ans=0.2
2023-11-24 19:43:51,717 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.795e+01 8.774e+01 9.424e+01 1.013e+02 1.303e+02, threshold=1.885e+02, percent-clipped=0.0
2023-11-24 19:43:53,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2968273.3333333335, ans=0.125
2023-11-24 19:43:55,380 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445250
2023-11-24 19:44:24,785 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.06 vs. limit=15.0
2023-11-24 19:44:28,045 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=2968473.3333333335, ans=0.125
2023-11-24 19:44:36,728 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 400, loss[loss=0.06833, simple_loss=0.09585, pruned_loss=0.01115, audio_tagging_loss=0.009252, over 15462.00 frames. ], tot_loss[loss=0.06894, simple_loss=0.09178, pruned_loss=0.01312, audio_tagging_loss=0.009924, over 2640299.36 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:44:38,477 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.98 vs. limit=22.5
2023-11-24 19:44:41,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2968540.0, ans=0.0
2023-11-24 19:44:44,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2968540.0, ans=0.0
2023-11-24 19:44:54,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2968606.6666666665, ans=0.1
2023-11-24 19:44:58,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445300
2023-11-24 19:45:06,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2968673.3333333335, ans=0.125
2023-11-24 19:45:10,450 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2968673.3333333335, ans=0.125
2023-11-24 19:45:10,667 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=2968673.3333333335, ans=22.5
2023-11-24 19:45:18,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2968740.0, ans=0.1
2023-11-24 19:45:21,250 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=2968740.0, ans=0.0
2023-11-24 19:45:39,741 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 450, loss[loss=0.09538, simple_loss=0.1411, pruned_loss=0.01648, audio_tagging_loss=0.008331, over 15906.00 frames. ], tot_loss[loss=0.06902, simple_loss=0.09221, pruned_loss=0.01321, audio_tagging_loss=0.0097, over 2732262.15 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:45:41,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.63 vs. limit=15.0
2023-11-24 19:45:43,628 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2968873.3333333335, ans=0.125
2023-11-24 19:45:46,424 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2968873.3333333335, ans=0.07
2023-11-24 19:45:56,765 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.914e+01 8.512e+01 9.246e+01 9.935e+01 1.248e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 19:46:00,407 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445350
2023-11-24 19:46:00,478 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2968940.0, ans=0.125
2023-11-24 19:46:05,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=2969006.6666666665, ans=0.2
2023-11-24 19:46:07,736 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=2969006.6666666665, ans=0.2
2023-11-24 19:46:10,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2969006.6666666665, ans=0.1
2023-11-24 19:46:28,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=2969140.0, ans=0.125
2023-11-24 19:46:32,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2969140.0, ans=0.1
2023-11-24 19:46:41,819 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 500, loss[loss=0.06534, simple_loss=0.09579, pruned_loss=0.009774, audio_tagging_loss=0.00767, over 16086.00 frames. ], tot_loss[loss=0.06875, simple_loss=0.09202, pruned_loss=0.01327, audio_tagging_loss=0.009463, over 2798751.88 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:46:59,715 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2969273.3333333335, ans=0.125
2023-11-24 19:47:03,044 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445400
2023-11-24 19:47:05,052 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=5.80 vs. limit=15.0
2023-11-24 19:47:15,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2969340.0, ans=0.125
2023-11-24 19:47:43,320 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2969540.0, ans=0.125
2023-11-24 19:47:44,343 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 550, loss[loss=0.07101, simple_loss=0.1042, pruned_loss=0.01257, audio_tagging_loss=0.006333, over 14785.00 frames. ], tot_loss[loss=0.06824, simple_loss=0.09161, pruned_loss=0.0131, audio_tagging_loss=0.009331, over 2857484.36 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:47:46,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2969540.0, ans=0.125
2023-11-24 19:47:51,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2969540.0, ans=0.0
2023-11-24 19:48:04,005 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.380e+01 8.415e+01 9.096e+01 9.896e+01 1.420e+02, threshold=1.819e+02, percent-clipped=0.0
2023-11-24 19:48:06,518 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445450
2023-11-24 19:48:12,006 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2969673.3333333335, ans=0.125
2023-11-24 19:48:24,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2969740.0, ans=0.2
2023-11-24 19:48:42,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2969806.6666666665, ans=0.125
2023-11-24 19:48:47,452 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 600, loss[loss=0.05835, simple_loss=0.07331, pruned_loss=0.01292, audio_tagging_loss=0.008777, over 15029.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.09086, pruned_loss=0.01281, audio_tagging_loss=0.009214, over 2902025.03 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:48:48,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2969873.3333333335, ans=0.0
2023-11-24 19:49:05,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=2969940.0, ans=0.0
2023-11-24 19:49:08,917 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445500
2023-11-24 19:49:11,404 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2970006.6666666665, ans=0.125
2023-11-24 19:49:25,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2970073.3333333335, ans=0.125
2023-11-24 19:49:31,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2970073.3333333335, ans=0.125
2023-11-24 19:49:49,973 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 650, loss[loss=0.05035, simple_loss=0.05824, pruned_loss=0.009728, audio_tagging_loss=0.0115, over 15395.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09062, pruned_loss=0.01296, audio_tagging_loss=0.0092, over 2930103.73 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:49:55,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2970206.6666666665, ans=0.2
2023-11-24 19:50:04,835 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2970273.3333333335, ans=0.0
2023-11-24 19:50:07,965 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.028e+01 8.559e+01 9.252e+01 1.019e+02 1.352e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 19:50:08,303 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 19:50:10,980 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445550
2023-11-24 19:50:14,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2970340.0, ans=0.125
2023-11-24 19:50:17,488 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.52 vs. limit=22.5
2023-11-24 19:50:43,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.80 vs. limit=15.0
2023-11-24 19:50:44,546 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=6.40 vs. limit=15.0
2023-11-24 19:50:45,328 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2970473.3333333335, ans=0.125
2023-11-24 19:50:48,151 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.73 vs. limit=15.0
2023-11-24 19:50:51,220 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 700, loss[loss=0.06151, simple_loss=0.08364, pruned_loss=0.009219, audio_tagging_loss=0.01047, over 15465.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09149, pruned_loss=0.01302, audio_tagging_loss=0.009145, over 2951259.77 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:50:54,156 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=7.37 vs. limit=15.0
2023-11-24 19:50:55,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=2970540.0, ans=0.125
2023-11-24 19:51:00,745 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2970540.0, ans=0.125
2023-11-24 19:51:07,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2970606.6666666665, ans=0.0
2023-11-24 19:51:13,053 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445600
2023-11-24 19:51:23,161 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2970673.3333333335, ans=0.125
2023-11-24 19:51:45,238 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2970806.6666666665, ans=0.2
2023-11-24 19:51:55,298 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 750, loss[loss=0.05206, simple_loss=0.0599, pruned_loss=0.009224, audio_tagging_loss=0.01289, over 15193.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.09117, pruned_loss=0.01284, audio_tagging_loss=0.009188, over 2973758.84 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:51:58,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2970873.3333333335, ans=0.125
2023-11-24 19:52:06,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=2970940.0, ans=0.125
2023-11-24 19:52:14,200 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.77 vs. limit=5.0
2023-11-24 19:52:14,360 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.538e+01 8.682e+01 9.156e+01 1.004e+02 1.177e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 19:52:16,877 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445650
2023-11-24 19:52:23,035 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2971006.6666666665, ans=0.2
2023-11-24 19:52:26,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2971006.6666666665, ans=0.125
2023-11-24 19:52:29,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2971006.6666666665, ans=0.2
2023-11-24 19:52:48,747 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.93 vs. limit=22.5
2023-11-24 19:52:52,424 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.37 vs. limit=22.5
2023-11-24 19:52:58,220 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 800, loss[loss=0.06001, simple_loss=0.08054, pruned_loss=0.01212, audio_tagging_loss=0.007612, over 15255.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.09168, pruned_loss=0.01288, audio_tagging_loss=0.009141, over 2989261.85 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:53:12,493 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0
2023-11-24 19:53:19,213 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445700
2023-11-24 19:53:28,927 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2971340.0, ans=0.2
2023-11-24 19:53:55,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=2971473.3333333335, ans=0.95
2023-11-24 19:54:00,950 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 850, loss[loss=0.06796, simple_loss=0.09643, pruned_loss=0.01402, audio_tagging_loss=0.005727, over 14622.00 frames. ], tot_loss[loss=0.06794, simple_loss=0.09168, pruned_loss=0.01292, audio_tagging_loss=0.009186, over 3004282.73 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 19:54:03,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=2971540.0, ans=0.125
2023-11-24 19:54:05,865 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2971540.0, ans=0.1
2023-11-24 19:54:19,783 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.117e+01 8.517e+01 9.234e+01 9.613e+01 1.155e+02, threshold=1.847e+02, percent-clipped=0.0
2023-11-24 19:54:22,328 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445750
2023-11-24 19:54:44,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2971740.0, ans=0.0
2023-11-24 19:54:47,660 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2971740.0, ans=0.0
2023-11-24 19:54:56,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2971806.6666666665, ans=0.0
2023-11-24 19:55:03,976 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 900, loss[loss=0.05815, simple_loss=0.08018, pruned_loss=0.008494, audio_tagging_loss=0.009563, over 15677.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09123, pruned_loss=0.01294, audio_tagging_loss=0.009343, over 3009430.86 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:55:11,656 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2971873.3333333335, ans=0.125
2023-11-24 19:55:12,215 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=11.71 vs. limit=15.0
2023-11-24 19:55:20,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2971940.0, ans=0.125
2023-11-24 19:55:26,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445800
2023-11-24 19:55:31,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=2972006.6666666665, ans=0.2
2023-11-24 19:55:34,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2972006.6666666665, ans=0.2
2023-11-24 19:56:07,369 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 950, loss[loss=0.05837, simple_loss=0.07965, pruned_loss=0.01083, audio_tagging_loss=0.007713, over 15705.00 frames. ], tot_loss[loss=0.06831, simple_loss=0.09187, pruned_loss=0.01319, audio_tagging_loss=0.009183, over 3018739.24 frames. ], batch size: 60, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:56:22,881 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=7.59 vs. limit=10.0
2023-11-24 19:56:26,836 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.519e+01 9.243e+01 9.717e+01 1.374e+02, threshold=1.849e+02, percent-clipped=0.0
2023-11-24 19:56:28,178 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445850
2023-11-24 19:56:28,443 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2972273.3333333335, ans=0.0
2023-11-24 19:56:29,498 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2972273.3333333335, ans=0.1
2023-11-24 19:56:37,271 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2972340.0, ans=0.125
2023-11-24 19:56:56,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2972473.3333333335, ans=0.0
2023-11-24 19:57:09,563 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1000, loss[loss=0.081, simple_loss=0.1119, pruned_loss=0.01901, audio_tagging_loss=0.006043, over 15872.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09133, pruned_loss=0.0131, audio_tagging_loss=0.009054, over 3022652.71 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:57:21,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2972606.6666666665, ans=0.035
2023-11-24 19:57:23,541 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2972606.6666666665, ans=0.0
2023-11-24 19:57:30,768 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445900
2023-11-24 19:57:35,506 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5Y6u9AlD9S0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:58:01,020 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=7.06 vs. limit=15.0
2023-11-24 19:58:05,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2972806.6666666665, ans=0.125
2023-11-24 19:58:11,864 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1050, loss[loss=0.05941, simple_loss=0.07764, pruned_loss=0.01161, audio_tagging_loss=0.008974, over 15118.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09135, pruned_loss=0.01309, audio_tagging_loss=0.008916, over 3025322.72 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:58:15,962 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2972873.3333333335, ans=0.1
2023-11-24 19:58:22,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=2972873.3333333335, ans=10.0
2023-11-24 19:58:32,041 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.010e+01 8.733e+01 9.359e+01 1.012e+02 1.444e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-24 19:58:33,427 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 445950
2023-11-24 19:58:41,147 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=15.05 vs. limit=22.5
2023-11-24 19:58:47,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=2973006.6666666665, ans=0.0
2023-11-24 19:59:14,420 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1100, loss[loss=0.09552, simple_loss=0.133, pruned_loss=0.02125, audio_tagging_loss=0.007782, over 15659.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09164, pruned_loss=0.0131, audio_tagging_loss=0.008778, over 3029944.37 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 19:59:17,457 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/AWHnJAqurec_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 19:59:32,562 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2973273.3333333335, ans=0.2
2023-11-24 19:59:35,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446000
2023-11-24 19:59:41,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=10.82 vs. limit=15.0
2023-11-24 19:59:53,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2973406.6666666665, ans=0.0
2023-11-24 19:59:55,392 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2973406.6666666665, ans=0.0
2023-11-24 20:00:16,981 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1150, loss[loss=0.05861, simple_loss=0.08866, pruned_loss=0.009135, audio_tagging_loss=0.00514, over 15991.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09132, pruned_loss=0.0131, audio_tagging_loss=0.008774, over 3038079.94 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:00:17,983 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.04 vs. limit=15.0
2023-11-24 20:00:33,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2973606.6666666665, ans=0.1
2023-11-24 20:00:37,598 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.661e+01 8.560e+01 9.055e+01 9.595e+01 1.732e+02, threshold=1.811e+02, percent-clipped=0.0
2023-11-24 20:00:38,911 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446050
2023-11-24 20:01:19,754 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1200, loss[loss=0.04677, simple_loss=0.05426, pruned_loss=0.00761, audio_tagging_loss=0.01202, over 13382.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09078, pruned_loss=0.01294, audio_tagging_loss=0.008764, over 3039681.10 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:01:20,499 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.19 vs. limit=15.0
2023-11-24 20:01:27,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2973873.3333333335, ans=0.2
2023-11-24 20:01:41,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446100
2023-11-24 20:01:43,717 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer2.prob, batch_count=2974006.6666666665, ans=0.125
2023-11-24 20:01:51,371 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.32 vs. limit=6.0
2023-11-24 20:01:54,534 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2974006.6666666665, ans=0.09899494936611666
2023-11-24 20:02:02,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2974073.3333333335, ans=0.1
2023-11-24 20:02:03,479 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2974073.3333333335, ans=0.125
2023-11-24 20:02:17,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=2974140.0, ans=0.09899494936611666
2023-11-24 20:02:20,093 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=2974140.0, ans=0.0
2023-11-24 20:02:20,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2974140.0, ans=0.0
2023-11-24 20:02:20,587 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=8.79 vs. limit=15.0
2023-11-24 20:02:22,291 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1250, loss[loss=0.05835, simple_loss=0.08289, pruned_loss=0.00845, audio_tagging_loss=0.008452, over 14557.00 frames. ], tot_loss[loss=0.0673, simple_loss=0.09119, pruned_loss=0.01303, audio_tagging_loss=0.008672, over 3032272.17 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:02:26,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=2974206.6666666665, ans=0.2
2023-11-24 20:02:29,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2974206.6666666665, ans=0.0
2023-11-24 20:02:31,227 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2974206.6666666665, ans=0.125
2023-11-24 20:02:42,694 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.011e+01 8.510e+01 9.127e+01 9.845e+01 1.206e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-24 20:02:42,856 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446150
2023-11-24 20:02:57,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.whiten.whitening_limit, batch_count=2974340.0, ans=12.0
2023-11-24 20:03:15,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2974473.3333333335, ans=0.125
2023-11-24 20:03:24,255 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1300, loss[loss=0.07872, simple_loss=0.1106, pruned_loss=0.01425, audio_tagging_loss=0.009164, over 15431.00 frames. ], tot_loss[loss=0.06786, simple_loss=0.0923, pruned_loss=0.01309, audio_tagging_loss=0.008617, over 3033709.74 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:03:24,852 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.34 vs. limit=22.5
2023-11-24 20:03:46,023 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446200
2023-11-24 20:03:46,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2974606.6666666665, ans=0.1
2023-11-24 20:03:57,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2974673.3333333335, ans=0.125
2023-11-24 20:04:11,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2974740.0, ans=0.125
2023-11-24 20:04:26,385 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1350, loss[loss=0.083, simple_loss=0.1188, pruned_loss=0.01598, audio_tagging_loss=0.007597, over 15806.00 frames. ], tot_loss[loss=0.06799, simple_loss=0.09232, pruned_loss=0.01314, audio_tagging_loss=0.008692, over 3039651.39 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:04:38,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=2974940.0, ans=0.0
2023-11-24 20:04:42,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2974940.0, ans=0.125
2023-11-24 20:04:44,680 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0
2023-11-24 20:04:48,067 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.451e+01 8.554e+01 9.036e+01 9.639e+01 2.297e+02, threshold=1.807e+02, percent-clipped=1.0
2023-11-24 20:04:48,213 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446250
2023-11-24 20:05:10,675 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XdmbboqRBmQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:05:13,803 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=2975073.3333333335, ans=0.035
2023-11-24 20:05:19,944 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.32 vs. limit=15.0
2023-11-24 20:05:29,115 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1400, loss[loss=0.0603, simple_loss=0.07716, pruned_loss=0.01249, audio_tagging_loss=0.009231, over 16375.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09147, pruned_loss=0.0129, audio_tagging_loss=0.008756, over 3037541.94 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:05:36,542 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.29 vs. limit=15.0
2023-11-24 20:05:39,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2975206.6666666665, ans=0.125
2023-11-24 20:05:49,958 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446300
2023-11-24 20:06:16,631 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=2975406.6666666665, ans=0.125
2023-11-24 20:06:31,106 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1450, loss[loss=0.0559, simple_loss=0.07282, pruned_loss=0.008931, audio_tagging_loss=0.01055, over 13774.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.0916, pruned_loss=0.0131, audio_tagging_loss=0.008819, over 3044814.79 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:06:48,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2975606.6666666665, ans=0.1
2023-11-24 20:06:51,829 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.634e+01 9.284e+01 1.040e+02 1.664e+02, threshold=1.857e+02, percent-clipped=0.0
2023-11-24 20:06:51,979 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446350
2023-11-24 20:06:59,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2975673.3333333335, ans=0.125
2023-11-24 20:07:06,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2975673.3333333335, ans=0.1
2023-11-24 20:07:07,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=2975740.0, ans=0.0
2023-11-24 20:07:08,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2975740.0, ans=0.0
2023-11-24 20:07:15,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2975740.0, ans=0.1
2023-11-24 20:07:21,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=2975806.6666666665, ans=0.125
2023-11-24 20:07:33,038 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1500, loss[loss=0.05885, simple_loss=0.08153, pruned_loss=0.01009, audio_tagging_loss=0.007997, over 16054.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09114, pruned_loss=0.01302, audio_tagging_loss=0.008993, over 3044707.76 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:07:54,874 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446400
2023-11-24 20:08:03,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=2976006.6666666665, ans=0.125
2023-11-24 20:08:07,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=2976006.6666666665, ans=0.125
2023-11-24 20:08:07,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2976006.6666666665, ans=0.0
2023-11-24 20:08:20,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2976073.3333333335, ans=0.1
2023-11-24 20:08:20,571 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2976073.3333333335, ans=0.125
2023-11-24 20:08:29,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2976140.0, ans=0.125
2023-11-24 20:08:35,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2976206.6666666665, ans=0.125
2023-11-24 20:08:35,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2976206.6666666665, ans=0.125
2023-11-24 20:08:36,377 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1550, loss[loss=0.068, simple_loss=0.09136, pruned_loss=0.01218, audio_tagging_loss=0.01014, over 14455.00 frames. ], tot_loss[loss=0.0678, simple_loss=0.09155, pruned_loss=0.01306, audio_tagging_loss=0.008971, over 3040959.06 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:08:39,090 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2976206.6666666665, ans=0.2
2023-11-24 20:08:47,903 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2976273.3333333335, ans=0.07
2023-11-24 20:08:57,005 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446450
2023-11-24 20:08:57,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2976273.3333333335, ans=0.0
2023-11-24 20:08:58,031 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.449e+01 8.670e+01 9.483e+01 1.014e+02 1.933e+02, threshold=1.897e+02, percent-clipped=2.0
2023-11-24 20:09:15,924 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module1.balancer2.prob, batch_count=2976406.6666666665, ans=0.125
2023-11-24 20:09:37,988 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1600, loss[loss=0.05743, simple_loss=0.0793, pruned_loss=0.009181, audio_tagging_loss=0.008598, over 15215.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09008, pruned_loss=0.01279, audio_tagging_loss=0.009159, over 3043130.94 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:09:58,763 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446500
2023-11-24 20:10:00,092 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:10:03,510 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.48 vs. limit=15.0
2023-11-24 20:10:06,624 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2976673.3333333335, ans=0.125
2023-11-24 20:10:17,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2976740.0, ans=0.2
2023-11-24 20:10:23,071 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=2976740.0, ans=0.2
2023-11-24 20:10:28,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2976806.6666666665, ans=0.125
2023-11-24 20:10:31,904 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.whiten, num_groups=1, num_channels=384, metric=5.65 vs. limit=12.0
2023-11-24 20:10:36,042 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2976806.6666666665, ans=0.125
2023-11-24 20:10:39,346 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1650, loss[loss=0.08733, simple_loss=0.1274, pruned_loss=0.0173, audio_tagging_loss=0.006303, over 15868.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09005, pruned_loss=0.01293, audio_tagging_loss=0.009128, over 3042587.04 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:10:43,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2976873.3333333335, ans=0.125
2023-11-24 20:10:57,772 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.00 vs. limit=10.0
2023-11-24 20:10:59,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2976940.0, ans=0.1
2023-11-24 20:11:01,276 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446550
2023-11-24 20:11:02,255 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.314e+01 8.733e+01 9.209e+01 9.885e+01 1.194e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 20:11:20,917 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=2977073.3333333335, ans=0.0
2023-11-24 20:11:40,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten.whitening_limit, batch_count=2977140.0, ans=15.0
2023-11-24 20:11:41,961 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1700, loss[loss=0.06873, simple_loss=0.09463, pruned_loss=0.01323, audio_tagging_loss=0.008186, over 14863.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.08998, pruned_loss=0.01292, audio_tagging_loss=0.009114, over 3045981.17 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:12:03,401 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446600
2023-11-24 20:12:06,633 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.30 vs. limit=15.0
2023-11-24 20:12:12,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2977340.0, ans=0.0
2023-11-24 20:12:18,058 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=2977406.6666666665, ans=0.0
2023-11-24 20:12:20,990 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:12:28,577 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=2977406.6666666665, ans=0.2
2023-11-24 20:12:45,069 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1750, loss[loss=0.05762, simple_loss=0.07469, pruned_loss=0.01156, audio_tagging_loss=0.008712, over 15209.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.08997, pruned_loss=0.01289, audio_tagging_loss=0.009113, over 3046796.14 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:13:04,224 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2977606.6666666665, ans=0.025
2023-11-24 20:13:05,305 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446650
2023-11-24 20:13:08,150 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.273e+01 8.455e+01 9.155e+01 9.735e+01 1.313e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 20:13:15,886 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2977673.3333333335, ans=0.0
2023-11-24 20:13:30,109 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2977740.0, ans=0.0
2023-11-24 20:13:45,564 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2977873.3333333335, ans=0.125
2023-11-24 20:13:45,625 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_na.min_abs, batch_count=2977873.3333333335, ans=0.02
2023-11-24 20:13:46,511 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1800, loss[loss=0.05056, simple_loss=0.06345, pruned_loss=0.009088, audio_tagging_loss=0.009744, over 14549.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.09057, pruned_loss=0.01286, audio_tagging_loss=0.008995, over 3042139.30 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:13:54,357 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2977873.3333333335, ans=0.125
2023-11-24 20:13:56,875 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:13:58,501 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2977940.0, ans=0.0
2023-11-24 20:14:02,852 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.90 vs. limit=15.0
2023-11-24 20:14:07,974 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446700
2023-11-24 20:14:16,597 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2978006.6666666665, ans=0.125
2023-11-24 20:14:22,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2978006.6666666665, ans=0.125
2023-11-24 20:14:27,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2978073.3333333335, ans=0.0
2023-11-24 20:14:27,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2978073.3333333335, ans=0.125
2023-11-24 20:14:49,188 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1850, loss[loss=0.07647, simple_loss=0.1103, pruned_loss=0.01592, audio_tagging_loss=0.005383, over 15375.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.0914, pruned_loss=0.01287, audio_tagging_loss=0.008896, over 3046021.66 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:14:53,433 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.95 vs. limit=15.0
2023-11-24 20:15:01,311 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2978273.3333333335, ans=0.125
2023-11-24 20:15:05,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.75 vs. limit=10.0
2023-11-24 20:15:07,650 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.50 vs. limit=15.0
2023-11-24 20:15:10,452 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446750
2023-11-24 20:15:12,637 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.316e+01 8.609e+01 9.248e+01 1.012e+02 1.189e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 20:15:13,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2978340.0, ans=0.125
2023-11-24 20:15:34,295 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2978406.6666666665, ans=0.2
2023-11-24 20:15:42,448 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2978473.3333333335, ans=0.2
2023-11-24 20:15:51,164 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1900, loss[loss=0.05125, simple_loss=0.06144, pruned_loss=0.009489, audio_tagging_loss=0.01105, over 17680.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09108, pruned_loss=0.01285, audio_tagging_loss=0.008782, over 3056333.53 frames. ], batch size: 68, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:16:11,294 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446800
2023-11-24 20:16:14,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2978673.3333333335, ans=0.125
2023-11-24 20:16:21,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2978673.3333333335, ans=0.125
2023-11-24 20:16:52,751 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 1950, loss[loss=0.06436, simple_loss=0.08666, pruned_loss=0.01131, audio_tagging_loss=0.009722, over 14382.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.08962, pruned_loss=0.0128, audio_tagging_loss=0.008842, over 3048509.04 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 4.0
2023-11-24 20:16:52,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2978873.3333333335, ans=0.0
2023-11-24 20:17:09,502 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2978940.0, ans=0.07
2023-11-24 20:17:13,903 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446850
2023-11-24 20:17:17,337 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.207e+01 8.609e+01 9.375e+01 1.004e+02 1.642e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 20:17:17,680 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2979006.6666666665, ans=0.0
2023-11-24 20:17:18,790 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2979006.6666666665, ans=0.5
2023-11-24 20:17:40,793 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.47 vs. limit=15.0
2023-11-24 20:17:55,520 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2000, loss[loss=0.07489, simple_loss=0.1048, pruned_loss=0.01449, audio_tagging_loss=0.008014, over 14072.00 frames. ], tot_loss[loss=0.06656, simple_loss=0.08962, pruned_loss=0.0129, audio_tagging_loss=0.008847, over 3045599.01 frames. ], batch size: 53, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:17:55,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2979206.6666666665, ans=0.0
2023-11-24 20:18:17,327 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446900
2023-11-24 20:18:49,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2979473.3333333335, ans=0.1
2023-11-24 20:18:57,598 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2050, loss[loss=0.07639, simple_loss=0.1093, pruned_loss=0.01272, audio_tagging_loss=0.009018, over 15355.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.09039, pruned_loss=0.0129, audio_tagging_loss=0.008751, over 3048199.56 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:19:02,021 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=2979540.0, ans=0.0
2023-11-24 20:19:08,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2979540.0, ans=0.0
2023-11-24 20:19:19,236 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 446950
2023-11-24 20:19:22,628 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.128e+01 8.521e+01 9.047e+01 9.602e+01 1.413e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 20:19:43,253 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=2979740.0, ans=0.2
2023-11-24 20:19:43,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2979740.0, ans=0.125
2023-11-24 20:19:43,657 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.90 vs. limit=15.0
2023-11-24 20:20:00,618 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2100, loss[loss=0.05525, simple_loss=0.07574, pruned_loss=0.008108, audio_tagging_loss=0.00927, over 14453.00 frames. ], tot_loss[loss=0.06673, simple_loss=0.09056, pruned_loss=0.01276, audio_tagging_loss=0.008694, over 3049669.58 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:20:22,475 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447000
2023-11-24 20:20:42,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2980073.3333333335, ans=0.125
2023-11-24 20:21:03,524 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2150, loss[loss=0.07838, simple_loss=0.1013, pruned_loss=0.015, audio_tagging_loss=0.01274, over 15814.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.09032, pruned_loss=0.01272, audio_tagging_loss=0.008739, over 3047890.25 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:21:13,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2980206.6666666665, ans=0.0
2023-11-24 20:21:13,826 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2980206.6666666665, ans=0.0
2023-11-24 20:21:24,848 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447050
2023-11-24 20:21:28,342 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.268e+01 8.688e+01 9.386e+01 1.033e+02 1.815e+02, threshold=1.877e+02, percent-clipped=1.0
2023-11-24 20:21:40,964 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XkQ8YVd8u38_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:21:48,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2980406.6666666665, ans=0.125
2023-11-24 20:21:50,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2980406.6666666665, ans=0.125
2023-11-24 20:21:56,648 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2980473.3333333335, ans=0.125
2023-11-24 20:22:00,012 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2980473.3333333335, ans=0.1
2023-11-24 20:22:05,667 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2200, loss[loss=0.04932, simple_loss=0.06068, pruned_loss=0.007449, audio_tagging_loss=0.01153, over 14504.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09098, pruned_loss=0.01294, audio_tagging_loss=0.00876, over 3052759.44 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:22:23,610 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=2980606.6666666665, ans=0.5
2023-11-24 20:22:26,954 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447100
2023-11-24 20:22:38,779 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=13.00 vs. limit=15.0
2023-11-24 20:23:01,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2980806.6666666665, ans=0.0
2023-11-24 20:23:07,833 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2250, loss[loss=0.06893, simple_loss=0.08862, pruned_loss=0.01461, audio_tagging_loss=0.01, over 16855.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09197, pruned_loss=0.01304, audio_tagging_loss=0.008788, over 3053062.77 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:23:08,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=2980873.3333333335, ans=0.0
2023-11-24 20:23:17,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=2980873.3333333335, ans=0.5
2023-11-24 20:23:29,842 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447150
2023-11-24 20:23:32,809 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.20 vs. limit=15.0
2023-11-24 20:23:33,355 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.261e+01 8.657e+01 9.252e+01 1.018e+02 2.312e+02, threshold=1.850e+02, percent-clipped=2.0
2023-11-24 20:23:33,647 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2981006.6666666665, ans=0.125
2023-11-24 20:23:47,146 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2981073.3333333335, ans=0.1
2023-11-24 20:24:03,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=2981140.0, ans=0.04949747468305833
2023-11-24 20:24:11,234 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2300, loss[loss=0.05318, simple_loss=0.06938, pruned_loss=0.007651, audio_tagging_loss=0.01084, over 14586.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09142, pruned_loss=0.01281, audio_tagging_loss=0.008827, over 3059609.37 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:24:20,246 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2981206.6666666665, ans=0.125
2023-11-24 20:24:20,751 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=16.12 vs. limit=22.5
2023-11-24 20:24:22,581 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2981273.3333333335, ans=0.125
2023-11-24 20:24:32,682 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447200
2023-11-24 20:24:40,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=2981340.0, ans=0.07
2023-11-24 20:24:46,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2981340.0, ans=0.125
2023-11-24 20:25:05,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2981473.3333333335, ans=0.125
2023-11-24 20:25:06,750 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/mx9RcUz8sr0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:25:10,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=2981473.3333333335, ans=0.125
2023-11-24 20:25:14,014 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2350, loss[loss=0.06987, simple_loss=0.09678, pruned_loss=0.01405, audio_tagging_loss=0.007435, over 15215.00 frames. ], tot_loss[loss=0.06779, simple_loss=0.09184, pruned_loss=0.01292, audio_tagging_loss=0.008947, over 3061512.22 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 8.0
2023-11-24 20:25:14,122 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2981540.0, ans=0.125
2023-11-24 20:25:15,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2981540.0, ans=0.1
2023-11-24 20:25:26,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2981606.6666666665, ans=0.125
2023-11-24 20:25:30,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=2981606.6666666665, ans=0.125
2023-11-24 20:25:34,009 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=2981606.6666666665, ans=0.5
2023-11-24 20:25:35,001 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447250
2023-11-24 20:25:39,007 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.011e+01 8.625e+01 9.128e+01 9.764e+01 1.329e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 20:25:48,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2981673.3333333335, ans=0.0
2023-11-24 20:25:51,057 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=2981740.0, ans=0.0
2023-11-24 20:25:55,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2981740.0, ans=0.125
2023-11-24 20:26:12,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2981806.6666666665, ans=0.125
2023-11-24 20:26:16,231 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2400, loss[loss=0.06651, simple_loss=0.1019, pruned_loss=0.008845, audio_tagging_loss=0.006734, over 15058.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09077, pruned_loss=0.01267, audio_tagging_loss=0.009128, over 3060839.17 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:26:21,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2981873.3333333335, ans=0.125
2023-11-24 20:26:38,249 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447300
2023-11-24 20:26:45,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2982006.6666666665, ans=0.125
2023-11-24 20:26:58,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2982073.3333333335, ans=0.125
2023-11-24 20:27:16,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2982140.0, ans=0.1
2023-11-24 20:27:18,571 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2450, loss[loss=0.07007, simple_loss=0.08845, pruned_loss=0.01166, audio_tagging_loss=0.01418, over 14319.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.0907, pruned_loss=0.01275, audio_tagging_loss=0.009215, over 3057765.57 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:27:40,050 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447350
2023-11-24 20:27:44,106 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.056e+01 8.597e+01 9.419e+01 1.015e+02 1.292e+02, threshold=1.884e+02, percent-clipped=0.0
2023-11-24 20:27:44,458 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=2982340.0, ans=0.0
2023-11-24 20:27:55,096 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2982406.6666666665, ans=0.125
2023-11-24 20:27:56,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2982406.6666666665, ans=0.125
2023-11-24 20:28:21,323 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2500, loss[loss=0.06307, simple_loss=0.07562, pruned_loss=0.01215, audio_tagging_loss=0.0131, over 14857.00 frames. ], tot_loss[loss=0.06752, simple_loss=0.09102, pruned_loss=0.0129, audio_tagging_loss=0.009114, over 3053442.26 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:28:36,523 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2982606.6666666665, ans=0.0
2023-11-24 20:28:40,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2982606.6666666665, ans=0.125
2023-11-24 20:28:41,313 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2982606.6666666665, ans=0.125
2023-11-24 20:28:42,238 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447400
2023-11-24 20:29:03,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2982740.0, ans=0.0
2023-11-24 20:29:05,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=2982740.0, ans=0.125
2023-11-24 20:29:07,179 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=2982740.0, ans=0.125
2023-11-24 20:29:13,534 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.74 vs. limit=15.0
2023-11-24 20:29:21,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2982806.6666666665, ans=0.0
2023-11-24 20:29:23,918 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2550, loss[loss=0.06655, simple_loss=0.09104, pruned_loss=0.01185, audio_tagging_loss=0.009171, over 14990.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.0904, pruned_loss=0.01279, audio_tagging_loss=0.009081, over 3046337.57 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:29:29,367 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.65 vs. limit=15.0
2023-11-24 20:29:31,279 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2982873.3333333335, ans=0.125
2023-11-24 20:29:32,440 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2982873.3333333335, ans=0.125
2023-11-24 20:29:44,542 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447450
2023-11-24 20:29:48,491 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.648e+01 8.553e+01 9.176e+01 1.003e+02 1.865e+02, threshold=1.835e+02, percent-clipped=0.0
2023-11-24 20:30:07,347 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:30:17,418 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:30:21,068 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=2983140.0, ans=0.0
2023-11-24 20:30:23,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2983140.0, ans=0.125
2023-11-24 20:30:25,789 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2600, loss[loss=0.07232, simple_loss=0.1, pruned_loss=0.01359, audio_tagging_loss=0.008714, over 15295.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.09051, pruned_loss=0.01268, audio_tagging_loss=0.008893, over 3042490.14 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:30:38,775 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2983273.3333333335, ans=0.125
2023-11-24 20:30:48,117 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447500
2023-11-24 20:31:03,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=2983406.6666666665, ans=0.0
2023-11-24 20:31:03,199 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2983406.6666666665, ans=0.125
2023-11-24 20:31:05,509 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=2983406.6666666665, ans=0.2
2023-11-24 20:31:12,870 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=2983406.6666666665, ans=0.125
2023-11-24 20:31:27,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2983473.3333333335, ans=0.1
2023-11-24 20:31:29,459 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2650, loss[loss=0.05808, simple_loss=0.07924, pruned_loss=0.01005, audio_tagging_loss=0.008409, over 14950.00 frames. ], tot_loss[loss=0.0664, simple_loss=0.08984, pruned_loss=0.01257, audio_tagging_loss=0.008912, over 3042325.96 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:31:33,540 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.85 vs. limit=12.0
2023-11-24 20:31:39,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2983540.0, ans=0.1
2023-11-24 20:31:50,189 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447550
2023-11-24 20:31:50,397 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:31:53,539 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.521e+01 8.637e+01 9.375e+01 1.000e+02 1.273e+02, threshold=1.875e+02, percent-clipped=0.0
2023-11-24 20:32:06,817 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.79 vs. limit=22.5
2023-11-24 20:32:12,019 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.04 vs. limit=15.0
2023-11-24 20:32:13,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2983740.0, ans=0.1
2023-11-24 20:32:22,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2983806.6666666665, ans=0.1
2023-11-24 20:32:22,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2983806.6666666665, ans=0.125
2023-11-24 20:32:24,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2983806.6666666665, ans=0.125
2023-11-24 20:32:24,940 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=12.66 vs. limit=15.0
2023-11-24 20:32:30,769 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2700, loss[loss=0.08404, simple_loss=0.1227, pruned_loss=0.01657, audio_tagging_loss=0.006115, over 15099.00 frames. ], tot_loss[loss=0.06607, simple_loss=0.08952, pruned_loss=0.01251, audio_tagging_loss=0.008796, over 3045959.81 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:32:39,650 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2983873.3333333335, ans=0.125
2023-11-24 20:32:52,234 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447600
2023-11-24 20:33:06,073 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff2_skip_rate, batch_count=2984006.6666666665, ans=0.0
2023-11-24 20:33:13,740 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.81 vs. limit=6.0
2023-11-24 20:33:19,488 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2984073.3333333335, ans=0.125
2023-11-24 20:33:33,448 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2750, loss[loss=0.05886, simple_loss=0.0843, pruned_loss=0.009119, audio_tagging_loss=0.007592, over 14796.00 frames. ], tot_loss[loss=0.06549, simple_loss=0.0886, pruned_loss=0.0124, audio_tagging_loss=0.008784, over 3043776.99 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:33:55,340 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447650
2023-11-24 20:33:59,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2984340.0, ans=0.2
2023-11-24 20:33:59,990 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.872e+01 8.580e+01 9.104e+01 1.006e+02 1.298e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 20:34:00,319 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2984340.0, ans=0.0
2023-11-24 20:34:27,047 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/IMdT8_tuNp0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:34:35,942 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2800, loss[loss=0.0547, simple_loss=0.07879, pruned_loss=0.0102, audio_tagging_loss=0.005103, over 15075.00 frames. ], tot_loss[loss=0.06611, simple_loss=0.08935, pruned_loss=0.0126, audio_tagging_loss=0.008836, over 3037049.29 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:34:38,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2984540.0, ans=0.125
2023-11-24 20:34:41,078 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0
2023-11-24 20:34:47,601 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2984606.6666666665, ans=0.1
2023-11-24 20:34:52,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2984606.6666666665, ans=0.07
2023-11-24 20:34:55,150 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=2984606.6666666665, ans=0.0
2023-11-24 20:34:57,490 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447700
2023-11-24 20:35:00,538 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.42 vs. limit=22.5
2023-11-24 20:35:03,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2984673.3333333335, ans=0.125
2023-11-24 20:35:21,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=2984740.0, ans=0.2
2023-11-24 20:35:21,427 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2984740.0, ans=0.0
2023-11-24 20:35:24,970 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2984806.6666666665, ans=0.125
2023-11-24 20:35:28,022 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2984806.6666666665, ans=0.1
2023-11-24 20:35:38,996 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2850, loss[loss=0.05082, simple_loss=0.06672, pruned_loss=0.005954, audio_tagging_loss=0.01151, over 14749.00 frames. ], tot_loss[loss=0.06642, simple_loss=0.0898, pruned_loss=0.0127, audio_tagging_loss=0.008816, over 3046052.75 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:35:41,946 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.80 vs. limit=10.0
2023-11-24 20:35:42,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=2984873.3333333335, ans=0.0
2023-11-24 20:35:45,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=2984873.3333333335, ans=0.0
2023-11-24 20:35:59,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447750
2023-11-24 20:36:03,343 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.85 vs. limit=22.5
2023-11-24 20:36:05,248 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.298e+01 8.522e+01 9.152e+01 9.707e+01 1.376e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 20:36:14,983 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=7.45 vs. limit=15.0
2023-11-24 20:36:30,267 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2985140.0, ans=0.125
2023-11-24 20:36:32,694 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2985140.0, ans=0.125
2023-11-24 20:36:37,489 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2985140.0, ans=0.125
2023-11-24 20:36:41,882 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2900, loss[loss=0.05922, simple_loss=0.08331, pruned_loss=0.007965, audio_tagging_loss=0.009598, over 15785.00 frames. ], tot_loss[loss=0.06645, simple_loss=0.09002, pruned_loss=0.01269, audio_tagging_loss=0.008752, over 3044390.85 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:36:43,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2985206.6666666665, ans=0.125
2023-11-24 20:37:03,294 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447800
2023-11-24 20:37:45,072 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 2950, loss[loss=0.05248, simple_loss=0.06746, pruned_loss=0.008615, audio_tagging_loss=0.01013, over 14385.00 frames. ], tot_loss[loss=0.06651, simple_loss=0.09022, pruned_loss=0.01265, audio_tagging_loss=0.008752, over 3043688.62 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:37:47,644 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=2985540.0, ans=0.125
2023-11-24 20:38:05,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2985606.6666666665, ans=0.1
2023-11-24 20:38:06,278 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447850
2023-11-24 20:38:06,417 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2985606.6666666665, ans=0.125
2023-11-24 20:38:10,883 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.767e+01 8.599e+01 9.301e+01 9.933e+01 1.313e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-24 20:38:12,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2985673.3333333335, ans=0.125
2023-11-24 20:38:43,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2985806.6666666665, ans=0.125
2023-11-24 20:38:47,547 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3000, loss[loss=0.0691, simple_loss=0.09067, pruned_loss=0.01275, audio_tagging_loss=0.01102, over 15060.00 frames. ], tot_loss[loss=0.06658, simple_loss=0.09019, pruned_loss=0.01264, audio_tagging_loss=0.008847, over 3043447.86 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:38:47,550 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 20:39:31,508 INFO [train_asr.py:1253] (0/4) Epoch 38, validation: loss=0.05738, simple_loss=0.0507, pruned_loss=0.005077, audio_tagging_loss=0.02696, over 4681554.00 frames. 
2023-11-24 20:39:31,509 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 20:39:35,911 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:39:43,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2985940.0, ans=0.125
2023-11-24 20:39:52,642 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447900
2023-11-24 20:40:04,712 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2986006.6666666665, ans=0.125
2023-11-24 20:40:07,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2986006.6666666665, ans=0.125
2023-11-24 20:40:26,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2986140.0, ans=0.0
2023-11-24 20:40:33,356 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer_ff2.min_abs, batch_count=2986206.6666666665, ans=0.1
2023-11-24 20:40:34,337 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3050, loss[loss=0.05931, simple_loss=0.08419, pruned_loss=0.01017, audio_tagging_loss=0.007035, over 15282.00 frames. ], tot_loss[loss=0.06763, simple_loss=0.09198, pruned_loss=0.01291, audio_tagging_loss=0.00873, over 3049394.36 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:40:36,910 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2986206.6666666665, ans=0.125
2023-11-24 20:40:55,640 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 447950
2023-11-24 20:41:00,358 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.683e+01 8.702e+01 9.218e+01 9.877e+01 1.257e+02, threshold=1.844e+02, percent-clipped=0.0
2023-11-24 20:41:10,484 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h0neUGB6j_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:41:17,294 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2986406.6666666665, ans=0.1
2023-11-24 20:41:37,186 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3100, loss[loss=0.07825, simple_loss=0.09349, pruned_loss=0.01881, audio_tagging_loss=0.0127, over 15127.00 frames. ], tot_loss[loss=0.06798, simple_loss=0.09246, pruned_loss=0.01294, audio_tagging_loss=0.008805, over 3048920.80 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:41:51,831 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2986606.6666666665, ans=0.125
2023-11-24 20:41:54,310 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2986606.6666666665, ans=0.2
2023-11-24 20:41:58,112 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448000
2023-11-24 20:41:59,615 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-448000.pt
2023-11-24 20:42:11,386 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn1.whiten, num_groups=1, num_channels=512, metric=16.74 vs. limit=22.5
2023-11-24 20:42:24,174 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=2986740.0, ans=0.2
2023-11-24 20:42:28,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2986740.0, ans=0.125
2023-11-24 20:42:31,132 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2986806.6666666665, ans=0.125
2023-11-24 20:42:39,454 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:42:42,708 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3150, loss[loss=0.07099, simple_loss=0.09552, pruned_loss=0.0158, audio_tagging_loss=0.007431, over 16538.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09183, pruned_loss=0.01288, audio_tagging_loss=0.008818, over 3051846.43 frames. ], batch size: 63, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:42:58,425 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=2986940.0, ans=0.125
2023-11-24 20:43:00,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=2986940.0, ans=0.125
2023-11-24 20:43:02,062 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2986940.0, ans=0.125
2023-11-24 20:43:04,333 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448050
2023-11-24 20:43:08,909 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.191e+01 8.654e+01 9.120e+01 9.857e+01 1.344e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 20:43:19,025 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.whiten.whitening_limit, batch_count=2987006.6666666665, ans=15.0
2023-11-24 20:43:26,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2987073.3333333335, ans=0.025
2023-11-24 20:43:30,429 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=2987073.3333333335, ans=0.1
2023-11-24 20:43:34,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=2987140.0, ans=0.0
2023-11-24 20:43:35,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=2987140.0, ans=0.2
2023-11-24 20:43:45,952 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3200, loss[loss=0.05746, simple_loss=0.07888, pruned_loss=0.007513, audio_tagging_loss=0.0105, over 15657.00 frames. ], tot_loss[loss=0.06793, simple_loss=0.09216, pruned_loss=0.013, audio_tagging_loss=0.008848, over 3053518.40 frames. ], batch size: 61, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:43:46,465 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.89 vs. limit=12.0
2023-11-24 20:43:56,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2987273.3333333335, ans=0.1
2023-11-24 20:44:05,267 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.51 vs. limit=15.0
2023-11-24 20:44:07,212 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448100
2023-11-24 20:44:15,026 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.17 vs. limit=10.0
2023-11-24 20:44:33,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2987406.6666666665, ans=0.0
2023-11-24 20:44:45,266 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2987473.3333333335, ans=0.125
2023-11-24 20:44:47,817 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3250, loss[loss=0.05067, simple_loss=0.05978, pruned_loss=0.009646, audio_tagging_loss=0.01114, over 14137.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09147, pruned_loss=0.0128, audio_tagging_loss=0.008931, over 3058864.32 frames. ], batch size: 57, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:44:58,344 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2987540.0, ans=0.0
2023-11-24 20:45:08,970 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448150
2023-11-24 20:45:15,410 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.226e+01 8.565e+01 9.106e+01 9.950e+01 1.221e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 20:45:28,432 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=9.25 vs. limit=12.0
2023-11-24 20:45:32,891 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=2987740.0, ans=0.125
2023-11-24 20:45:49,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=2987873.3333333335, ans=0.0
2023-11-24 20:45:50,295 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3300, loss[loss=0.07331, simple_loss=0.09533, pruned_loss=0.01571, audio_tagging_loss=0.009935, over 14766.00 frames. ], tot_loss[loss=0.06804, simple_loss=0.09237, pruned_loss=0.01291, audio_tagging_loss=0.00895, over 3064771.14 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:45:54,440 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=512, metric=10.63 vs. limit=12.0
2023-11-24 20:45:59,829 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.15 vs. limit=6.0
2023-11-24 20:46:03,015 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=2987940.0, ans=0.2
2023-11-24 20:46:11,544 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448200
2023-11-24 20:46:14,462 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:46:20,362 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2988006.6666666665, ans=0.1
2023-11-24 20:46:33,305 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2988073.3333333335, ans=0.125
2023-11-24 20:46:42,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=2988140.0, ans=0.125
2023-11-24 20:46:44,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2988140.0, ans=0.0
2023-11-24 20:46:53,182 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3350, loss[loss=0.05479, simple_loss=0.07821, pruned_loss=0.006961, audio_tagging_loss=0.008719, over 16710.00 frames. ], tot_loss[loss=0.06782, simple_loss=0.09191, pruned_loss=0.01293, audio_tagging_loss=0.008937, over 3057365.65 frames. ], batch size: 64, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:46:53,466 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:46:54,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=2988206.6666666665, ans=0.05
2023-11-24 20:47:01,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=2988206.6666666665, ans=0.125
2023-11-24 20:47:13,912 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448250
2023-11-24 20:47:16,579 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2988340.0, ans=0.125
2023-11-24 20:47:19,742 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.200e+01 8.779e+01 9.397e+01 1.017e+02 1.316e+02, threshold=1.879e+02, percent-clipped=0.0
2023-11-24 20:47:20,771 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:47:25,375 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2988340.0, ans=0.125
2023-11-24 20:47:33,113 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.40 vs. limit=15.0
2023-11-24 20:47:39,500 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=6.97 vs. limit=15.0
2023-11-24 20:47:49,097 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=2988473.3333333335, ans=0.0
2023-11-24 20:47:54,745 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3400, loss[loss=0.06523, simple_loss=0.08919, pruned_loss=0.01299, audio_tagging_loss=0.007642, over 15831.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09158, pruned_loss=0.01289, audio_tagging_loss=0.008863, over 3052268.99 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:48:02,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=2988540.0, ans=0.0
2023-11-24 20:48:16,209 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448300
2023-11-24 20:48:16,372 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.balancer1.prob, batch_count=2988606.6666666665, ans=0.125
2023-11-24 20:48:16,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=2988606.6666666665, ans=0.0
2023-11-24 20:48:22,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2988673.3333333335, ans=0.125
2023-11-24 20:48:48,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=2988806.6666666665, ans=0.0
2023-11-24 20:48:50,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2988806.6666666665, ans=0.125
2023-11-24 20:48:50,666 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2988806.6666666665, ans=0.1
2023-11-24 20:48:57,292 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3450, loss[loss=0.06886, simple_loss=0.09646, pruned_loss=0.01363, audio_tagging_loss=0.006997, over 15753.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09101, pruned_loss=0.01273, audio_tagging_loss=0.008798, over 3053950.94 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:49:13,044 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=2988940.0, ans=0.0
2023-11-24 20:49:15,282 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.63 vs. limit=15.0
2023-11-24 20:49:19,560 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448350
2023-11-24 20:49:25,647 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.019e+01 8.675e+01 9.257e+01 9.906e+01 1.659e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 20:49:38,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2989073.3333333335, ans=0.125
2023-11-24 20:49:43,514 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=2989073.3333333335, ans=0.0
2023-11-24 20:49:58,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2989140.0, ans=0.0
2023-11-24 20:49:58,842 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=2989140.0, ans=0.0
2023-11-24 20:50:01,052 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3500, loss[loss=0.06691, simple_loss=0.09206, pruned_loss=0.01287, audio_tagging_loss=0.008006, over 14367.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09156, pruned_loss=0.0129, audio_tagging_loss=0.008717, over 3055399.40 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:50:22,480 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448400
2023-11-24 20:50:32,110 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/DdDpuDqOyrA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:50:37,094 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.84 vs. limit=15.0
2023-11-24 20:50:41,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.min_positive, batch_count=2989406.6666666665, ans=0.025
2023-11-24 20:50:41,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=2989406.6666666665, ans=0.125
2023-11-24 20:50:55,874 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:50:56,367 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.83 vs. limit=15.0
2023-11-24 20:50:59,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=2989473.3333333335, ans=0.2
2023-11-24 20:51:01,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=2989473.3333333335, ans=0.125
2023-11-24 20:51:03,976 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3550, loss[loss=0.07543, simple_loss=0.1055, pruned_loss=0.01623, audio_tagging_loss=0.006434, over 15567.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.09033, pruned_loss=0.01272, audio_tagging_loss=0.00874, over 3053318.86 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 16.0
2023-11-24 20:51:15,626 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2989606.6666666665, ans=0.125
2023-11-24 20:51:25,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448450
2023-11-24 20:51:27,005 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=2989606.6666666665, ans=0.125
2023-11-24 20:51:28,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=2989673.3333333335, ans=0.125
2023-11-24 20:51:32,072 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.760e+01 8.602e+01 9.062e+01 9.844e+01 1.252e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 20:52:03,698 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=2989806.6666666665, ans=0.0
2023-11-24 20:52:06,928 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3600, loss[loss=0.06643, simple_loss=0.09867, pruned_loss=0.009881, audio_tagging_loss=0.007216, over 14338.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.09015, pruned_loss=0.01276, audio_tagging_loss=0.00875, over 3045559.74 frames. ], batch size: 54, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:52:12,260 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.66 vs. limit=22.5
2023-11-24 20:52:23,846 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2989940.0, ans=0.0
2023-11-24 20:52:29,121 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448500
2023-11-24 20:52:37,781 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.98 vs. limit=15.0
2023-11-24 20:52:42,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=2990006.6666666665, ans=0.125
2023-11-24 20:52:53,521 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 20:53:09,707 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3650, loss[loss=0.08036, simple_loss=0.115, pruned_loss=0.01485, audio_tagging_loss=0.007994, over 15257.00 frames. ], tot_loss[loss=0.06692, simple_loss=0.09061, pruned_loss=0.01294, audio_tagging_loss=0.008666, over 3049588.34 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:53:14,170 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2990206.6666666665, ans=0.1
2023-11-24 20:53:30,722 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448550
2023-11-24 20:53:36,945 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.710e+01 8.766e+01 9.382e+01 1.003e+02 1.165e+02, threshold=1.876e+02, percent-clipped=0.0
2023-11-24 20:53:39,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2990340.0, ans=0.1
2023-11-24 20:54:11,719 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3700, loss[loss=0.03957, simple_loss=0.04574, pruned_loss=0.005654, audio_tagging_loss=0.01105, over 14127.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09061, pruned_loss=0.01302, audio_tagging_loss=0.008708, over 3047661.37 frames. ], batch size: 55, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:54:32,530 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448600
2023-11-24 20:54:34,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2990606.6666666665, ans=0.125
2023-11-24 20:54:37,944 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=2990673.3333333335, ans=0.2
2023-11-24 20:54:55,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2990740.0, ans=0.2
2023-11-24 20:55:12,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=2990806.6666666665, ans=0.0
2023-11-24 20:55:14,088 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3750, loss[loss=0.06456, simple_loss=0.08044, pruned_loss=0.01487, audio_tagging_loss=0.009469, over 14692.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09141, pruned_loss=0.01325, audio_tagging_loss=0.008761, over 3056768.47 frames. ], batch size: 56, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:55:23,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=2990873.3333333335, ans=0.125
2023-11-24 20:55:35,650 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448650
2023-11-24 20:55:38,110 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2991006.6666666665, ans=0.125
2023-11-24 20:55:40,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=384, metric=16.17 vs. limit=22.5
2023-11-24 20:55:41,382 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.318e+01 8.492e+01 9.238e+01 9.913e+01 1.162e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 20:55:46,557 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.73 vs. limit=15.0
2023-11-24 20:55:49,087 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.77 vs. limit=22.5
2023-11-24 20:55:54,937 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ZY_Bsi-RNuk_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 20:55:55,464 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.52 vs. limit=15.0
2023-11-24 20:55:57,608 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=2991073.3333333335, ans=0.2
2023-11-24 20:56:15,253 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3800, loss[loss=0.05918, simple_loss=0.07777, pruned_loss=0.0106, audio_tagging_loss=0.009691, over 14557.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09153, pruned_loss=0.01315, audio_tagging_loss=0.00874, over 3065164.93 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:56:27,942 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=18.72 vs. limit=22.5
2023-11-24 20:56:36,671 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448700
2023-11-24 20:57:17,966 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3850, loss[loss=0.07355, simple_loss=0.1003, pruned_loss=0.01416, audio_tagging_loss=0.009257, over 15005.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09123, pruned_loss=0.01302, audio_tagging_loss=0.008774, over 3053123.71 frames. ], batch size: 58, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:57:38,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448750
2023-11-24 20:57:38,756 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.prob, batch_count=2991606.6666666665, ans=0.125
2023-11-24 20:57:42,235 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2991673.3333333335, ans=0.125
2023-11-24 20:57:44,194 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.441e+01 8.712e+01 9.172e+01 9.948e+01 1.223e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 20:58:08,938 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2991806.6666666665, ans=0.2
2023-11-24 20:58:19,601 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3900, loss[loss=0.0749, simple_loss=0.1015, pruned_loss=0.0125, audio_tagging_loss=0.01164, over 16005.00 frames. ], tot_loss[loss=0.06755, simple_loss=0.09148, pruned_loss=0.01302, audio_tagging_loss=0.008786, over 3055332.81 frames. ], batch size: 59, lr: 1.79e-03, grad_scale: 32.0
2023-11-24 20:58:27,043 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2991873.3333333335, ans=0.0
2023-11-24 20:58:32,919 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2991940.0, ans=0.1
2023-11-24 20:58:40,345 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448800
2023-11-24 20:58:57,359 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=2992073.3333333335, ans=0.125
2023-11-24 20:59:13,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2992140.0, ans=0.1
2023-11-24 20:59:17,746 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2992140.0, ans=0.1
2023-11-24 20:59:20,994 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 3950, loss[loss=0.0631, simple_loss=0.0889, pruned_loss=0.008378, audio_tagging_loss=0.01027, over 15596.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09108, pruned_loss=0.01286, audio_tagging_loss=0.008851, over 3053016.16 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 20:59:32,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2992206.6666666665, ans=0.125
2023-11-24 20:59:42,951 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448850
2023-11-24 20:59:46,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2992340.0, ans=0.0
2023-11-24 20:59:48,710 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.939e+01 8.553e+01 9.326e+01 9.949e+01 1.288e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 20:59:48,933 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=2992340.0, ans=0.0
2023-11-24 20:59:59,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=2992406.6666666665, ans=0.0
2023-11-24 21:00:00,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=2992406.6666666665, ans=0.125
2023-11-24 21:00:12,568 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2992473.3333333335, ans=0.0
2023-11-24 21:00:14,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=2992473.3333333335, ans=0.0
2023-11-24 21:00:23,987 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4000, loss[loss=0.07009, simple_loss=0.09139, pruned_loss=0.01357, audio_tagging_loss=0.01082, over 14021.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09195, pruned_loss=0.01301, audio_tagging_loss=0.008922, over 3059491.52 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:00:44,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448900
2023-11-24 21:00:46,074 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=2992606.6666666665, ans=0.125
2023-11-24 21:00:49,088 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.13 vs. limit=10.0
2023-11-24 21:00:53,322 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=2992673.3333333335, ans=0.0
2023-11-24 21:01:16,265 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2992806.6666666665, ans=0.1
2023-11-24 21:01:25,960 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4050, loss[loss=0.08037, simple_loss=0.1164, pruned_loss=0.01578, audio_tagging_loss=0.006416, over 15774.00 frames. ], tot_loss[loss=0.06842, simple_loss=0.09269, pruned_loss=0.01312, audio_tagging_loss=0.008957, over 3055113.74 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:01:27,225 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/-7b0f9TyPFU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:01:35,956 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=2992873.3333333335, ans=0.1
2023-11-24 21:01:47,015 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 448950
2023-11-24 21:01:51,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2993006.6666666665, ans=0.0
2023-11-24 21:01:54,345 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.560e+01 8.711e+01 9.505e+01 1.006e+02 1.290e+02, threshold=1.901e+02, percent-clipped=0.0
2023-11-24 21:02:09,928 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.97 vs. limit=15.0
2023-11-24 21:02:27,593 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4100, loss[loss=0.05573, simple_loss=0.07428, pruned_loss=0.01054, audio_tagging_loss=0.00805, over 15014.00 frames. ], tot_loss[loss=0.06805, simple_loss=0.09182, pruned_loss=0.0131, audio_tagging_loss=0.009036, over 3052624.20 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:02:34,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2993206.6666666665, ans=0.125
2023-11-24 21:02:35,032 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=8.17 vs. limit=15.0
2023-11-24 21:02:50,057 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449000
2023-11-24 21:02:55,595 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2993340.0, ans=0.0
2023-11-24 21:03:03,726 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.34 vs. limit=15.0
2023-11-24 21:03:13,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2993406.6666666665, ans=0.125
2023-11-24 21:03:22,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2993473.3333333335, ans=0.1
2023-11-24 21:03:31,086 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4150, loss[loss=0.07067, simple_loss=0.09131, pruned_loss=0.01695, audio_tagging_loss=0.008064, over 16040.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09145, pruned_loss=0.01289, audio_tagging_loss=0.008928, over 3049491.84 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:03:52,548 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449050
2023-11-24 21:03:54,315 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.36 vs. limit=22.5
2023-11-24 21:03:59,448 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.744e+01 8.629e+01 9.149e+01 9.710e+01 1.194e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 21:04:15,016 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/5BkClLNthIQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:04:19,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=6.51 vs. limit=15.0
2023-11-24 21:04:20,698 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.15 vs. limit=15.0
2023-11-24 21:04:33,291 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4200, loss[loss=0.07307, simple_loss=0.09921, pruned_loss=0.01518, audio_tagging_loss=0.008287, over 15340.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09109, pruned_loss=0.01274, audio_tagging_loss=0.008798, over 3049026.22 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:04:52,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=2993940.0, ans=0.0
2023-11-24 21:04:53,868 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449100
2023-11-24 21:04:54,468 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.57 vs. limit=15.0
2023-11-24 21:04:55,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2993940.0, ans=0.015
2023-11-24 21:05:18,607 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2994073.3333333335, ans=0.1
2023-11-24 21:05:26,403 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2994140.0, ans=0.1
2023-11-24 21:05:35,513 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4250, loss[loss=0.08361, simple_loss=0.1186, pruned_loss=0.01846, audio_tagging_loss=0.005821, over 16670.00 frames. ], tot_loss[loss=0.0677, simple_loss=0.09221, pruned_loss=0.01287, audio_tagging_loss=0.008721, over 3051523.69 frames. ], batch size: 64, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:05:35,836 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:05:39,467 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=2994206.6666666665, ans=0.2
2023-11-24 21:05:56,954 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449150
2023-11-24 21:06:05,024 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.615e+01 8.581e+01 9.086e+01 9.848e+01 1.238e+02, threshold=1.817e+02, percent-clipped=0.0
2023-11-24 21:06:08,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer2.prob, batch_count=2994340.0, ans=0.125
2023-11-24 21:06:13,508 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2994406.6666666665, ans=0.1
2023-11-24 21:06:18,273 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=2994406.6666666665, ans=0.125
2023-11-24 21:06:37,941 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4300, loss[loss=0.06322, simple_loss=0.08564, pruned_loss=0.01184, audio_tagging_loss=0.008566, over 16126.00 frames. ], tot_loss[loss=0.06821, simple_loss=0.093, pruned_loss=0.01309, audio_tagging_loss=0.008624, over 3058298.85 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:06:40,780 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=2994540.0, ans=0.04949747468305833
2023-11-24 21:06:44,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=2994540.0, ans=0.2
2023-11-24 21:06:59,456 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449200
2023-11-24 21:07:06,020 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=2994673.3333333335, ans=0.025
2023-11-24 21:07:07,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2994673.3333333335, ans=0.1
2023-11-24 21:07:14,481 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2994740.0, ans=0.125
2023-11-24 21:07:29,724 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2994806.6666666665, ans=0.125
2023-11-24 21:07:40,226 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4350, loss[loss=0.08883, simple_loss=0.1252, pruned_loss=0.02011, audio_tagging_loss=0.006103, over 15844.00 frames. ], tot_loss[loss=0.06789, simple_loss=0.09242, pruned_loss=0.01306, audio_tagging_loss=0.008617, over 3055103.00 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:07:54,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2994940.0, ans=0.1
2023-11-24 21:08:01,547 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449250
2023-11-24 21:08:10,474 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.763e+01 8.676e+01 9.413e+01 1.023e+02 1.276e+02, threshold=1.883e+02, percent-clipped=0.0
2023-11-24 21:08:20,649 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2995073.3333333335, ans=0.1
2023-11-24 21:08:20,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2995073.3333333335, ans=0.1
2023-11-24 21:08:42,993 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4400, loss[loss=0.07193, simple_loss=0.1049, pruned_loss=0.0133, audio_tagging_loss=0.006153, over 15340.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09247, pruned_loss=0.01306, audio_tagging_loss=0.008515, over 3049897.24 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:08:45,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2995206.6666666665, ans=0.0
2023-11-24 21:08:50,658 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2995206.6666666665, ans=0.125
2023-11-24 21:09:04,482 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449300
2023-11-24 21:09:23,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=2995406.6666666665, ans=0.1
2023-11-24 21:09:40,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=2995473.3333333335, ans=0.0
2023-11-24 21:09:45,322 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4450, loss[loss=0.07084, simple_loss=0.09673, pruned_loss=0.01428, audio_tagging_loss=0.008201, over 15996.00 frames. ], tot_loss[loss=0.06735, simple_loss=0.09155, pruned_loss=0.01297, audio_tagging_loss=0.008608, over 3049230.30 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:09:49,829 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2995540.0, ans=0.125
2023-11-24 21:09:55,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=2995540.0, ans=0.015
2023-11-24 21:10:06,673 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449350
2023-11-24 21:10:07,423 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.68 vs. limit=15.0
2023-11-24 21:10:16,934 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.434e+01 8.708e+01 9.388e+01 1.016e+02 1.957e+02, threshold=1.878e+02, percent-clipped=1.0
2023-11-24 21:10:25,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=2995740.0, ans=0.0
2023-11-24 21:10:40,979 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=2995806.6666666665, ans=0.0
2023-11-24 21:10:47,836 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4500, loss[loss=0.06515, simple_loss=0.0918, pruned_loss=0.0109, audio_tagging_loss=0.008355, over 15838.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09214, pruned_loss=0.0131, audio_tagging_loss=0.008579, over 3047787.66 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:11:02,252 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2995940.0, ans=0.125
2023-11-24 21:11:03,470 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=2995940.0, ans=0.0
2023-11-24 21:11:09,499 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449400
2023-11-24 21:11:26,815 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=2996073.3333333335, ans=0.0
2023-11-24 21:11:30,823 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.05 vs. limit=15.0
2023-11-24 21:11:39,679 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=2996140.0, ans=0.2
2023-11-24 21:11:39,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=2996140.0, ans=0.125
2023-11-24 21:11:51,474 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4550, loss[loss=0.07612, simple_loss=0.108, pruned_loss=0.0147, audio_tagging_loss=0.007418, over 15153.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09122, pruned_loss=0.013, audio_tagging_loss=0.008611, over 3046095.99 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:12:12,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2996273.3333333335, ans=0.125
2023-11-24 21:12:13,003 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449450
2023-11-24 21:12:22,245 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.502e+01 8.617e+01 8.939e+01 9.828e+01 1.230e+02, threshold=1.788e+02, percent-clipped=0.0
2023-11-24 21:12:22,511 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=2996340.0, ans=0.2
2023-11-24 21:12:31,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=2996406.6666666665, ans=0.0
2023-11-24 21:12:35,865 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_II2Klfnn4Y_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:12:42,839 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=2996473.3333333335, ans=0.2
2023-11-24 21:12:44,977 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=2996473.3333333335, ans=0.125
2023-11-24 21:12:53,666 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4600, loss[loss=0.05765, simple_loss=0.06835, pruned_loss=0.0124, audio_tagging_loss=0.01107, over 14369.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.09009, pruned_loss=0.01284, audio_tagging_loss=0.008794, over 3042180.55 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:13:14,398 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449500
2023-11-24 21:13:20,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=2996673.3333333335, ans=0.125
2023-11-24 21:13:31,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=2996740.0, ans=0.125
2023-11-24 21:13:32,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2996740.0, ans=0.0
2023-11-24 21:13:45,325 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=2996806.6666666665, ans=0.0
2023-11-24 21:13:48,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=2996806.6666666665, ans=0.125
2023-11-24 21:13:50,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2996806.6666666665, ans=0.125
2023-11-24 21:13:55,723 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4650, loss[loss=0.08158, simple_loss=0.113, pruned_loss=0.01667, audio_tagging_loss=0.008393, over 15117.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.09005, pruned_loss=0.01282, audio_tagging_loss=0.008846, over 3047130.38 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:13:56,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2996873.3333333335, ans=0.0
2023-11-24 21:14:16,913 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449550
2023-11-24 21:14:27,681 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.713e+01 8.430e+01 9.300e+01 1.028e+02 1.616e+02, threshold=1.860e+02, percent-clipped=0.0
2023-11-24 21:14:29,105 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2997006.6666666665, ans=0.1
2023-11-24 21:14:30,141 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2997006.6666666665, ans=0.125
2023-11-24 21:14:44,984 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=2997140.0, ans=0.2
2023-11-24 21:14:53,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten.whitening_limit, batch_count=2997140.0, ans=15.0
2023-11-24 21:14:58,454 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4700, loss[loss=0.05599, simple_loss=0.06837, pruned_loss=0.009194, audio_tagging_loss=0.01261, over 14762.00 frames. ], tot_loss[loss=0.0676, simple_loss=0.09119, pruned_loss=0.01319, audio_tagging_loss=0.008809, over 3051479.61 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:15:20,738 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449600
2023-11-24 21:15:24,295 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.16 vs. limit=10.0
2023-11-24 21:15:43,401 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2997406.6666666665, ans=0.125
2023-11-24 21:16:02,249 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4750, loss[loss=0.06666, simple_loss=0.0937, pruned_loss=0.0115, audio_tagging_loss=0.008318, over 14744.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.09081, pruned_loss=0.01304, audio_tagging_loss=0.008916, over 3050467.30 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:16:07,360 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward3.hidden_balancer.prob, batch_count=2997540.0, ans=0.125
2023-11-24 21:16:16,473 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=11.73 vs. limit=15.0
2023-11-24 21:16:18,544 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2997606.6666666665, ans=0.125
2023-11-24 21:16:23,100 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449650
2023-11-24 21:16:23,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2997606.6666666665, ans=0.125
2023-11-24 21:16:32,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=2997673.3333333335, ans=0.025
2023-11-24 21:16:33,255 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.234e+01 8.730e+01 9.403e+01 9.994e+01 1.287e+02, threshold=1.881e+02, percent-clipped=0.0
2023-11-24 21:16:45,745 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=8.68 vs. limit=10.0
2023-11-24 21:16:54,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=2997806.6666666665, ans=0.125
2023-11-24 21:17:04,670 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4800, loss[loss=0.07088, simple_loss=0.09701, pruned_loss=0.01238, audio_tagging_loss=0.009994, over 15278.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.09028, pruned_loss=0.01289, audio_tagging_loss=0.008966, over 3052302.05 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:17:21,125 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=2997940.0, ans=0.0
2023-11-24 21:17:25,548 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449700
2023-11-24 21:17:32,144 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=10.90 vs. limit=15.0
2023-11-24 21:17:46,396 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.82 vs. limit=15.0
2023-11-24 21:17:57,061 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2998140.0, ans=0.125
2023-11-24 21:17:58,758 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.91 vs. limit=8.0
2023-11-24 21:18:01,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2998140.0, ans=0.1
2023-11-24 21:18:06,863 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4850, loss[loss=0.0654, simple_loss=0.08303, pruned_loss=0.01122, audio_tagging_loss=0.01267, over 14988.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.08991, pruned_loss=0.01281, audio_tagging_loss=0.009102, over 3051569.98 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:18:28,652 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449750
2023-11-24 21:18:38,539 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.216e+01 8.567e+01 9.194e+01 1.005e+02 1.604e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 21:19:09,500 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4900, loss[loss=0.07944, simple_loss=0.1084, pruned_loss=0.01863, audio_tagging_loss=0.006616, over 15708.00 frames. ], tot_loss[loss=0.06726, simple_loss=0.09058, pruned_loss=0.01291, audio_tagging_loss=0.009058, over 3049518.78 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:19:16,847 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=2998540.0, ans=0.0
2023-11-24 21:19:28,151 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2998606.6666666665, ans=0.1
2023-11-24 21:19:31,529 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449800
2023-11-24 21:19:45,583 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=2998673.3333333335, ans=0.95
2023-11-24 21:20:13,078 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 4950, loss[loss=0.07223, simple_loss=0.1085, pruned_loss=0.01252, audio_tagging_loss=0.005444, over 14786.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09103, pruned_loss=0.0131, audio_tagging_loss=0.008924, over 3046997.41 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:20:15,049 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.45 vs. limit=15.0
2023-11-24 21:20:31,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=2998940.0, ans=0.125
2023-11-24 21:20:33,959 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449850
2023-11-24 21:20:43,264 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.094e+01 8.508e+01 9.344e+01 9.867e+01 1.192e+02, threshold=1.869e+02, percent-clipped=0.0
2023-11-24 21:20:44,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=2999006.6666666665, ans=0.125
2023-11-24 21:21:00,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2999073.3333333335, ans=0.1
2023-11-24 21:21:15,154 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5000, loss[loss=0.07062, simple_loss=0.09943, pruned_loss=0.01438, audio_tagging_loss=0.00652, over 15056.00 frames. ], tot_loss[loss=0.06646, simple_loss=0.08957, pruned_loss=0.01279, audio_tagging_loss=0.008882, over 3049005.24 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:21:36,333 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449900
2023-11-24 21:21:37,763 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2999273.3333333335, ans=0.1
2023-11-24 21:22:09,037 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=2999473.3333333335, ans=0.07
2023-11-24 21:22:16,957 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5050, loss[loss=0.07304, simple_loss=0.1041, pruned_loss=0.01234, audio_tagging_loss=0.00863, over 15646.00 frames. ], tot_loss[loss=0.0662, simple_loss=0.08931, pruned_loss=0.01274, audio_tagging_loss=0.008803, over 3043388.12 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:22:18,515 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:22:26,183 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=2999540.0, ans=0.0
2023-11-24 21:22:27,570 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=2999540.0, ans=0.0
2023-11-24 21:22:32,735 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=2999606.6666666665, ans=0.125
2023-11-24 21:22:36,185 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=2999606.6666666665, ans=0.125
2023-11-24 21:22:38,478 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 449950
2023-11-24 21:22:48,384 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.944e+01 8.754e+01 9.208e+01 9.764e+01 1.294e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 21:22:58,226 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=2999740.0, ans=10.0
2023-11-24 21:23:07,729 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=2999806.6666666665, ans=0.0
2023-11-24 21:23:17,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=2999806.6666666665, ans=0.125
2023-11-24 21:23:19,854 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5100, loss[loss=0.06413, simple_loss=0.08401, pruned_loss=0.00942, audio_tagging_loss=0.0127, over 15380.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09123, pruned_loss=0.01316, audio_tagging_loss=0.008618, over 3046652.01 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:23:34,667 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=7.58 vs. limit=12.0
2023-11-24 21:23:40,656 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450000
2023-11-24 21:23:53,675 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=9.14 vs. limit=10.0
2023-11-24 21:24:08,067 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3000073.3333333335, ans=0.0
2023-11-24 21:24:12,899 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3000140.0, ans=0.125
2023-11-24 21:24:22,065 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5150, loss[loss=0.09749, simple_loss=0.1483, pruned_loss=0.01876, audio_tagging_loss=0.004597, over 15487.00 frames. ], tot_loss[loss=0.06747, simple_loss=0.09172, pruned_loss=0.01305, audio_tagging_loss=0.008569, over 3047921.81 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:24:42,856 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450050
2023-11-24 21:24:47,520 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.89 vs. limit=12.0
2023-11-24 21:24:49,948 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.73 vs. limit=15.0
2023-11-24 21:24:53,308 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.156e+01 8.721e+01 9.361e+01 9.842e+01 1.528e+02, threshold=1.872e+02, percent-clipped=0.0
2023-11-24 21:25:14,002 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3000473.3333333335, ans=0.1
2023-11-24 21:25:21,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=3000473.3333333335, ans=10.0
2023-11-24 21:25:24,408 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5200, loss[loss=0.06419, simple_loss=0.08456, pruned_loss=0.0125, audio_tagging_loss=0.009411, over 14050.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09214, pruned_loss=0.01309, audio_tagging_loss=0.00842, over 3047257.01 frames. ], batch size: 52, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:25:37,327 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=3000606.6666666665, ans=0.125
2023-11-24 21:25:44,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3000606.6666666665, ans=0.125
2023-11-24 21:25:45,791 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450100
2023-11-24 21:25:45,858 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3000606.6666666665, ans=0.125
2023-11-24 21:25:47,912 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3000606.6666666665, ans=0.1
2023-11-24 21:25:47,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3000606.6666666665, ans=0.125
2023-11-24 21:25:48,253 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=10.25 vs. limit=15.0
2023-11-24 21:25:53,623 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3000673.3333333335, ans=0.125
2023-11-24 21:25:56,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3000673.3333333335, ans=0.0
2023-11-24 21:26:13,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=3000806.6666666665, ans=0.2
2023-11-24 21:26:15,872 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3000806.6666666665, ans=0.125
2023-11-24 21:26:17,073 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:26:23,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3000806.6666666665, ans=0.1
2023-11-24 21:26:27,393 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5250, loss[loss=0.08991, simple_loss=0.1263, pruned_loss=0.01997, audio_tagging_loss=0.006816, over 14333.00 frames. ], tot_loss[loss=0.06775, simple_loss=0.09252, pruned_loss=0.01304, audio_tagging_loss=0.008447, over 3054443.46 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:26:43,506 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3000940.0, ans=0.1
2023-11-24 21:26:48,594 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450150
2023-11-24 21:26:51,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3001006.6666666665, ans=0.125
2023-11-24 21:26:56,092 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3001006.6666666665, ans=0.125
2023-11-24 21:26:59,279 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.195e+01 8.390e+01 9.326e+01 9.807e+01 1.099e+02, threshold=1.865e+02, percent-clipped=0.0
2023-11-24 21:26:59,561 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=3001006.6666666665, ans=0.0
2023-11-24 21:27:02,103 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=3001006.6666666665, ans=0.0
2023-11-24 21:27:05,946 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=3001073.3333333335, ans=0.125
2023-11-24 21:27:29,181 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5300, loss[loss=0.08422, simple_loss=0.1147, pruned_loss=0.01851, audio_tagging_loss=0.008369, over 15559.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.09234, pruned_loss=0.01303, audio_tagging_loss=0.008514, over 3051296.72 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:27:40,605 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3001273.3333333335, ans=0.0
2023-11-24 21:27:50,017 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450200
2023-11-24 21:27:57,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=3001340.0, ans=0.125
2023-11-24 21:28:31,200 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5350, loss[loss=0.07651, simple_loss=0.1096, pruned_loss=0.01438, audio_tagging_loss=0.007314, over 15645.00 frames. ], tot_loss[loss=0.06753, simple_loss=0.09197, pruned_loss=0.01297, audio_tagging_loss=0.008574, over 3044396.87 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:28:31,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=3001540.0, ans=0.0
2023-11-24 21:28:32,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.balancer.min_positive, batch_count=3001540.0, ans=0.05
2023-11-24 21:28:38,471 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3001540.0, ans=0.1
2023-11-24 21:28:52,495 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450250
2023-11-24 21:28:56,213 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=3001673.3333333335, ans=0.2
2023-11-24 21:29:00,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=3001673.3333333335, ans=0.125
2023-11-24 21:29:03,593 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.946e+01 8.563e+01 9.107e+01 9.848e+01 1.290e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 21:29:12,724 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:29:21,384 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3001806.6666666665, ans=0.125
2023-11-24 21:29:33,376 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5400, loss[loss=0.06777, simple_loss=0.09507, pruned_loss=0.01068, audio_tagging_loss=0.009548, over 14969.00 frames. ], tot_loss[loss=0.06754, simple_loss=0.09192, pruned_loss=0.0129, audio_tagging_loss=0.00868, over 3042930.99 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:29:55,192 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450300
2023-11-24 21:30:07,034 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=3002006.6666666665, ans=0.0
2023-11-24 21:30:07,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3002006.6666666665, ans=0.1
2023-11-24 21:30:15,721 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=3002073.3333333335, ans=0.125
2023-11-24 21:30:22,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=3002140.0, ans=0.125
2023-11-24 21:30:23,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=3002140.0, ans=0.2
2023-11-24 21:30:34,893 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5450, loss[loss=0.07318, simple_loss=0.08283, pruned_loss=0.02061, audio_tagging_loss=0.01115, over 15538.00 frames. ], tot_loss[loss=0.06811, simple_loss=0.0924, pruned_loss=0.01311, audio_tagging_loss=0.0088, over 3049730.27 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:30:36,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=3002206.6666666665, ans=0.125
2023-11-24 21:30:56,245 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450350
2023-11-24 21:30:58,782 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=3002340.0, ans=0.07
2023-11-24 21:31:07,239 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.066e+01 8.563e+01 9.171e+01 9.760e+01 1.153e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 21:31:15,861 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.24 vs. limit=12.0
2023-11-24 21:31:22,374 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=3002406.6666666665, ans=0.0
2023-11-24 21:31:25,256 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.85 vs. limit=15.0
2023-11-24 21:31:33,738 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=3002473.3333333335, ans=0.2
2023-11-24 21:31:36,965 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5500, loss[loss=0.0564, simple_loss=0.07606, pruned_loss=0.01081, audio_tagging_loss=0.00756, over 15452.00 frames. ], tot_loss[loss=0.06766, simple_loss=0.09184, pruned_loss=0.01295, audio_tagging_loss=0.008786, over 3056005.95 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:31:39,720 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff2_skip_rate, batch_count=3002540.0, ans=0.0
2023-11-24 21:31:57,994 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450400
2023-11-24 21:32:31,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=3002806.6666666665, ans=0.2
2023-11-24 21:32:38,336 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.88 vs. limit=6.0
2023-11-24 21:32:38,616 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5550, loss[loss=0.06367, simple_loss=0.08995, pruned_loss=0.01124, audio_tagging_loss=0.007462, over 15092.00 frames. ], tot_loss[loss=0.06722, simple_loss=0.09096, pruned_loss=0.01284, audio_tagging_loss=0.008901, over 3053086.42 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 21:32:54,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3002940.0, ans=0.0
2023-11-24 21:32:59,873 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450450
2023-11-24 21:33:06,077 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3003006.6666666665, ans=0.125
2023-11-24 21:33:12,771 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.287e+01 8.457e+01 9.165e+01 9.990e+01 1.185e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 21:33:14,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3003006.6666666665, ans=0.1
2023-11-24 21:33:38,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3003140.0, ans=0.1
2023-11-24 21:33:40,973 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5600, loss[loss=0.09868, simple_loss=0.1425, pruned_loss=0.01815, audio_tagging_loss=0.009291, over 16111.00 frames. ], tot_loss[loss=0.06792, simple_loss=0.09214, pruned_loss=0.01299, audio_tagging_loss=0.008854, over 3050851.64 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:34:02,679 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450500
2023-11-24 21:34:16,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=3003340.0, ans=0.04949747468305833
2023-11-24 21:34:23,716 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/ze0LsBtoDm0_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:34:44,289 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5650, loss[loss=0.07143, simple_loss=0.09577, pruned_loss=0.01457, audio_tagging_loss=0.008982, over 15212.00 frames. ], tot_loss[loss=0.0679, simple_loss=0.09189, pruned_loss=0.01299, audio_tagging_loss=0.008961, over 3048907.36 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:34:52,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3003540.0, ans=0.125
2023-11-24 21:34:59,408 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:35:05,661 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450550
2023-11-24 21:35:17,494 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.220e+01 8.565e+01 9.205e+01 1.008e+02 1.241e+02, threshold=1.841e+02, percent-clipped=0.0
2023-11-24 21:35:33,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=3003806.6666666665, ans=0.2
2023-11-24 21:35:46,682 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5700, loss[loss=0.07706, simple_loss=0.1109, pruned_loss=0.01456, audio_tagging_loss=0.007048, over 15689.00 frames. ], tot_loss[loss=0.06774, simple_loss=0.09171, pruned_loss=0.01299, audio_tagging_loss=0.008896, over 3047256.74 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:35:56,200 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3003873.3333333335, ans=0.125
2023-11-24 21:36:02,267 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:36:08,129 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450600
2023-11-24 21:36:12,573 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=9.00 vs. limit=15.0
2023-11-24 21:36:15,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3004006.6666666665, ans=0.125
2023-11-24 21:36:26,336 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=3004073.3333333335, ans=0.125
2023-11-24 21:36:30,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=3004073.3333333335, ans=0.125
2023-11-24 21:36:49,709 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5750, loss[loss=0.08065, simple_loss=0.1107, pruned_loss=0.01745, audio_tagging_loss=0.007838, over 15094.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09084, pruned_loss=0.0128, audio_tagging_loss=0.008843, over 3043675.10 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:36:53,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=3004206.6666666665, ans=0.0
2023-11-24 21:36:57,286 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:37:06,212 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3004273.3333333335, ans=0.1
2023-11-24 21:37:11,206 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450650
2023-11-24 21:37:13,771 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3004340.0, ans=0.125
2023-11-24 21:37:14,958 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3004340.0, ans=0.125
2023-11-24 21:37:23,499 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.183e+01 8.366e+01 9.124e+01 1.023e+02 1.214e+02, threshold=1.825e+02, percent-clipped=0.0
2023-11-24 21:37:37,707 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.79 vs. limit=12.0
2023-11-24 21:37:47,419 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.59 vs. limit=12.0
2023-11-24 21:37:49,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3004473.3333333335, ans=0.125
2023-11-24 21:37:52,014 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5800, loss[loss=0.05776, simple_loss=0.08529, pruned_loss=0.006012, audio_tagging_loss=0.009105, over 15238.00 frames. ], tot_loss[loss=0.0665, simple_loss=0.09005, pruned_loss=0.01274, audio_tagging_loss=0.00873, over 3046310.80 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:37:53,414 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3004540.0, ans=0.125
2023-11-24 21:38:13,119 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450700
2023-11-24 21:38:32,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3004740.0, ans=0.1
2023-11-24 21:38:40,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=3004806.6666666665, ans=0.0
2023-11-24 21:38:54,022 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5850, loss[loss=0.06115, simple_loss=0.08196, pruned_loss=0.01011, audio_tagging_loss=0.01006, over 15414.00 frames. ], tot_loss[loss=0.06593, simple_loss=0.08904, pruned_loss=0.01264, audio_tagging_loss=0.008777, over 3035529.09 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:38:57,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=3004873.3333333335, ans=0.035
2023-11-24 21:39:14,679 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450750
2023-11-24 21:39:20,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3005006.6666666665, ans=0.125
2023-11-24 21:39:26,994 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.612e+01 8.737e+01 9.217e+01 9.924e+01 1.196e+02, threshold=1.843e+02, percent-clipped=0.0
2023-11-24 21:39:28,393 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=3005006.6666666665, ans=0.035
2023-11-24 21:39:32,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=3005073.3333333335, ans=0.07
2023-11-24 21:39:44,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=3005140.0, ans=0.125
2023-11-24 21:39:51,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3005140.0, ans=0.125
2023-11-24 21:39:55,827 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5900, loss[loss=0.06374, simple_loss=0.08935, pruned_loss=0.01264, audio_tagging_loss=0.006424, over 14928.00 frames. ], tot_loss[loss=0.06548, simple_loss=0.08858, pruned_loss=0.0125, audio_tagging_loss=0.008687, over 3034251.64 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:40:00,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=3005206.6666666665, ans=0.0
2023-11-24 21:40:14,278 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff3_skip_rate, batch_count=3005273.3333333335, ans=0.0
2023-11-24 21:40:16,500 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450800
2023-11-24 21:40:21,830 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3005340.0, ans=0.125
2023-11-24 21:40:24,032 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3005340.0, ans=0.125
2023-11-24 21:40:43,396 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=3005406.6666666665, ans=0.2
2023-11-24 21:40:47,143 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:40:57,900 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 5950, loss[loss=0.06413, simple_loss=0.08598, pruned_loss=0.01428, audio_tagging_loss=0.006865, over 13961.00 frames. ], tot_loss[loss=0.06523, simple_loss=0.08822, pruned_loss=0.01235, audio_tagging_loss=0.008771, over 3034978.66 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:41:07,637 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3005540.0, ans=0.125
2023-11-24 21:41:17,996 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3005606.6666666665, ans=0.1
2023-11-24 21:41:19,141 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450850
2023-11-24 21:41:21,668 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=3005673.3333333335, ans=0.125
2023-11-24 21:41:31,423 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.501e+01 8.441e+01 9.152e+01 9.875e+01 1.330e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 21:41:36,444 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3005740.0, ans=0.125
2023-11-24 21:41:47,940 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.91 vs. limit=12.0
2023-11-24 21:41:49,229 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.79 vs. limit=15.0
2023-11-24 21:41:53,609 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3005806.6666666665, ans=0.125
2023-11-24 21:41:59,110 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6000, loss[loss=0.06732, simple_loss=0.09175, pruned_loss=0.01291, audio_tagging_loss=0.008535, over 15156.00 frames. ], tot_loss[loss=0.06537, simple_loss=0.08861, pruned_loss=0.01232, audio_tagging_loss=0.008742, over 3038541.27 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:41:59,113 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 21:42:40,949 INFO [train_asr.py:1253] (0/4) Epoch 38, validation: loss=0.05788, simple_loss=0.05074, pruned_loss=0.005119, audio_tagging_loss=0.02739, over 4681554.00 frames. 
2023-11-24 21:42:40,950 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 21:42:43,828 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=12.45 vs. limit=15.0
2023-11-24 21:43:01,672 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450900
2023-11-24 21:43:03,031 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.scale_min, batch_count=3005940.0, ans=0.2
2023-11-24 21:43:23,376 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NoNxFjwXuuc_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 21:43:30,711 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:43:35,435 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=3006140.0, ans=0.0
2023-11-24 21:43:40,578 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=3006140.0, ans=0.0
2023-11-24 21:43:42,802 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6050, loss[loss=0.06043, simple_loss=0.08091, pruned_loss=0.009733, audio_tagging_loss=0.01025, over 14422.00 frames. ], tot_loss[loss=0.06579, simple_loss=0.08938, pruned_loss=0.01248, audio_tagging_loss=0.008616, over 3041847.49 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:43:57,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=3006273.3333333335, ans=0.0
2023-11-24 21:44:04,060 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 450950
2023-11-24 21:44:04,548 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.64 vs. limit=15.0
2023-11-24 21:44:17,480 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.275e+01 8.516e+01 8.966e+01 9.859e+01 1.258e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 21:44:19,497 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.52 vs. limit=15.0
2023-11-24 21:44:44,319 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6100, loss[loss=0.06009, simple_loss=0.07187, pruned_loss=0.01245, audio_tagging_loss=0.01171, over 15555.00 frames. ], tot_loss[loss=0.06558, simple_loss=0.08919, pruned_loss=0.01236, audio_tagging_loss=0.008614, over 3050191.86 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:44:59,231 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=3006606.6666666665, ans=0.125
2023-11-24 21:45:06,043 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451000
2023-11-24 21:45:15,856 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.46 vs. limit=15.0
2023-11-24 21:45:33,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3006806.6666666665, ans=0.0
2023-11-24 21:45:42,026 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.65 vs. limit=15.0
2023-11-24 21:45:47,790 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6150, loss[loss=0.07077, simple_loss=0.09209, pruned_loss=0.01588, audio_tagging_loss=0.008843, over 15406.00 frames. ], tot_loss[loss=0.06597, simple_loss=0.08981, pruned_loss=0.01241, audio_tagging_loss=0.008651, over 3049665.02 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:46:08,499 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451050
2023-11-24 21:46:08,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3006940.0, ans=0.125
2023-11-24 21:46:21,898 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.214e+01 8.834e+01 9.340e+01 1.011e+02 1.357e+02, threshold=1.868e+02, percent-clipped=0.0
2023-11-24 21:46:44,116 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.93 vs. limit=6.0
2023-11-24 21:46:49,556 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6200, loss[loss=0.07985, simple_loss=0.1075, pruned_loss=0.01853, audio_tagging_loss=0.007558, over 16038.00 frames. ], tot_loss[loss=0.0667, simple_loss=0.09051, pruned_loss=0.01272, audio_tagging_loss=0.008732, over 3050112.87 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:46:56,951 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=3007206.6666666665, ans=0.125
2023-11-24 21:47:04,591 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.77 vs. limit=10.0
2023-11-24 21:47:10,332 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451100
2023-11-24 21:47:12,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3007340.0, ans=0.125
2023-11-24 21:47:32,773 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.92 vs. limit=22.5
2023-11-24 21:47:37,307 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.61 vs. limit=15.0
2023-11-24 21:47:38,676 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0
2023-11-24 21:47:40,454 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=3007473.3333333335, ans=0.09899494936611666
2023-11-24 21:47:50,827 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6250, loss[loss=0.05947, simple_loss=0.08162, pruned_loss=0.009815, audio_tagging_loss=0.008849, over 15198.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.09022, pruned_loss=0.01278, audio_tagging_loss=0.008854, over 3046224.68 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:48:12,966 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451150
2023-11-24 21:48:15,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3007673.3333333335, ans=0.125
2023-11-24 21:48:26,314 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.790e+01 8.537e+01 9.157e+01 9.825e+01 1.470e+02, threshold=1.831e+02, percent-clipped=0.0
2023-11-24 21:48:28,252 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.20 vs. limit=10.0
2023-11-24 21:48:31,301 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=3007740.0, ans=0.0
2023-11-24 21:48:34,998 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3007740.0, ans=0.125
2023-11-24 21:48:40,963 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3007806.6666666665, ans=0.0
2023-11-24 21:48:45,885 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=3007806.6666666665, ans=0.125
2023-11-24 21:48:53,257 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6300, loss[loss=0.04958, simple_loss=0.0625, pruned_loss=0.007563, audio_tagging_loss=0.01076, over 15559.00 frames. ], tot_loss[loss=0.06679, simple_loss=0.08999, pruned_loss=0.01273, audio_tagging_loss=0.00906, over 3043567.91 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:49:00,155 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=3007873.3333333335, ans=0.2
2023-11-24 21:49:14,677 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451200
2023-11-24 21:49:22,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3008006.6666666665, ans=0.0
2023-11-24 21:49:25,928 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3008006.6666666665, ans=0.125
2023-11-24 21:49:29,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3008073.3333333335, ans=0.125
2023-11-24 21:49:34,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=3008073.3333333335, ans=0.125
2023-11-24 21:49:56,234 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6350, loss[loss=0.05567, simple_loss=0.0777, pruned_loss=0.007627, audio_tagging_loss=0.009198, over 17031.00 frames. ], tot_loss[loss=0.06685, simple_loss=0.08996, pruned_loss=0.01274, audio_tagging_loss=0.009126, over 3041021.62 frames. ], batch size: 64, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 21:50:01,758 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.61 vs. limit=6.0
2023-11-24 21:50:03,760 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=3008206.6666666665, ans=0.2
2023-11-24 21:50:04,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3008206.6666666665, ans=0.125
2023-11-24 21:50:07,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3008273.3333333335, ans=0.1
2023-11-24 21:50:15,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=3008273.3333333335, ans=0.2
2023-11-24 21:50:17,228 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451250
2023-11-24 21:50:17,549 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3008273.3333333335, ans=0.1
2023-11-24 21:50:28,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3008340.0, ans=0.125
2023-11-24 21:50:30,744 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.711e+01 8.435e+01 8.876e+01 9.772e+01 1.151e+02, threshold=1.775e+02, percent-clipped=0.0
2023-11-24 21:50:44,584 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=11.40 vs. limit=15.0
2023-11-24 21:50:54,613 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3008473.3333333335, ans=0.125
2023-11-24 21:50:57,003 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3008540.0, ans=0.0
2023-11-24 21:50:57,813 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6400, loss[loss=0.07568, simple_loss=0.1031, pruned_loss=0.01945, audio_tagging_loss=0.004702, over 15521.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09079, pruned_loss=0.0127, audio_tagging_loss=0.009087, over 3039836.19 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:51:15,964 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.31 vs. limit=10.0
2023-11-24 21:51:19,002 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451300
2023-11-24 21:51:53,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=3008806.6666666665, ans=10.0
2023-11-24 21:51:58,818 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.83 vs. limit=15.0
2023-11-24 21:51:59,920 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6450, loss[loss=0.04778, simple_loss=0.06077, pruned_loss=0.008506, audio_tagging_loss=0.008889, over 15701.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09037, pruned_loss=0.01268, audio_tagging_loss=0.009146, over 3037690.21 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:52:18,151 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.25 vs. limit=6.0
2023-11-24 21:52:21,344 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451350
2023-11-24 21:52:23,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=3009006.6666666665, ans=0.2
2023-11-24 21:52:34,134 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.918e+01 8.612e+01 9.313e+01 1.021e+02 1.233e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 21:52:48,602 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3009140.0, ans=0.125
2023-11-24 21:53:01,737 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6500, loss[loss=0.05816, simple_loss=0.07516, pruned_loss=0.01152, audio_tagging_loss=0.009058, over 14526.00 frames. ], tot_loss[loss=0.06683, simple_loss=0.09008, pruned_loss=0.01268, audio_tagging_loss=0.009107, over 3041099.59 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:53:06,303 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3009206.6666666665, ans=0.0
2023-11-24 21:53:19,210 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=3009273.3333333335, ans=0.2
2023-11-24 21:53:19,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer1.prob, batch_count=3009273.3333333335, ans=0.125
2023-11-24 21:53:22,645 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451400
2023-11-24 21:53:26,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3009340.0, ans=0.125
2023-11-24 21:53:48,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3009406.6666666665, ans=0.1
2023-11-24 21:53:48,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=3009406.6666666665, ans=0.2
2023-11-24 21:54:04,591 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6550, loss[loss=0.06761, simple_loss=0.09449, pruned_loss=0.01246, audio_tagging_loss=0.007909, over 15136.00 frames. ], tot_loss[loss=0.06705, simple_loss=0.09041, pruned_loss=0.01285, audio_tagging_loss=0.008991, over 3047109.35 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:54:15,048 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:54:25,935 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451450
2023-11-24 21:54:39,837 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.039e+01 8.549e+01 9.293e+01 1.004e+02 1.709e+02, threshold=1.859e+02, percent-clipped=0.0
2023-11-24 21:54:48,422 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=3009740.0, ans=0.09899494936611666
2023-11-24 21:54:51,876 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 21:55:06,537 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6600, loss[loss=0.07219, simple_loss=0.09557, pruned_loss=0.01405, audio_tagging_loss=0.01036, over 15572.00 frames. ], tot_loss[loss=0.06663, simple_loss=0.08992, pruned_loss=0.01278, audio_tagging_loss=0.008893, over 3046918.69 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:55:14,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=3009873.3333333335, ans=0.125
2023-11-24 21:55:28,533 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451500
2023-11-24 21:55:45,676 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=6.07 vs. limit=12.0
2023-11-24 21:55:46,859 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=10.71 vs. limit=15.0
2023-11-24 21:55:53,883 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=3010073.3333333335, ans=0.025
2023-11-24 21:55:55,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3010140.0, ans=0.1
2023-11-24 21:55:57,287 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.86 vs. limit=15.0
2023-11-24 21:56:08,149 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=384, metric=18.69 vs. limit=22.5
2023-11-24 21:56:08,549 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6650, loss[loss=0.04982, simple_loss=0.06366, pruned_loss=0.007419, audio_tagging_loss=0.01057, over 14478.00 frames. ], tot_loss[loss=0.06662, simple_loss=0.09001, pruned_loss=0.01276, audio_tagging_loss=0.008853, over 3038467.45 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:56:13,099 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3010206.6666666665, ans=0.0
2023-11-24 21:56:24,531 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3010273.3333333335, ans=0.125
2023-11-24 21:56:30,298 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451550
2023-11-24 21:56:44,246 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.352e+01 8.506e+01 9.140e+01 9.928e+01 1.246e+02, threshold=1.828e+02, percent-clipped=0.0
2023-11-24 21:57:11,180 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6700, loss[loss=0.06001, simple_loss=0.08391, pruned_loss=0.009537, audio_tagging_loss=0.008515, over 15019.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09054, pruned_loss=0.01292, audio_tagging_loss=0.008795, over 3032351.94 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:57:11,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=3010540.0, ans=0.0
2023-11-24 21:57:19,710 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3010540.0, ans=0.125
2023-11-24 21:57:19,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3010540.0, ans=0.125
2023-11-24 21:57:23,812 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass.scale_min, batch_count=3010606.6666666665, ans=0.2
2023-11-24 21:57:25,014 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=3010606.6666666665, ans=0.2
2023-11-24 21:57:28,733 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=3010606.6666666665, ans=0.2
2023-11-24 21:57:32,711 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451600
2023-11-24 21:57:43,906 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=3010673.3333333335, ans=0.0
2023-11-24 21:58:05,662 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.20 vs. limit=15.0
2023-11-24 21:58:13,697 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6750, loss[loss=0.06247, simple_loss=0.0784, pruned_loss=0.01312, audio_tagging_loss=0.01016, over 13605.00 frames. ], tot_loss[loss=0.06661, simple_loss=0.08991, pruned_loss=0.01283, audio_tagging_loss=0.008826, over 3026677.55 frames. ], batch size: 52, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:58:15,493 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.77 vs. limit=15.0
2023-11-24 21:58:34,509 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451650
2023-11-24 21:58:43,046 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3011006.6666666665, ans=0.1
2023-11-24 21:58:46,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=3011006.6666666665, ans=0.05
2023-11-24 21:58:49,689 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.221e+01 8.416e+01 8.949e+01 9.766e+01 1.528e+02, threshold=1.790e+02, percent-clipped=0.0
2023-11-24 21:58:55,876 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3011073.3333333335, ans=0.1
2023-11-24 21:58:59,968 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3011073.3333333335, ans=0.1
2023-11-24 21:59:03,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3011140.0, ans=0.125
2023-11-24 21:59:05,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3011140.0, ans=0.0
2023-11-24 21:59:15,546 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6800, loss[loss=0.05276, simple_loss=0.06523, pruned_loss=0.009496, audio_tagging_loss=0.01066, over 15434.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09077, pruned_loss=0.01289, audio_tagging_loss=0.008742, over 3038769.20 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 21:59:15,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.attention_skip_rate, batch_count=3011206.6666666665, ans=0.0
2023-11-24 21:59:36,670 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451700
2023-11-24 21:59:54,539 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3011406.6666666665, ans=0.0
2023-11-24 21:59:54,586 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3011406.6666666665, ans=0.0
2023-11-24 21:59:58,843 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.28 vs. limit=22.5
2023-11-24 22:00:06,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3011473.3333333335, ans=0.1
2023-11-24 22:00:18,174 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6850, loss[loss=0.06561, simple_loss=0.08954, pruned_loss=0.01335, audio_tagging_loss=0.00749, over 14688.00 frames. ], tot_loss[loss=0.06704, simple_loss=0.09067, pruned_loss=0.01299, audio_tagging_loss=0.008721, over 3037033.65 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:00:28,948 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=3011606.6666666665, ans=0.125
2023-11-24 22:00:39,368 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451750
2023-11-24 22:00:54,774 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.186e+01 8.792e+01 9.209e+01 9.855e+01 1.264e+02, threshold=1.842e+02, percent-clipped=0.0
2023-11-24 22:01:19,656 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6900, loss[loss=0.06707, simple_loss=0.09311, pruned_loss=0.01232, audio_tagging_loss=0.008191, over 15506.00 frames. ], tot_loss[loss=0.06669, simple_loss=0.0903, pruned_loss=0.0128, audio_tagging_loss=0.008734, over 3043497.71 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:01:36,522 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3011940.0, ans=0.125
2023-11-24 22:01:41,160 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451800
2023-11-24 22:01:45,706 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.83 vs. limit=15.0
2023-11-24 22:01:50,847 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.09 vs. limit=6.0
2023-11-24 22:01:51,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3012006.6666666665, ans=0.125
2023-11-24 22:01:54,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3012006.6666666665, ans=0.1
2023-11-24 22:01:55,808 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=3012006.6666666665, ans=0.125
2023-11-24 22:01:57,054 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=3012073.3333333335, ans=0.0
2023-11-24 22:02:06,141 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Xez1ffAcb0w_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:02:12,840 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=3012140.0, ans=0.125
2023-11-24 22:02:22,859 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 6950, loss[loss=0.08767, simple_loss=0.1258, pruned_loss=0.01721, audio_tagging_loss=0.007561, over 16506.00 frames. ], tot_loss[loss=0.06698, simple_loss=0.09077, pruned_loss=0.01284, audio_tagging_loss=0.008757, over 3045952.37 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:02:29,050 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3012206.6666666665, ans=0.0
2023-11-24 22:02:43,368 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3012273.3333333335, ans=0.125
2023-11-24 22:02:44,277 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451850
2023-11-24 22:02:44,743 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.self_attn1.whiten, num_groups=1, num_channels=384, metric=14.85 vs. limit=22.5
2023-11-24 22:03:00,154 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.498e+01 8.404e+01 9.015e+01 9.725e+01 1.456e+02, threshold=1.803e+02, percent-clipped=0.0
2023-11-24 22:03:02,467 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.84 vs. limit=22.5
2023-11-24 22:03:24,970 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7000, loss[loss=0.06662, simple_loss=0.08242, pruned_loss=0.01467, audio_tagging_loss=0.01074, over 15575.00 frames. ], tot_loss[loss=0.06614, simple_loss=0.08966, pruned_loss=0.01247, audio_tagging_loss=0.008847, over 3048438.84 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:03:32,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3012540.0, ans=0.0
2023-11-24 22:03:46,401 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451900
2023-11-24 22:03:49,640 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=3012673.3333333335, ans=0.07
2023-11-24 22:04:05,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3012740.0, ans=0.1
2023-11-24 22:04:11,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3012740.0, ans=0.1
2023-11-24 22:04:13,749 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3012806.6666666665, ans=0.1
2023-11-24 22:04:24,988 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=3012806.6666666665, ans=0.125
2023-11-24 22:04:27,280 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7050, loss[loss=0.07679, simple_loss=0.09643, pruned_loss=0.02064, audio_tagging_loss=0.007935, over 15119.00 frames. ], tot_loss[loss=0.06536, simple_loss=0.08807, pruned_loss=0.01237, audio_tagging_loss=0.008956, over 3047212.74 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:04:48,538 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 451950
2023-11-24 22:04:49,271 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=6.44 vs. limit=15.0
2023-11-24 22:05:04,199 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.796e+01 8.687e+01 9.290e+01 1.031e+02 1.279e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-24 22:05:04,548 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3013073.3333333335, ans=0.125
2023-11-24 22:05:27,603 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.73 vs. limit=10.0
2023-11-24 22:05:29,458 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7100, loss[loss=0.05554, simple_loss=0.07341, pruned_loss=0.007551, audio_tagging_loss=0.01128, over 15774.00 frames. ], tot_loss[loss=0.06563, simple_loss=0.08858, pruned_loss=0.01236, audio_tagging_loss=0.008989, over 3045422.49 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:05:29,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=3013206.6666666665, ans=0.0
2023-11-24 22:05:43,276 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=3013273.3333333335, ans=0.2
2023-11-24 22:05:47,935 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=3013273.3333333335, ans=0.125
2023-11-24 22:05:50,298 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452000
2023-11-24 22:05:51,695 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-452000.pt
2023-11-24 22:06:04,837 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=3013340.0, ans=0.0
2023-11-24 22:06:21,426 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.52 vs. limit=12.0
2023-11-24 22:06:23,921 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=12.10 vs. limit=15.0
2023-11-24 22:06:24,767 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3013473.3333333335, ans=0.125
2023-11-24 22:06:25,908 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=3013473.3333333335, ans=0.0
2023-11-24 22:06:35,710 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7150, loss[loss=0.08831, simple_loss=0.1243, pruned_loss=0.01744, audio_tagging_loss=0.008708, over 15964.00 frames. ], tot_loss[loss=0.06703, simple_loss=0.09077, pruned_loss=0.01275, audio_tagging_loss=0.008886, over 3052326.12 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:06:38,232 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=3013540.0, ans=0.0
2023-11-24 22:06:38,299 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=3013540.0, ans=0.0
2023-11-24 22:06:50,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=3013606.6666666665, ans=0.125
2023-11-24 22:06:50,499 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3013606.6666666665, ans=0.125
2023-11-24 22:06:51,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=3013606.6666666665, ans=0.125
2023-11-24 22:06:56,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452050
2023-11-24 22:07:10,984 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:07:12,928 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.275e+01 8.770e+01 9.367e+01 1.013e+02 1.404e+02, threshold=1.873e+02, percent-clipped=0.0
2023-11-24 22:07:32,142 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten.whitening_limit, batch_count=3013806.6666666665, ans=15.0
2023-11-24 22:07:37,809 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7200, loss[loss=0.05757, simple_loss=0.07247, pruned_loss=0.009437, audio_tagging_loss=0.01189, over 14484.00 frames. ], tot_loss[loss=0.06745, simple_loss=0.0912, pruned_loss=0.01291, audio_tagging_loss=0.008949, over 3042671.19 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:07:58,900 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452100
2023-11-24 22:08:09,363 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=3014006.6666666665, ans=0.2
2023-11-24 22:08:14,063 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3014073.3333333335, ans=0.125
2023-11-24 22:08:40,401 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7250, loss[loss=0.08284, simple_loss=0.1109, pruned_loss=0.01531, audio_tagging_loss=0.01208, over 15341.00 frames. ], tot_loss[loss=0.06768, simple_loss=0.09159, pruned_loss=0.01286, audio_tagging_loss=0.009028, over 3046840.74 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:09:01,063 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452150
2023-11-24 22:09:18,540 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.621e+01 8.553e+01 9.150e+01 9.601e+01 1.170e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 22:09:27,588 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=3014406.6666666665, ans=0.5
2023-11-24 22:09:33,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=3014473.3333333335, ans=0.0
2023-11-24 22:09:41,862 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7300, loss[loss=0.07079, simple_loss=0.09729, pruned_loss=0.01539, audio_tagging_loss=0.006753, over 15007.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09062, pruned_loss=0.01277, audio_tagging_loss=0.009098, over 3040449.93 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:09:47,321 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:09:50,076 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=9.53 vs. limit=15.0
2023-11-24 22:10:02,579 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452200
2023-11-24 22:10:05,645 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.97 vs. limit=6.0
2023-11-24 22:10:43,905 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7350, loss[loss=0.07034, simple_loss=0.09115, pruned_loss=0.01547, audio_tagging_loss=0.009295, over 14889.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09107, pruned_loss=0.01298, audio_tagging_loss=0.008899, over 3035877.08 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:11:05,660 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452250
2023-11-24 22:11:18,284 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3015006.6666666665, ans=0.1
2023-11-24 22:11:22,651 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.188e+01 8.406e+01 8.894e+01 9.896e+01 1.273e+02, threshold=1.779e+02, percent-clipped=0.0
2023-11-24 22:11:46,721 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7400, loss[loss=0.08085, simple_loss=0.1156, pruned_loss=0.0164, audio_tagging_loss=0.006635, over 16252.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09049, pruned_loss=0.01294, audio_tagging_loss=0.008891, over 3045483.37 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:12:02,286 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3015273.3333333335, ans=0.1
2023-11-24 22:12:03,538 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3015273.3333333335, ans=0.125
2023-11-24 22:12:07,412 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452300
2023-11-24 22:12:22,181 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.42 vs. limit=22.5
2023-11-24 22:12:45,165 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.78 vs. limit=6.0
2023-11-24 22:12:47,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.93 vs. limit=15.0
2023-11-24 22:12:48,016 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7450, loss[loss=0.06874, simple_loss=0.096, pruned_loss=0.01384, audio_tagging_loss=0.006899, over 15810.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09065, pruned_loss=0.01301, audio_tagging_loss=0.008766, over 3037279.11 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:12:52,800 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=512, metric=6.88 vs. limit=15.0
2023-11-24 22:13:01,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=3015606.6666666665, ans=0.125
2023-11-24 22:13:06,584 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=3015606.6666666665, ans=0.0
2023-11-24 22:13:08,746 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452350
2023-11-24 22:13:23,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3015673.3333333335, ans=0.125
2023-11-24 22:13:28,044 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.262e+01 8.690e+01 9.275e+01 1.001e+02 1.240e+02, threshold=1.855e+02, percent-clipped=0.0
2023-11-24 22:13:42,066 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=3.98 vs. limit=6.0
2023-11-24 22:13:45,244 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=3015806.6666666665, ans=0.0
2023-11-24 22:13:49,774 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7500, loss[loss=0.0693, simple_loss=0.09276, pruned_loss=0.012, audio_tagging_loss=0.01092, over 14488.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09087, pruned_loss=0.01305, audio_tagging_loss=0.008759, over 3037566.64 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:14:00,160 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3015873.3333333335, ans=0.0
2023-11-24 22:14:02,500 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:14:03,045 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.79 vs. limit=22.5
2023-11-24 22:14:11,906 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452400
2023-11-24 22:14:26,856 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3016073.3333333335, ans=0.0
2023-11-24 22:14:52,042 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7550, loss[loss=0.07085, simple_loss=0.08919, pruned_loss=0.01648, audio_tagging_loss=0.009779, over 15433.00 frames. ], tot_loss[loss=0.06714, simple_loss=0.0909, pruned_loss=0.01299, audio_tagging_loss=0.008691, over 3039756.16 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:15:14,481 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452450
2023-11-24 22:15:32,167 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.402e+01 8.333e+01 9.038e+01 9.761e+01 1.363e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 22:15:33,030 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=9.81 vs. limit=22.5
2023-11-24 22:15:35,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=3016406.6666666665, ans=0.2
2023-11-24 22:15:42,113 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3016473.3333333335, ans=0.1
2023-11-24 22:15:42,175 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=3016473.3333333335, ans=0.125
2023-11-24 22:15:47,951 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=7.11 vs. limit=12.0
2023-11-24 22:15:49,930 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=3016473.3333333335, ans=0.05
2023-11-24 22:15:55,619 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7600, loss[loss=0.06873, simple_loss=0.09748, pruned_loss=0.01229, audio_tagging_loss=0.007702, over 14962.00 frames. ], tot_loss[loss=0.06717, simple_loss=0.09098, pruned_loss=0.01297, audio_tagging_loss=0.008709, over 3045037.14 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:16:03,527 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=3016540.0, ans=0.0
2023-11-24 22:16:10,676 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=3016606.6666666665, ans=0.2
2023-11-24 22:16:12,039 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3016606.6666666665, ans=0.125
2023-11-24 22:16:16,329 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452500
2023-11-24 22:16:21,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3016673.3333333335, ans=0.0
2023-11-24 22:16:24,423 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3016673.3333333335, ans=0.125
2023-11-24 22:16:33,351 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3016740.0, ans=0.125
2023-11-24 22:16:44,590 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3016806.6666666665, ans=0.0
2023-11-24 22:16:57,869 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7650, loss[loss=0.0695, simple_loss=0.08888, pruned_loss=0.0159, audio_tagging_loss=0.009152, over 14784.00 frames. ], tot_loss[loss=0.06694, simple_loss=0.09071, pruned_loss=0.01286, audio_tagging_loss=0.008716, over 3045284.62 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:17:04,049 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=3016873.3333333335, ans=0.2
2023-11-24 22:17:15,885 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.79 vs. limit=6.0
2023-11-24 22:17:19,350 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452550
2023-11-24 22:17:31,814 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3017006.6666666665, ans=0.1
2023-11-24 22:17:37,944 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.771e+01 8.417e+01 9.176e+01 1.003e+02 1.926e+02, threshold=1.835e+02, percent-clipped=1.0
2023-11-24 22:18:00,073 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7700, loss[loss=0.05516, simple_loss=0.07437, pruned_loss=0.01041, audio_tagging_loss=0.007562, over 15655.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09065, pruned_loss=0.01283, audio_tagging_loss=0.008714, over 3048660.34 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:18:21,486 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452600
2023-11-24 22:18:24,634 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3017340.0, ans=0.0
2023-11-24 22:18:41,560 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=3017406.6666666665, ans=0.125
2023-11-24 22:18:42,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=3017406.6666666665, ans=0.0
2023-11-24 22:18:45,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=3017406.6666666665, ans=0.125
2023-11-24 22:18:48,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=3017406.6666666665, ans=0.125
2023-11-24 22:18:49,255 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=3017473.3333333335, ans=0.2
2023-11-24 22:18:49,383 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer1.min_positive, batch_count=3017473.3333333335, ans=0.025
2023-11-24 22:18:57,140 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3017473.3333333335, ans=0.125
2023-11-24 22:18:59,302 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=3017473.3333333335, ans=0.125
2023-11-24 22:19:02,805 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7750, loss[loss=0.05607, simple_loss=0.07091, pruned_loss=0.01191, audio_tagging_loss=0.008701, over 15192.00 frames. ], tot_loss[loss=0.06679, simple_loss=0.09045, pruned_loss=0.01271, audio_tagging_loss=0.008849, over 3046179.40 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:19:07,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3017540.0, ans=0.0
2023-11-24 22:19:24,008 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452650
2023-11-24 22:19:28,038 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.05 vs. limit=10.0
2023-11-24 22:19:42,996 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.087e+01 8.572e+01 9.238e+01 9.882e+01 1.332e+02, threshold=1.848e+02, percent-clipped=0.0
2023-11-24 22:19:43,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=3017740.0, ans=0.2
2023-11-24 22:19:45,592 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=3017740.0, ans=0.0
2023-11-24 22:19:53,804 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=3017806.6666666665, ans=0.0
2023-11-24 22:19:59,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=11.71 vs. limit=15.0
2023-11-24 22:20:03,758 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=384, metric=19.06 vs. limit=22.5
2023-11-24 22:20:05,367 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7800, loss[loss=0.08179, simple_loss=0.1033, pruned_loss=0.02253, audio_tagging_loss=0.0076, over 14825.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.0911, pruned_loss=0.01291, audio_tagging_loss=0.008856, over 3039140.69 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:20:16,367 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3017940.0, ans=0.125
2023-11-24 22:20:18,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3017940.0, ans=0.0
2023-11-24 22:20:26,876 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452700
2023-11-24 22:20:31,974 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.59 vs. limit=15.0
2023-11-24 22:20:39,268 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.46 vs. limit=10.0
2023-11-24 22:20:56,707 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer1.prob, batch_count=3018140.0, ans=0.125
2023-11-24 22:21:06,952 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7850, loss[loss=0.07934, simple_loss=0.1102, pruned_loss=0.01848, audio_tagging_loss=0.005778, over 15552.00 frames. ], tot_loss[loss=0.06739, simple_loss=0.09111, pruned_loss=0.01299, audio_tagging_loss=0.008841, over 3039254.07 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:21:19,157 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.min_positive, batch_count=3018273.3333333335, ans=0.05
2023-11-24 22:21:28,275 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452750
2023-11-24 22:21:46,954 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.360e+01 8.800e+01 9.383e+01 9.995e+01 1.679e+02, threshold=1.877e+02, percent-clipped=0.0
2023-11-24 22:22:04,503 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3018473.3333333335, ans=0.0
2023-11-24 22:22:09,986 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7900, loss[loss=0.06863, simple_loss=0.09721, pruned_loss=0.01266, audio_tagging_loss=0.007372, over 14230.00 frames. ], tot_loss[loss=0.06743, simple_loss=0.09102, pruned_loss=0.01297, audio_tagging_loss=0.008952, over 3043422.85 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:22:11,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3018540.0, ans=0.125
2023-11-24 22:22:30,889 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452800
2023-11-24 22:22:39,686 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=3018673.3333333335, ans=0.1
2023-11-24 22:22:42,306 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=3018673.3333333335, ans=0.0
2023-11-24 22:22:55,298 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3018740.0, ans=0.125
2023-11-24 22:22:57,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3018740.0, ans=0.1
2023-11-24 22:23:02,324 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=3018806.6666666665, ans=0.0
2023-11-24 22:23:06,449 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=8.57 vs. limit=15.0
2023-11-24 22:23:12,340 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 7950, loss[loss=0.05985, simple_loss=0.07737, pruned_loss=0.01081, audio_tagging_loss=0.01036, over 15433.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09074, pruned_loss=0.01297, audio_tagging_loss=0.009, over 3050054.01 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:23:12,824 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.28 vs. limit=15.0
2023-11-24 22:23:25,671 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/uQjH4tNUZ_g_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:23:27,104 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3018940.0, ans=0.125
2023-11-24 22:23:33,464 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452850
2023-11-24 22:23:45,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3019006.6666666665, ans=0.125
2023-11-24 22:23:52,665 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.512e+01 8.496e+01 9.034e+01 9.610e+01 1.973e+02, threshold=1.807e+02, percent-clipped=1.0
2023-11-24 22:23:52,989 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3019073.3333333335, ans=0.125
2023-11-24 22:24:13,976 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8000, loss[loss=0.06244, simple_loss=0.0836, pruned_loss=0.01192, audio_tagging_loss=0.008722, over 16390.00 frames. ], tot_loss[loss=0.06712, simple_loss=0.09031, pruned_loss=0.0129, audio_tagging_loss=0.009069, over 3045036.58 frames. ], batch size: 64, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:24:19,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3019206.6666666665, ans=0.125
2023-11-24 22:24:24,169 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=11.55 vs. limit=15.0
2023-11-24 22:24:27,520 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3019273.3333333335, ans=0.1
2023-11-24 22:24:31,053 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=3019273.3333333335, ans=0.125
2023-11-24 22:24:35,551 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452900
2023-11-24 22:24:41,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=3019340.0, ans=0.0
2023-11-24 22:25:13,355 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3019473.3333333335, ans=0.1
2023-11-24 22:25:16,649 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8050, loss[loss=0.06232, simple_loss=0.07752, pruned_loss=0.01588, audio_tagging_loss=0.007684, over 15003.00 frames. ], tot_loss[loss=0.06686, simple_loss=0.08987, pruned_loss=0.0128, audio_tagging_loss=0.009127, over 3041342.95 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:25:22,716 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3019540.0, ans=0.0
2023-11-24 22:25:38,382 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 452950
2023-11-24 22:25:56,982 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=3019740.0, ans=0.0
2023-11-24 22:25:59,468 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.598e+01 8.520e+01 9.232e+01 9.839e+01 1.227e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 22:26:03,540 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3019740.0, ans=0.1
2023-11-24 22:26:15,537 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3019806.6666666665, ans=0.125
2023-11-24 22:26:18,642 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8100, loss[loss=0.06376, simple_loss=0.08586, pruned_loss=0.01095, audio_tagging_loss=0.009884, over 15452.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.09021, pruned_loss=0.013, audio_tagging_loss=0.009095, over 3043130.89 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:26:24,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_skip_rate, batch_count=3019873.3333333335, ans=0.0
2023-11-24 22:26:40,120 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453000
2023-11-24 22:26:46,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3020006.6666666665, ans=0.1
2023-11-24 22:27:08,342 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3020140.0, ans=0.1
2023-11-24 22:27:13,372 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.07 vs. limit=15.0
2023-11-24 22:27:21,689 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8150, loss[loss=0.07554, simple_loss=0.1009, pruned_loss=0.01618, audio_tagging_loss=0.008891, over 14966.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09077, pruned_loss=0.0129, audio_tagging_loss=0.008861, over 3054876.09 frames. ], batch size: 54, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:27:31,978 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=3020206.6666666665, ans=0.125
2023-11-24 22:27:40,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=3020273.3333333335, ans=0.125
2023-11-24 22:27:43,169 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453050
2023-11-24 22:28:01,773 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3020406.6666666665, ans=0.125
2023-11-24 22:28:04,269 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.238e+01 8.627e+01 9.105e+01 9.732e+01 1.211e+02, threshold=1.821e+02, percent-clipped=0.0
2023-11-24 22:28:14,083 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=3020473.3333333335, ans=0.0
2023-11-24 22:28:23,167 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/8C7biyx9TQ4_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:28:24,325 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8200, loss[loss=0.09297, simple_loss=0.129, pruned_loss=0.02198, audio_tagging_loss=0.006467, over 16472.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09073, pruned_loss=0.01278, audio_tagging_loss=0.008828, over 3049951.25 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:28:28,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3020540.0, ans=0.125
2023-11-24 22:28:43,181 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3020606.6666666665, ans=0.125
2023-11-24 22:28:45,376 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453100
2023-11-24 22:28:55,732 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3020673.3333333335, ans=0.0
2023-11-24 22:28:55,971 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=7.39 vs. limit=15.0
2023-11-24 22:29:18,865 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.97 vs. limit=15.0
2023-11-24 22:29:26,348 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8250, loss[loss=0.05677, simple_loss=0.07625, pruned_loss=0.01073, audio_tagging_loss=0.007913, over 14767.00 frames. ], tot_loss[loss=0.06671, simple_loss=0.09057, pruned_loss=0.01272, audio_tagging_loss=0.008708, over 3049930.03 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:29:47,430 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.20 vs. limit=22.5
2023-11-24 22:29:47,860 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453150
2023-11-24 22:29:58,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=3021006.6666666665, ans=0.2
2023-11-24 22:30:00,287 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.52 vs. limit=15.0
2023-11-24 22:30:08,937 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.191e+01 8.522e+01 9.040e+01 9.801e+01 1.361e+02, threshold=1.808e+02, percent-clipped=0.0
2023-11-24 22:30:28,676 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8300, loss[loss=0.05216, simple_loss=0.07154, pruned_loss=0.008124, audio_tagging_loss=0.00827, over 14947.00 frames. ], tot_loss[loss=0.06606, simple_loss=0.08959, pruned_loss=0.01252, audio_tagging_loss=0.008741, over 3047356.57 frames. ], batch size: 59, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:30:50,905 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453200
2023-11-24 22:30:52,535 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:30:58,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=3021340.0, ans=0.0
2023-11-24 22:30:59,765 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3021340.0, ans=0.1
2023-11-24 22:31:13,139 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=3021406.6666666665, ans=0.125
2023-11-24 22:31:14,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=3021406.6666666665, ans=0.0
2023-11-24 22:31:29,827 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=3021473.3333333335, ans=0.125
2023-11-24 22:31:32,471 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8350, loss[loss=0.05378, simple_loss=0.07844, pruned_loss=0.006066, audio_tagging_loss=0.008497, over 15894.00 frames. ], tot_loss[loss=0.06555, simple_loss=0.08862, pruned_loss=0.01245, audio_tagging_loss=0.008794, over 3045165.70 frames. ], batch size: 61, lr: 1.78e-03, grad_scale: 8.0
2023-11-24 22:31:45,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=3021606.6666666665, ans=0.5
2023-11-24 22:31:48,114 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=3021606.6666666665, ans=0.125
2023-11-24 22:31:49,270 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=3021606.6666666665, ans=0.125
2023-11-24 22:31:52,716 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453250
2023-11-24 22:32:14,493 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.506e+01 8.493e+01 9.150e+01 9.803e+01 1.531e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 22:32:27,197 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=3021806.6666666665, ans=0.125
2023-11-24 22:32:34,014 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8400, loss[loss=0.07438, simple_loss=0.1049, pruned_loss=0.0145, audio_tagging_loss=0.007447, over 15195.00 frames. ], tot_loss[loss=0.06573, simple_loss=0.08915, pruned_loss=0.01254, audio_tagging_loss=0.008612, over 3038230.59 frames. ], batch size: 55, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:32:50,214 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3021940.0, ans=0.125
2023-11-24 22:32:54,793 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453300
2023-11-24 22:33:26,335 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3022140.0, ans=0.125
2023-11-24 22:33:30,461 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3022140.0, ans=0.1
2023-11-24 22:33:30,507 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3022140.0, ans=0.0
2023-11-24 22:33:31,819 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3022140.0, ans=0.125
2023-11-24 22:33:36,192 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8450, loss[loss=0.08014, simple_loss=0.1039, pruned_loss=0.01871, audio_tagging_loss=0.009475, over 15852.00 frames. ], tot_loss[loss=0.06649, simple_loss=0.09018, pruned_loss=0.01268, audio_tagging_loss=0.008716, over 3041500.93 frames. ], batch size: 60, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:33:36,591 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=3022206.6666666665, ans=0.125
2023-11-24 22:33:37,683 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3022206.6666666665, ans=0.125
2023-11-24 22:33:58,116 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453350
2023-11-24 22:34:07,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=3022340.0, ans=0.07
2023-11-24 22:34:11,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3022340.0, ans=0.125
2023-11-24 22:34:18,674 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.351e+01 8.708e+01 9.231e+01 1.022e+02 1.410e+02, threshold=1.846e+02, percent-clipped=0.0
2023-11-24 22:34:38,770 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8500, loss[loss=0.05232, simple_loss=0.07077, pruned_loss=0.007038, audio_tagging_loss=0.009898, over 15351.00 frames. ], tot_loss[loss=0.06603, simple_loss=0.08951, pruned_loss=0.01252, audio_tagging_loss=0.008761, over 3041932.63 frames. ], batch size: 58, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:34:55,023 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3022606.6666666665, ans=0.1
2023-11-24 22:34:59,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453400
2023-11-24 22:35:01,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3022606.6666666665, ans=0.1
2023-11-24 22:35:16,070 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.self_attn_weights.pos_emb_skip_rate, batch_count=3022740.0, ans=0.0
2023-11-24 22:35:36,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.prob, batch_count=3022806.6666666665, ans=0.125
2023-11-24 22:35:41,412 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8550, loss[loss=0.05993, simple_loss=0.08583, pruned_loss=0.009391, audio_tagging_loss=0.007623, over 15800.00 frames. ], tot_loss[loss=0.06596, simple_loss=0.08947, pruned_loss=0.01241, audio_tagging_loss=0.008808, over 3047298.46 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:36:02,321 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453450
2023-11-24 22:36:16,949 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=3023006.6666666665, ans=0.125
2023-11-24 22:36:21,604 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3023073.3333333335, ans=0.125
2023-11-24 22:36:23,755 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.604e+01 8.608e+01 9.249e+01 9.944e+01 1.269e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 22:36:24,016 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3023073.3333333335, ans=0.125
2023-11-24 22:36:27,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3023073.3333333335, ans=0.125
2023-11-24 22:36:43,372 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8600, loss[loss=0.06729, simple_loss=0.09588, pruned_loss=0.00887, audio_tagging_loss=0.01048, over 15797.00 frames. ], tot_loss[loss=0.06586, simple_loss=0.08936, pruned_loss=0.01236, audio_tagging_loss=0.008821, over 3051749.22 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:37:05,766 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453500
2023-11-24 22:37:30,144 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=3023406.6666666665, ans=0.125
2023-11-24 22:37:30,168 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3023406.6666666665, ans=0.125
2023-11-24 22:37:45,794 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8650, loss[loss=0.0531, simple_loss=0.06324, pruned_loss=0.00967, audio_tagging_loss=0.01181, over 16116.00 frames. ], tot_loss[loss=0.06651, simple_loss=0.09044, pruned_loss=0.01255, audio_tagging_loss=0.008745, over 3060017.10 frames. ], batch size: 63, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:37:54,518 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3023540.0, ans=0.1
2023-11-24 22:38:02,701 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3023606.6666666665, ans=0.125
2023-11-24 22:38:07,200 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453550
2023-11-24 22:38:24,079 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.91 vs. limit=15.0
2023-11-24 22:38:25,307 INFO [scaling.py:1022] (0/4) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=6.05 vs. limit=8.0
2023-11-24 22:38:25,945 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3023740.0, ans=0.125
2023-11-24 22:38:27,884 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.254e+01 8.604e+01 9.306e+01 1.022e+02 1.267e+02, threshold=1.861e+02, percent-clipped=0.0
2023-11-24 22:38:48,898 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8700, loss[loss=0.06666, simple_loss=0.09048, pruned_loss=0.01407, audio_tagging_loss=0.007351, over 14986.00 frames. ], tot_loss[loss=0.06647, simple_loss=0.09026, pruned_loss=0.01256, audio_tagging_loss=0.008774, over 3057085.26 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:38:58,731 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=3023873.3333333335, ans=0.2
2023-11-24 22:39:02,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=3023940.0, ans=0.125
2023-11-24 22:39:09,865 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453600
2023-11-24 22:39:46,780 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=512, metric=9.58 vs. limit=12.0
2023-11-24 22:39:51,308 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8750, loss[loss=0.06074, simple_loss=0.07779, pruned_loss=0.01265, audio_tagging_loss=0.009191, over 16361.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09159, pruned_loss=0.01298, audio_tagging_loss=0.008875, over 3058888.07 frames. ], batch size: 62, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:39:54,078 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3024206.6666666665, ans=0.125
2023-11-24 22:39:54,184 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:40:02,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3024273.3333333335, ans=0.125
2023-11-24 22:40:12,865 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453650
2023-11-24 22:40:32,405 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3024406.6666666665, ans=0.125
2023-11-24 22:40:33,324 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.012e+01 8.979e+01 9.554e+01 1.051e+02 1.529e+02, threshold=1.911e+02, percent-clipped=0.0
2023-11-24 22:40:52,690 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8800, loss[loss=0.075, simple_loss=0.109, pruned_loss=0.01323, audio_tagging_loss=0.007279, over 14548.00 frames. ], tot_loss[loss=0.06781, simple_loss=0.09129, pruned_loss=0.01311, audio_tagging_loss=0.009057, over 3051311.60 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:41:04,026 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=7.41 vs. limit=15.0
2023-11-24 22:41:14,689 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453700
2023-11-24 22:41:21,232 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=4.05 vs. limit=15.0
2023-11-24 22:41:25,085 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=3024673.3333333335, ans=0.2
2023-11-24 22:41:32,158 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3024740.0, ans=0.125
2023-11-24 22:41:43,528 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3024806.6666666665, ans=0.1
2023-11-24 22:41:46,546 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=3024806.6666666665, ans=0.0
2023-11-24 22:41:56,238 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8850, loss[loss=0.09038, simple_loss=0.1109, pruned_loss=0.02674, audio_tagging_loss=0.008204, over 14667.00 frames. ], tot_loss[loss=0.06819, simple_loss=0.09165, pruned_loss=0.01329, audio_tagging_loss=0.009073, over 3051724.89 frames. ], batch size: 53, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:41:57,735 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:42:02,898 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=13.56 vs. limit=15.0
2023-11-24 22:42:05,738 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/1Dq7QH61iXQ_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:42:06,096 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:42:07,329 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=3024940.0, ans=0.2
2023-11-24 22:42:09,688 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3024940.0, ans=0.125
2023-11-24 22:42:16,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453750
2023-11-24 22:42:36,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=3025073.3333333335, ans=0.125
2023-11-24 22:42:37,812 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.566e+01 8.570e+01 9.101e+01 9.826e+01 1.259e+02, threshold=1.820e+02, percent-clipped=0.0
2023-11-24 22:42:52,980 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3025140.0, ans=0.1
2023-11-24 22:42:57,360 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8900, loss[loss=0.05782, simple_loss=0.0801, pruned_loss=0.009989, audio_tagging_loss=0.007782, over 15059.00 frames. ], tot_loss[loss=0.06732, simple_loss=0.09081, pruned_loss=0.01299, audio_tagging_loss=0.008928, over 3049437.22 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:43:02,757 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=3025206.6666666665, ans=0.2
2023-11-24 22:43:03,145 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.35 vs. limit=15.0
2023-11-24 22:43:12,569 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=3025273.3333333335, ans=0.0
2023-11-24 22:43:18,692 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453800
2023-11-24 22:43:22,262 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=3025340.0, ans=0.09899494936611666
2023-11-24 22:43:24,410 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=3025340.0, ans=0.125
2023-11-24 22:43:30,886 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.50 vs. limit=12.0
2023-11-24 22:43:55,205 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3025473.3333333335, ans=0.1
2023-11-24 22:43:59,732 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 8950, loss[loss=0.06707, simple_loss=0.09361, pruned_loss=0.01152, audio_tagging_loss=0.008739, over 15834.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09044, pruned_loss=0.01288, audio_tagging_loss=0.008856, over 3053644.56 frames. ], batch size: 57, lr: 1.78e-03, grad_scale: 32.0
2023-11-24 22:44:21,886 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453850
2023-11-24 22:44:42,863 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=3025740.0, ans=0.125
2023-11-24 22:44:43,570 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.683e+01 8.565e+01 9.194e+01 9.956e+01 1.408e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 22:44:49,118 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=512, metric=7.24 vs. limit=15.0
2023-11-24 22:44:51,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=3025806.6666666665, ans=0.0
2023-11-24 22:44:54,167 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:44:55,999 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.40 vs. limit=22.5
2023-11-24 22:44:58,126 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3025806.6666666665, ans=0.1
2023-11-24 22:45:01,857 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=3025873.3333333335, ans=0.2
2023-11-24 22:45:02,692 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9000, loss[loss=0.05907, simple_loss=0.08337, pruned_loss=0.008682, audio_tagging_loss=0.008702, over 15106.00 frames. ], tot_loss[loss=0.06695, simple_loss=0.09075, pruned_loss=0.01277, audio_tagging_loss=0.008802, over 3053716.58 frames. ], batch size: 56, lr: 1.78e-03, grad_scale: 16.0
2023-11-24 22:45:02,695 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 22:45:46,046 INFO [train_asr.py:1253] (0/4) Epoch 38, validation: loss=0.05855, simple_loss=0.05069, pruned_loss=0.005085, audio_tagging_loss=0.02812, over 4681554.00 frames. 
2023-11-24 22:45:46,046 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 22:45:49,154 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=7.95 vs. limit=15.0
2023-11-24 22:45:53,379 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=3025873.3333333335, ans=0.09899494936611666
2023-11-24 22:46:07,185 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453900
2023-11-24 22:46:09,854 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=3026006.6666666665, ans=0.2
2023-11-24 22:46:21,088 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3026006.6666666665, ans=0.0
2023-11-24 22:46:47,368 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9050, loss[loss=0.09378, simple_loss=0.1349, pruned_loss=0.0198, audio_tagging_loss=0.006555, over 14926.00 frames. ], tot_loss[loss=0.06718, simple_loss=0.09122, pruned_loss=0.01281, audio_tagging_loss=0.008756, over 3050504.28 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:47:08,728 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 453950
2023-11-24 22:47:11,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3026340.0, ans=0.125
2023-11-24 22:47:18,377 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=3026340.0, ans=0.2
2023-11-24 22:47:22,269 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.87 vs. limit=10.0
2023-11-24 22:47:30,937 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.553e+01 8.549e+01 9.034e+01 9.764e+01 1.451e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 22:47:40,937 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=3026473.3333333335, ans=0.125
2023-11-24 22:47:43,221 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=3026473.3333333335, ans=0.0
2023-11-24 22:47:50,110 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9100, loss[loss=0.06796, simple_loss=0.08802, pruned_loss=0.01674, audio_tagging_loss=0.007206, over 14948.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09115, pruned_loss=0.01293, audio_tagging_loss=0.00872, over 3053151.31 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:48:01,850 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=3026606.6666666665, ans=0.2
2023-11-24 22:48:03,270 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.77 vs. limit=15.0
2023-11-24 22:48:11,699 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454000
2023-11-24 22:48:11,950 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3026606.6666666665, ans=0.125
2023-11-24 22:48:27,807 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3026740.0, ans=0.125
2023-11-24 22:48:37,159 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=3026740.0, ans=0.125
2023-11-24 22:48:40,693 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer2.prob, batch_count=3026806.6666666665, ans=0.125
2023-11-24 22:48:53,405 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9150, loss[loss=0.0752, simple_loss=0.1066, pruned_loss=0.01343, audio_tagging_loss=0.008475, over 16167.00 frames. ], tot_loss[loss=0.06733, simple_loss=0.09166, pruned_loss=0.01289, audio_tagging_loss=0.008606, over 3048729.32 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:48:53,641 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3026873.3333333335, ans=0.125
2023-11-24 22:49:14,969 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454050
2023-11-24 22:49:17,659 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3027006.6666666665, ans=0.125
2023-11-24 22:49:29,416 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3027073.3333333335, ans=0.1
2023-11-24 22:49:36,304 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.113e+01 8.421e+01 9.061e+01 9.734e+01 1.251e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 22:49:43,629 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3027140.0, ans=0.1
2023-11-24 22:49:51,282 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.50 vs. limit=15.0
2023-11-24 22:49:55,225 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9200, loss[loss=0.07304, simple_loss=0.1132, pruned_loss=0.0112, audio_tagging_loss=0.005219, over 14999.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09171, pruned_loss=0.01296, audio_tagging_loss=0.008588, over 3044995.91 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 22:50:11,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3027273.3333333335, ans=0.125
2023-11-24 22:50:16,197 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454100
2023-11-24 22:50:18,726 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3027340.0, ans=0.125
2023-11-24 22:50:26,529 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=3027340.0, ans=0.0
2023-11-24 22:50:42,600 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3027406.6666666665, ans=0.125
2023-11-24 22:50:55,084 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_positive, batch_count=3027473.3333333335, ans=0.05
2023-11-24 22:50:57,390 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9250, loss[loss=0.06445, simple_loss=0.08604, pruned_loss=0.01279, audio_tagging_loss=0.008634, over 15952.00 frames. ], tot_loss[loss=0.06723, simple_loss=0.09135, pruned_loss=0.01295, audio_tagging_loss=0.008594, over 3057788.11 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:51:02,343 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=3027540.0, ans=0.0
2023-11-24 22:51:02,722 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.08 vs. limit=6.0
2023-11-24 22:51:14,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3027606.6666666665, ans=0.1
2023-11-24 22:51:18,299 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454150
2023-11-24 22:51:18,832 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_whiten, num_groups=1, num_channels=512, metric=7.87 vs. limit=15.0
2023-11-24 22:51:41,749 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.157e+01 8.619e+01 9.253e+01 1.002e+02 1.218e+02, threshold=1.851e+02, percent-clipped=0.0
2023-11-24 22:51:53,925 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.prob, batch_count=3027806.6666666665, ans=0.125
2023-11-24 22:51:56,163 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3027806.6666666665, ans=0.1
2023-11-24 22:51:57,296 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3027873.3333333335, ans=0.125
2023-11-24 22:51:58,232 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9300, loss[loss=0.05541, simple_loss=0.07288, pruned_loss=0.01024, audio_tagging_loss=0.008732, over 14871.00 frames. ], tot_loss[loss=0.06691, simple_loss=0.09088, pruned_loss=0.01289, audio_tagging_loss=0.008588, over 3059115.19 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:52:01,269 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=3027873.3333333335, ans=0.015
2023-11-24 22:52:08,166 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.37 vs. limit=15.0
2023-11-24 22:52:17,314 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3027940.0, ans=0.1
2023-11-24 22:52:17,438 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3027940.0, ans=0.125
2023-11-24 22:52:20,089 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454200
2023-11-24 22:52:20,609 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=5.56 vs. limit=10.0
2023-11-24 22:52:37,582 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3028073.3333333335, ans=0.125
2023-11-24 22:52:48,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3028140.0, ans=0.125
2023-11-24 22:53:00,909 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9350, loss[loss=0.07237, simple_loss=0.09502, pruned_loss=0.01705, audio_tagging_loss=0.007813, over 15613.00 frames. ], tot_loss[loss=0.06702, simple_loss=0.09079, pruned_loss=0.01302, audio_tagging_loss=0.008608, over 3056240.37 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:53:08,784 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=3028206.6666666665, ans=10.0
2023-11-24 22:53:22,137 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454250
2023-11-24 22:53:30,614 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=3028340.0, ans=0.0
2023-11-24 22:53:35,758 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=3028340.0, ans=0.125
2023-11-24 22:53:45,747 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.676e+01 8.804e+01 9.416e+01 1.006e+02 1.917e+02, threshold=1.883e+02, percent-clipped=1.0
2023-11-24 22:53:57,961 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.ff3_skip_rate, batch_count=3028473.3333333335, ans=0.0
2023-11-24 22:54:03,458 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9400, loss[loss=0.06158, simple_loss=0.08395, pruned_loss=0.01167, audio_tagging_loss=0.00794, over 15095.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09168, pruned_loss=0.01323, audio_tagging_loss=0.008695, over 3052518.41 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:54:08,339 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3028540.0, ans=0.0
2023-11-24 22:54:13,176 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3028540.0, ans=0.0
2023-11-24 22:54:14,323 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=3028606.6666666665, ans=0.2
2023-11-24 22:54:21,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.attention_skip_rate, batch_count=3028606.6666666665, ans=0.0
2023-11-24 22:54:24,307 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454300
2023-11-24 22:54:38,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3028740.0, ans=0.0
2023-11-24 22:54:54,924 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.28 vs. limit=15.0
2023-11-24 22:54:56,795 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3028806.6666666665, ans=0.125
2023-11-24 22:55:01,323 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/jmSuJWEIizA_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 22:55:05,010 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9450, loss[loss=0.05495, simple_loss=0.07492, pruned_loss=0.01073, audio_tagging_loss=0.00676, over 15970.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09119, pruned_loss=0.01319, audio_tagging_loss=0.008816, over 3047146.80 frames. ], batch size: 62, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:55:12,895 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3028873.3333333335, ans=0.1
2023-11-24 22:55:15,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.16 vs. limit=10.0
2023-11-24 22:55:15,731 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.58 vs. limit=15.0
2023-11-24 22:55:25,349 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3028940.0, ans=0.1
2023-11-24 22:55:26,442 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454350
2023-11-24 22:55:49,909 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.559e+01 8.713e+01 9.194e+01 9.823e+01 1.241e+02, threshold=1.839e+02, percent-clipped=0.0
2023-11-24 22:55:54,939 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3029140.0, ans=0.125
2023-11-24 22:56:00,112 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 22:56:07,695 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9500, loss[loss=0.04851, simple_loss=0.06357, pruned_loss=0.006389, audio_tagging_loss=0.01033, over 14977.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09101, pruned_loss=0.013, audio_tagging_loss=0.008913, over 3046949.94 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:56:19,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_module2.balancer1.max_abs, batch_count=3029273.3333333335, ans=10.0
2023-11-24 22:56:29,263 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454400
2023-11-24 22:56:49,400 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=3029406.6666666665, ans=0.0
2023-11-24 22:56:54,075 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=3029406.6666666665, ans=0.0
2023-11-24 22:57:05,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=3029473.3333333335, ans=0.0
2023-11-24 22:57:10,764 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9550, loss[loss=0.06235, simple_loss=0.08513, pruned_loss=0.008949, audio_tagging_loss=0.01084, over 14904.00 frames. ], tot_loss[loss=0.06719, simple_loss=0.09041, pruned_loss=0.01294, audio_tagging_loss=0.009048, over 3040933.57 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 22:57:18,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3029540.0, ans=0.1
2023-11-24 22:57:31,107 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454450
2023-11-24 22:57:56,029 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.670e+01 8.429e+01 9.044e+01 9.550e+01 1.234e+02, threshold=1.809e+02, percent-clipped=0.0
2023-11-24 22:58:00,055 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=3029806.6666666665, ans=0.2
2023-11-24 22:58:13,001 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9600, loss[loss=0.069, simple_loss=0.08994, pruned_loss=0.01223, audio_tagging_loss=0.0118, over 15777.00 frames. ], tot_loss[loss=0.06731, simple_loss=0.09068, pruned_loss=0.01294, audio_tagging_loss=0.009036, over 3043007.73 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 22:58:18,524 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.47 vs. limit=6.0
2023-11-24 22:58:34,176 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454500
2023-11-24 22:59:06,516 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=384, metric=6.99 vs. limit=15.0
2023-11-24 22:59:14,832 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9650, loss[loss=0.06824, simple_loss=0.09708, pruned_loss=0.01158, audio_tagging_loss=0.00812, over 14150.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.09091, pruned_loss=0.01299, audio_tagging_loss=0.008967, over 3042581.47 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 22:59:21,180 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3030206.6666666665, ans=0.0
2023-11-24 22:59:36,365 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=384, metric=9.15 vs. limit=10.0
2023-11-24 22:59:36,987 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454550
2023-11-24 22:59:45,542 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3030340.0, ans=0.125
2023-11-24 22:59:51,671 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=384, metric=16.78 vs. limit=22.5
2023-11-24 23:00:01,103 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.083e+01 8.723e+01 9.168e+01 1.005e+02 1.303e+02, threshold=1.834e+02, percent-clipped=0.0
2023-11-24 23:00:03,044 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.03 vs. limit=6.0
2023-11-24 23:00:07,386 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3030473.3333333335, ans=0.125
2023-11-24 23:00:11,072 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3030473.3333333335, ans=0.125
2023-11-24 23:00:18,277 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9700, loss[loss=0.04883, simple_loss=0.06406, pruned_loss=0.009643, audio_tagging_loss=0.007157, over 14827.00 frames. ], tot_loss[loss=0.06772, simple_loss=0.09168, pruned_loss=0.01311, audio_tagging_loss=0.008768, over 3037942.65 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:00:22,154 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3030540.0, ans=0.0
2023-11-24 23:00:38,665 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454600
2023-11-24 23:00:55,469 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3030740.0, ans=0.0
2023-11-24 23:01:03,970 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=4.66 vs. limit=15.0
2023-11-24 23:01:19,918 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3030873.3333333335, ans=0.1
2023-11-24 23:01:19,981 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3030873.3333333335, ans=0.125
2023-11-24 23:01:20,991 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9750, loss[loss=0.07074, simple_loss=0.0969, pruned_loss=0.0128, audio_tagging_loss=0.009492, over 15480.00 frames. ], tot_loss[loss=0.06697, simple_loss=0.09077, pruned_loss=0.01284, audio_tagging_loss=0.008747, over 3041573.19 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:01:26,120 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=3030873.3333333335, ans=0.0
2023-11-24 23:01:29,852 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3030873.3333333335, ans=0.1
2023-11-24 23:01:41,966 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454650
2023-11-24 23:01:42,442 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.04 vs. limit=15.0
2023-11-24 23:02:06,526 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.511e+01 8.357e+01 8.880e+01 9.692e+01 1.346e+02, threshold=1.776e+02, percent-clipped=0.0
2023-11-24 23:02:09,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.ff2_skip_rate, batch_count=3031140.0, ans=0.0
2023-11-24 23:02:12,824 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3031140.0, ans=0.0
2023-11-24 23:02:13,075 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.01 vs. limit=15.0
2023-11-24 23:02:15,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=3031140.0, ans=0.04949747468305833
2023-11-24 23:02:22,312 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.85 vs. limit=6.0
2023-11-24 23:02:22,677 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9800, loss[loss=0.06434, simple_loss=0.08737, pruned_loss=0.008896, audio_tagging_loss=0.01176, over 14947.00 frames. ], tot_loss[loss=0.06736, simple_loss=0.0915, pruned_loss=0.01292, audio_tagging_loss=0.008694, over 3043646.77 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:02:36,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=3031273.3333333335, ans=0.0
2023-11-24 23:02:44,575 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454700
2023-11-24 23:02:44,805 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3031273.3333333335, ans=0.0
2023-11-24 23:02:44,948 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=14.50 vs. limit=15.0
2023-11-24 23:02:47,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3031340.0, ans=0.0
2023-11-24 23:02:51,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3031340.0, ans=0.0
2023-11-24 23:03:06,997 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3031406.6666666665, ans=0.1
2023-11-24 23:03:16,406 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/Bo4LcZjitzU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:03:25,431 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9850, loss[loss=0.07248, simple_loss=0.1126, pruned_loss=0.01019, audio_tagging_loss=0.005993, over 15839.00 frames. ], tot_loss[loss=0.06756, simple_loss=0.09152, pruned_loss=0.01304, audio_tagging_loss=0.008759, over 3041861.53 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:03:31,817 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.skip_rate, batch_count=3031540.0, ans=0.09899494936611666
2023-11-24 23:03:36,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=3031540.0, ans=0.025
2023-11-24 23:03:42,741 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.67 vs. limit=12.0
2023-11-24 23:03:43,750 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=3031606.6666666665, ans=0.0
2023-11-24 23:03:47,139 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454750
2023-11-24 23:03:50,869 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.ff2_skip_rate, batch_count=3031673.3333333335, ans=0.0
2023-11-24 23:03:52,470 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.70 vs. limit=6.0
2023-11-24 23:04:03,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=3031740.0, ans=0.95
2023-11-24 23:04:03,769 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.02 vs. limit=6.0
2023-11-24 23:04:11,386 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.76 vs. limit=15.0
2023-11-24 23:04:13,182 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.582e+01 8.592e+01 9.130e+01 1.016e+02 1.240e+02, threshold=1.826e+02, percent-clipped=0.0
2023-11-24 23:04:28,564 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9900, loss[loss=0.0612, simple_loss=0.07586, pruned_loss=0.01334, audio_tagging_loss=0.009928, over 14488.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09086, pruned_loss=0.01307, audio_tagging_loss=0.008738, over 3041808.78 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:04:34,851 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3031873.3333333335, ans=0.125
2023-11-24 23:04:40,138 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.80 vs. limit=10.0
2023-11-24 23:04:40,843 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=3031940.0, ans=0.0
2023-11-24 23:04:48,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3031940.0, ans=0.125
2023-11-24 23:04:49,708 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454800
2023-11-24 23:04:50,266 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=512, metric=3.02 vs. limit=15.0
2023-11-24 23:04:53,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3032006.6666666665, ans=0.1
2023-11-24 23:04:53,802 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=3032006.6666666665, ans=0.125
2023-11-24 23:05:05,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3032073.3333333335, ans=0.125
2023-11-24 23:05:21,292 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=5.35 vs. limit=15.0
2023-11-24 23:05:22,027 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:05:25,697 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:05:31,866 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 9950, loss[loss=0.06833, simple_loss=0.08547, pruned_loss=0.01407, audio_tagging_loss=0.01153, over 15315.00 frames. ], tot_loss[loss=0.06748, simple_loss=0.09133, pruned_loss=0.01306, audio_tagging_loss=0.008754, over 3044394.86 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:05:53,017 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454850
2023-11-24 23:06:15,436 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=12.45 vs. limit=15.0
2023-11-24 23:06:16,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3032406.6666666665, ans=0.0
2023-11-24 23:06:18,927 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.147e+01 8.463e+01 8.965e+01 9.807e+01 1.516e+02, threshold=1.793e+02, percent-clipped=0.0
2023-11-24 23:06:33,547 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.89 vs. limit=6.0
2023-11-24 23:06:33,776 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10000, loss[loss=0.05801, simple_loss=0.07647, pruned_loss=0.01112, audio_tagging_loss=0.008655, over 15768.00 frames. ], tot_loss[loss=0.06734, simple_loss=0.09126, pruned_loss=0.01298, audio_tagging_loss=0.008733, over 3044860.78 frames. ], batch size: 61, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:06:33,954 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3032540.0, ans=0.1
2023-11-24 23:06:54,908 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454900
2023-11-24 23:07:05,047 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3032673.3333333335, ans=0.125
2023-11-24 23:07:07,638 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3032673.3333333335, ans=0.1
2023-11-24 23:07:35,229 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10050, loss[loss=0.08105, simple_loss=0.1161, pruned_loss=0.01605, audio_tagging_loss=0.006933, over 15264.00 frames. ], tot_loss[loss=0.06741, simple_loss=0.0912, pruned_loss=0.01312, audio_tagging_loss=0.008693, over 3040942.45 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:07:47,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=3032940.0, ans=0.05
2023-11-24 23:07:53,395 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass_mid.scale_min, batch_count=3032940.0, ans=0.2
2023-11-24 23:07:56,658 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 454950
2023-11-24 23:07:58,498 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=10.51 vs. limit=15.0
2023-11-24 23:08:00,627 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=384, metric=3.28 vs. limit=15.0
2023-11-24 23:08:14,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=3033073.3333333335, ans=0.0
2023-11-24 23:08:22,922 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3033073.3333333335, ans=0.0
2023-11-24 23:08:23,659 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.056e+01 8.450e+01 9.036e+01 9.785e+01 1.299e+02, threshold=1.807e+02, percent-clipped=0.0
2023-11-24 23:08:26,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=3033140.0, ans=0.05
2023-11-24 23:08:29,189 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:08:37,246 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10100, loss[loss=0.07608, simple_loss=0.1101, pruned_loss=0.01378, audio_tagging_loss=0.007242, over 15538.00 frames. ], tot_loss[loss=0.06771, simple_loss=0.0915, pruned_loss=0.01319, audio_tagging_loss=0.008776, over 3044314.15 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:08:56,345 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=3033273.3333333335, ans=0.2
2023-11-24 23:08:57,563 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=3033273.3333333335, ans=0.0
2023-11-24 23:08:59,053 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455000
2023-11-24 23:09:02,149 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=3033340.0, ans=0.0
2023-11-24 23:09:26,554 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/_eq1Ry0UZGU_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:09:40,114 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10150, loss[loss=0.08328, simple_loss=0.1168, pruned_loss=0.01719, audio_tagging_loss=0.007689, over 14834.00 frames. ], tot_loss[loss=0.06791, simple_loss=0.09192, pruned_loss=0.01316, audio_tagging_loss=0.008788, over 3048092.97 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:09:45,783 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3033540.0, ans=0.125
2023-11-24 23:09:49,162 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.80 vs. limit=10.0
2023-11-24 23:10:01,339 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455050
2023-11-24 23:10:07,819 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/cw-21cbk02A_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:10:29,123 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.319e+01 8.639e+01 9.334e+01 9.887e+01 1.234e+02, threshold=1.867e+02, percent-clipped=0.0
2023-11-24 23:10:38,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=3033806.6666666665, ans=0.0
2023-11-24 23:10:43,084 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10200, loss[loss=0.06629, simple_loss=0.08548, pruned_loss=0.01349, audio_tagging_loss=0.01006, over 14789.00 frames. ], tot_loss[loss=0.06776, simple_loss=0.09143, pruned_loss=0.01311, audio_tagging_loss=0.008941, over 3049357.43 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:10:54,001 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=3033940.0, ans=0.0
2023-11-24 23:11:04,534 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/hOT6Yokob90_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:11:04,592 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455100
2023-11-24 23:11:45,353 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10250, loss[loss=0.06281, simple_loss=0.08871, pruned_loss=0.01016, audio_tagging_loss=0.00829, over 15884.00 frames. ], tot_loss[loss=0.0672, simple_loss=0.0905, pruned_loss=0.01288, audio_tagging_loss=0.009076, over 3050140.28 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:11:59,081 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=3034273.3333333335, ans=0.0
2023-11-24 23:12:02,674 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=3034273.3333333335, ans=0.0
2023-11-24 23:12:06,513 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455150
2023-11-24 23:12:14,591 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:12:26,209 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3034406.6666666665, ans=0.1
2023-11-24 23:12:33,698 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.500e+01 8.646e+01 9.167e+01 9.835e+01 1.179e+02, threshold=1.833e+02, percent-clipped=0.0
2023-11-24 23:12:39,223 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3034473.3333333335, ans=0.125
2023-11-24 23:12:47,009 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10300, loss[loss=0.08721, simple_loss=0.1244, pruned_loss=0.01869, audio_tagging_loss=0.006346, over 15245.00 frames. ], tot_loss[loss=0.06765, simple_loss=0.09126, pruned_loss=0.01296, audio_tagging_loss=0.009052, over 3052830.95 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:12:51,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.hidden_balancer.prob, batch_count=3034540.0, ans=0.125
2023-11-24 23:13:01,734 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3034606.6666666665, ans=0.125
2023-11-24 23:13:05,248 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=3034606.6666666665, ans=0.2
2023-11-24 23:13:08,700 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455200
2023-11-24 23:13:11,672 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=3034673.3333333335, ans=0.05
2023-11-24 23:13:16,215 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=3034673.3333333335, ans=10.0
2023-11-24 23:13:28,101 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3034740.0, ans=0.0
2023-11-24 23:13:29,205 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:13:31,594 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3034740.0, ans=0.1
2023-11-24 23:13:36,468 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.bypass_mid.scale_min, batch_count=3034806.6666666665, ans=0.2
2023-11-24 23:13:38,076 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=20.11 vs. limit=22.5
2023-11-24 23:13:50,482 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10350, loss[loss=0.07137, simple_loss=0.09195, pruned_loss=0.01645, audio_tagging_loss=0.008944, over 13694.00 frames. ], tot_loss[loss=0.0674, simple_loss=0.09079, pruned_loss=0.01281, audio_tagging_loss=0.009193, over 3056051.57 frames. ], batch size: 53, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:13:50,878 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3034873.3333333335, ans=0.125
2023-11-24 23:13:51,962 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:13:55,681 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_skip_rate, batch_count=3034873.3333333335, ans=0.0
2023-11-24 23:14:11,334 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455250
2023-11-24 23:14:17,163 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=384, metric=9.72 vs. limit=15.0
2023-11-24 23:14:35,682 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=22.5
2023-11-24 23:14:37,820 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.conv_module2.whiten, num_groups=1, num_channels=512, metric=12.48 vs. limit=15.0
2023-11-24 23:14:38,310 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.131e+01 8.693e+01 9.178e+01 1.025e+02 1.318e+02, threshold=1.836e+02, percent-clipped=0.0
2023-11-24 23:14:48,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.79 vs. limit=6.0
2023-11-24 23:14:51,345 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10400, loss[loss=0.06741, simple_loss=0.09165, pruned_loss=0.01393, audio_tagging_loss=0.007657, over 15837.00 frames. ], tot_loss[loss=0.06742, simple_loss=0.09109, pruned_loss=0.01274, audio_tagging_loss=0.009134, over 3058662.54 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:14:52,894 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=3035206.6666666665, ans=0.125
2023-11-24 23:15:07,206 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=3035273.3333333335, ans=0.2
2023-11-24 23:15:10,037 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.31 vs. limit=15.0
2023-11-24 23:15:12,893 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455300
2023-11-24 23:15:13,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3035273.3333333335, ans=0.125
2023-11-24 23:15:17,556 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3035340.0, ans=0.125
2023-11-24 23:15:23,275 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3035340.0, ans=0.1
2023-11-24 23:15:23,653 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=3035340.0, ans=15.0
2023-11-24 23:15:29,520 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0
2023-11-24 23:15:40,171 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=384, metric=6.96 vs. limit=12.0
2023-11-24 23:15:53,554 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10450, loss[loss=0.08238, simple_loss=0.1115, pruned_loss=0.01986, audio_tagging_loss=0.006793, over 14845.00 frames. ], tot_loss[loss=0.06778, simple_loss=0.09167, pruned_loss=0.0129, audio_tagging_loss=0.009048, over 3058598.24 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:15:54,867 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=3035540.0, ans=0.125
2023-11-24 23:15:57,801 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=8.68 vs. limit=15.0
2023-11-24 23:16:09,188 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:16:14,329 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=384, metric=20.24 vs. limit=22.5
2023-11-24 23:16:14,908 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455350
2023-11-24 23:16:16,504 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=11.47 vs. limit=15.0
2023-11-24 23:16:20,494 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.00 vs. limit=22.5
2023-11-24 23:16:20,959 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3035673.3333333335, ans=0.1
2023-11-24 23:16:28,082 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3035673.3333333335, ans=0.125
2023-11-24 23:16:37,645 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3035740.0, ans=0.0
2023-11-24 23:16:42,053 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.009e+01 8.886e+01 9.517e+01 1.018e+02 1.385e+02, threshold=1.903e+02, percent-clipped=0.0
2023-11-24 23:16:45,916 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3035806.6666666665, ans=0.1
2023-11-24 23:16:50,205 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=6.16 vs. limit=15.0
2023-11-24 23:16:56,071 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10500, loss[loss=0.06352, simple_loss=0.08498, pruned_loss=0.01093, audio_tagging_loss=0.0101, over 14511.00 frames. ], tot_loss[loss=0.06758, simple_loss=0.09125, pruned_loss=0.01297, audio_tagging_loss=0.008993, over 3055741.00 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:16:58,585 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3035873.3333333335, ans=0.125
2023-11-24 23:17:03,874 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.90 vs. limit=22.5
2023-11-24 23:17:08,756 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=8.48 vs. limit=10.0
2023-11-24 23:17:10,713 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3035940.0, ans=0.0
2023-11-24 23:17:13,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3035940.0, ans=0.1
2023-11-24 23:17:16,593 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455400
2023-11-24 23:17:16,702 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=3035940.0, ans=0.04949747468305833
2023-11-24 23:17:23,525 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3036006.6666666665, ans=0.0
2023-11-24 23:17:47,880 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward2.hidden_balancer.prob, batch_count=3036140.0, ans=0.125
2023-11-24 23:17:57,337 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.conv_skip_rate, batch_count=3036206.6666666665, ans=0.0
2023-11-24 23:17:58,256 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10550, loss[loss=0.05715, simple_loss=0.0756, pruned_loss=0.01008, audio_tagging_loss=0.00927, over 15830.00 frames. ], tot_loss[loss=0.06701, simple_loss=0.09054, pruned_loss=0.01282, audio_tagging_loss=0.008921, over 3060525.38 frames. ], batch size: 61, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:18:10,929 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=3036273.3333333335, ans=0.0
2023-11-24 23:18:19,463 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455450
2023-11-24 23:18:20,791 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3036273.3333333335, ans=0.1
2023-11-24 23:18:29,536 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=384, metric=6.61 vs. limit=12.0
2023-11-24 23:18:38,615 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module2.balancer1.max_abs, batch_count=3036406.6666666665, ans=10.0
2023-11-24 23:18:46,506 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.999e+01 8.547e+01 9.162e+01 9.903e+01 1.156e+02, threshold=1.832e+02, percent-clipped=0.0
2023-11-24 23:19:00,140 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10600, loss[loss=0.05812, simple_loss=0.0767, pruned_loss=0.01137, audio_tagging_loss=0.0084, over 14624.00 frames. ], tot_loss[loss=0.06664, simple_loss=0.09052, pruned_loss=0.01258, audio_tagging_loss=0.008798, over 3060185.82 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:19:04,665 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3036540.0, ans=0.0
2023-11-24 23:19:04,695 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3036540.0, ans=0.0
2023-11-24 23:19:06,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=3036540.0, ans=0.025
2023-11-24 23:19:19,416 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=512, metric=19.93 vs. limit=22.5
2023-11-24 23:19:22,452 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455500
2023-11-24 23:19:22,677 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.3.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:19:31,036 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=3036673.3333333335, ans=0.09899494936611666
2023-11-24 23:19:31,363 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=6.61 vs. limit=10.0
2023-11-24 23:19:37,900 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=3036740.0, ans=0.125
2023-11-24 23:20:01,907 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3036806.6666666665, ans=0.125
2023-11-24 23:20:03,966 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10650, loss[loss=0.08468, simple_loss=0.116, pruned_loss=0.01882, audio_tagging_loss=0.007885, over 14445.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.09087, pruned_loss=0.01269, audio_tagging_loss=0.008864, over 3057177.45 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:20:05,476 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3036873.3333333335, ans=0.125
2023-11-24 23:20:06,572 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3036873.3333333335, ans=0.0
2023-11-24 23:20:21,117 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.self_attn2.whiten, num_groups=1, num_channels=512, metric=15.92 vs. limit=22.5
2023-11-24 23:20:24,049 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455550
2023-11-24 23:20:25,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3036940.0, ans=0.125
2023-11-24 23:20:31,871 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3037006.6666666665, ans=0.0
2023-11-24 23:20:37,819 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:20:52,474 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.272e+01 8.714e+01 9.529e+01 1.031e+02 1.468e+02, threshold=1.906e+02, percent-clipped=0.0
2023-11-24 23:20:57,877 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.65 vs. limit=15.0
2023-11-24 23:21:05,489 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10700, loss[loss=0.05111, simple_loss=0.06574, pruned_loss=0.008098, audio_tagging_loss=0.01015, over 15897.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.0909, pruned_loss=0.01263, audio_tagging_loss=0.008698, over 3057339.58 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:21:08,187 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.balancer1.prob, batch_count=3037206.6666666665, ans=0.125
2023-11-24 23:21:09,257 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3037206.6666666665, ans=0.1
2023-11-24 23:21:26,165 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455600
2023-11-24 23:21:30,016 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=512, metric=3.83 vs. limit=15.0
2023-11-24 23:21:39,455 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.73 vs. limit=10.0
2023-11-24 23:22:07,801 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10750, loss[loss=0.06402, simple_loss=0.08863, pruned_loss=0.01253, audio_tagging_loss=0.007169, over 14999.00 frames. ], tot_loss[loss=0.06715, simple_loss=0.09142, pruned_loss=0.01276, audio_tagging_loss=0.008676, over 3049586.18 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:22:25,135 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=3037606.6666666665, ans=0.125
2023-11-24 23:22:26,766 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.11 vs. limit=12.0
2023-11-24 23:22:29,851 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455650
2023-11-24 23:22:47,748 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=3037740.0, ans=0.04949747468305833
2023-11-24 23:22:55,842 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=7.66 vs. limit=15.0
2023-11-24 23:22:56,323 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.273e+01 8.544e+01 9.121e+01 9.679e+01 1.132e+02, threshold=1.824e+02, percent-clipped=0.0
2023-11-24 23:23:02,552 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3037806.6666666665, ans=0.125
2023-11-24 23:23:10,661 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10800, loss[loss=0.06114, simple_loss=0.09098, pruned_loss=0.007391, audio_tagging_loss=0.008257, over 15732.00 frames. ], tot_loss[loss=0.06725, simple_loss=0.09164, pruned_loss=0.01276, audio_tagging_loss=0.008671, over 3058938.68 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:23:15,706 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3037873.3333333335, ans=0.125
2023-11-24 23:23:21,330 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=3037940.0, ans=0.125
2023-11-24 23:23:27,426 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=3037940.0, ans=0.0
2023-11-24 23:23:30,773 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455700
2023-11-24 23:23:58,233 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module2.whiten, num_groups=1, num_channels=384, metric=2.82 vs. limit=15.0
2023-11-24 23:24:00,331 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=3038140.0, ans=0.0
2023-11-24 23:24:08,365 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=3038140.0, ans=0.0
2023-11-24 23:24:11,715 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10850, loss[loss=0.07148, simple_loss=0.0982, pruned_loss=0.01387, audio_tagging_loss=0.008509, over 15113.00 frames. ], tot_loss[loss=0.06668, simple_loss=0.09045, pruned_loss=0.01268, audio_tagging_loss=0.008776, over 3060481.67 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:24:19,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3038206.6666666665, ans=0.1
2023-11-24 23:24:23,953 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=3038273.3333333335, ans=0.0
2023-11-24 23:24:27,936 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3038273.3333333335, ans=0.1
2023-11-24 23:24:32,485 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455750
2023-11-24 23:24:59,987 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.004e+01 8.552e+01 9.146e+01 9.923e+01 1.188e+02, threshold=1.829e+02, percent-clipped=0.0
2023-11-24 23:25:07,137 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/XMxq2pgttuY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:25:13,543 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10900, loss[loss=0.08261, simple_loss=0.114, pruned_loss=0.01748, audio_tagging_loss=0.00814, over 14904.00 frames. ], tot_loss[loss=0.06675, simple_loss=0.09075, pruned_loss=0.01258, audio_tagging_loss=0.0088, over 3059942.81 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:25:23,188 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3038540.0, ans=0.125
2023-11-24 23:25:26,818 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=3038606.6666666665, ans=0.125
2023-11-24 23:25:35,536 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455800
2023-11-24 23:25:36,098 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.17 vs. limit=15.0
2023-11-24 23:25:46,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3038673.3333333335, ans=0.1
2023-11-24 23:26:05,196 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass.skip_rate, batch_count=3038806.6666666665, ans=0.09899494936611666
2023-11-24 23:26:07,515 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=3038806.6666666665, ans=0.0
2023-11-24 23:26:13,646 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:26:16,211 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 10950, loss[loss=0.0628, simple_loss=0.08366, pruned_loss=0.01386, audio_tagging_loss=0.007107, over 14212.00 frames. ], tot_loss[loss=0.06738, simple_loss=0.09176, pruned_loss=0.0127, audio_tagging_loss=0.008799, over 3057919.35 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:26:18,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=3038873.3333333335, ans=0.5
2023-11-24 23:26:21,781 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.attention_skip_rate, batch_count=3038873.3333333335, ans=0.0
2023-11-24 23:26:26,370 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.72 vs. limit=22.5
2023-11-24 23:26:31,675 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3038940.0, ans=0.1
2023-11-24 23:26:35,401 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.62 vs. limit=15.0
2023-11-24 23:26:37,327 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455850
2023-11-24 23:26:45,762 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=3039006.6666666665, ans=0.2
2023-11-24 23:27:05,051 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.217e+01 8.762e+01 9.251e+01 9.897e+01 1.242e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 23:27:07,708 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=3039140.0, ans=0.125
2023-11-24 23:27:18,792 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11000, loss[loss=0.06505, simple_loss=0.08535, pruned_loss=0.01143, audio_tagging_loss=0.01094, over 15193.00 frames. ], tot_loss[loss=0.06724, simple_loss=0.09166, pruned_loss=0.01265, audio_tagging_loss=0.008756, over 3055530.68 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:27:24,383 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.01 vs. limit=15.0
2023-11-24 23:27:26,054 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/h6R5rMXN6pY_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:27:39,721 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455900
2023-11-24 23:27:39,967 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3039273.3333333335, ans=0.125
2023-11-24 23:27:40,268 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=512, metric=19.46 vs. limit=22.5
2023-11-24 23:27:41,282 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.26 vs. limit=15.0
2023-11-24 23:27:42,817 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.29 vs. limit=12.0
2023-11-24 23:28:05,138 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=3039406.6666666665, ans=0.2
2023-11-24 23:28:10,444 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.25 vs. limit=6.0
2023-11-24 23:28:11,652 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=8.80 vs. limit=22.5
2023-11-24 23:28:12,348 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=3039473.3333333335, ans=0.0
2023-11-24 23:28:20,382 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11050, loss[loss=0.05005, simple_loss=0.06347, pruned_loss=0.008652, audio_tagging_loss=0.009662, over 15482.00 frames. ], tot_loss[loss=0.067, simple_loss=0.09127, pruned_loss=0.01252, audio_tagging_loss=0.008849, over 3061261.75 frames. ], batch size: 62, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:28:35,669 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.whiten, num_groups=1, num_channels=512, metric=8.48 vs. limit=12.0
2023-11-24 23:28:42,228 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 455950
2023-11-24 23:28:49,434 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=10.48 vs. limit=15.0
2023-11-24 23:28:53,816 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3039673.3333333335, ans=0.125
2023-11-24 23:29:08,951 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.264e+01 8.642e+01 9.315e+01 1.003e+02 1.536e+02, threshold=1.863e+02, percent-clipped=0.0
2023-11-24 23:29:22,460 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11100, loss[loss=0.05188, simple_loss=0.0695, pruned_loss=0.007308, audio_tagging_loss=0.009825, over 14211.00 frames. ], tot_loss[loss=0.06624, simple_loss=0.08969, pruned_loss=0.01238, audio_tagging_loss=0.009021, over 3054862.98 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:29:37,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=3039940.0, ans=0.0
2023-11-24 23:29:44,640 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456000
2023-11-24 23:29:46,117 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/checkpoint-456000.pt
2023-11-24 23:29:53,828 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3040006.6666666665, ans=0.125
2023-11-24 23:29:55,242 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.2.encoder.layers.2.conv_module1.whiten, num_groups=1, num_channels=384, metric=2.97 vs. limit=15.0
2023-11-24 23:29:57,347 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=3040006.6666666665, ans=0.2
2023-11-24 23:30:02,516 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3040006.6666666665, ans=0.125
2023-11-24 23:30:25,503 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.58 vs. limit=15.0
2023-11-24 23:30:30,069 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11150, loss[loss=0.06565, simple_loss=0.09513, pruned_loss=0.01003, audio_tagging_loss=0.008052, over 15375.00 frames. ], tot_loss[loss=0.06659, simple_loss=0.08993, pruned_loss=0.01244, audio_tagging_loss=0.009181, over 3052596.39 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:30:36,249 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=3040206.6666666665, ans=0.125
2023-11-24 23:30:44,596 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3040273.3333333335, ans=0.0
2023-11-24 23:30:50,904 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456050
2023-11-24 23:30:55,704 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3040340.0, ans=0.1
2023-11-24 23:30:57,993 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=3040340.0, ans=0.125
2023-11-24 23:30:59,441 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=3040340.0, ans=0.2
2023-11-24 23:31:18,466 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.284e+01 8.591e+01 9.320e+01 9.957e+01 1.253e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 23:31:28,086 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=3040473.3333333335, ans=0.2
2023-11-24 23:31:31,386 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11200, loss[loss=0.04284, simple_loss=0.05348, pruned_loss=0.004142, audio_tagging_loss=0.01195, over 15318.00 frames. ], tot_loss[loss=0.06687, simple_loss=0.09025, pruned_loss=0.01249, audio_tagging_loss=0.009248, over 3046670.04 frames. ], batch size: 61, lr: 1.77e-03, grad_scale: 32.0
2023-11-24 23:31:35,696 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=3040540.0, ans=0.125
2023-11-24 23:31:40,496 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=3040540.0, ans=0.125
2023-11-24 23:31:53,234 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456100
2023-11-24 23:32:20,473 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.balancer_ff3.min_abs, batch_count=3040806.6666666665, ans=0.2
2023-11-24 23:32:20,628 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.95 vs. limit=15.0
2023-11-24 23:32:33,944 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11250, loss[loss=0.07815, simple_loss=0.1136, pruned_loss=0.0132, audio_tagging_loss=0.008183, over 15373.00 frames. ], tot_loss[loss=0.06622, simple_loss=0.08929, pruned_loss=0.01234, audio_tagging_loss=0.00923, over 3046228.48 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:32:55,778 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456150
2023-11-24 23:33:04,380 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3041006.6666666665, ans=0.1
2023-11-24 23:33:14,512 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_skip_rate, batch_count=3041073.3333333335, ans=0.0
2023-11-24 23:33:23,493 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.642e+01 8.456e+01 9.058e+01 9.747e+01 1.230e+02, threshold=1.812e+02, percent-clipped=0.0
2023-11-24 23:33:35,954 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11300, loss[loss=0.05344, simple_loss=0.06161, pruned_loss=0.009392, audio_tagging_loss=0.01324, over 14944.00 frames. ], tot_loss[loss=0.06624, simple_loss=0.0893, pruned_loss=0.01252, audio_tagging_loss=0.009072, over 3044011.70 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:33:47,484 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=3041273.3333333335, ans=0.125
2023-11-24 23:33:49,964 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3041273.3333333335, ans=0.1
2023-11-24 23:33:56,680 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456200
2023-11-24 23:34:00,241 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.scale_min, batch_count=3041340.0, ans=0.2
2023-11-24 23:34:11,969 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3041406.6666666665, ans=0.0
2023-11-24 23:34:37,459 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11350, loss[loss=0.06095, simple_loss=0.08319, pruned_loss=0.01015, audio_tagging_loss=0.0092, over 15613.00 frames. ], tot_loss[loss=0.06664, simple_loss=0.09036, pruned_loss=0.01255, audio_tagging_loss=0.008913, over 3046055.08 frames. ], batch size: 62, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:34:38,728 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=3041540.0, ans=0.125
2023-11-24 23:34:38,890 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3041540.0, ans=0.0
2023-11-24 23:34:39,264 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=8.87 vs. limit=15.0
2023-11-24 23:34:58,559 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456250
2023-11-24 23:35:01,810 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3041673.3333333335, ans=0.1
2023-11-24 23:35:20,321 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3041740.0, ans=0.125
2023-11-24 23:35:21,399 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3041740.0, ans=0.125
2023-11-24 23:35:25,065 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.nonlin_attention.balancer.prob, batch_count=3041740.0, ans=0.125
2023-11-24 23:35:28,353 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.570e+01 8.600e+01 9.248e+01 9.972e+01 1.236e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 23:35:32,719 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3041806.6666666665, ans=0.125
2023-11-24 23:35:33,723 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=3041806.6666666665, ans=0.125
2023-11-24 23:35:39,399 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11400, loss[loss=0.07333, simple_loss=0.1057, pruned_loss=0.01498, audio_tagging_loss=0.005498, over 14407.00 frames. ], tot_loss[loss=0.06636, simple_loss=0.09011, pruned_loss=0.0125, audio_tagging_loss=0.0088, over 3042026.04 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:35:43,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=3041873.3333333335, ans=10.0
2023-11-24 23:36:00,240 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456300
2023-11-24 23:36:01,882 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3041940.0, ans=0.125
2023-11-24 23:36:11,772 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3042006.6666666665, ans=0.1
2023-11-24 23:36:22,184 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=3042073.3333333335, ans=10.0
2023-11-24 23:36:23,115 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3042073.3333333335, ans=0.125
2023-11-24 23:36:27,075 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.38 vs. limit=15.0
2023-11-24 23:36:27,974 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3042140.0, ans=0.0
2023-11-24 23:36:29,283 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=3042140.0, ans=0.0
2023-11-24 23:36:39,228 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3042140.0, ans=0.125
2023-11-24 23:36:39,292 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3042140.0, ans=0.0
2023-11-24 23:36:41,457 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11450, loss[loss=0.07953, simple_loss=0.1008, pruned_loss=0.01687, audio_tagging_loss=0.01225, over 15743.00 frames. ], tot_loss[loss=0.0669, simple_loss=0.09086, pruned_loss=0.01277, audio_tagging_loss=0.008698, over 3041759.24 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:36:52,967 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=512, metric=7.97 vs. limit=15.0
2023-11-24 23:36:53,796 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3042273.3333333335, ans=0.125
2023-11-24 23:37:02,364 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456350
2023-11-24 23:37:06,662 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3042340.0, ans=0.125
2023-11-24 23:37:23,639 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3042406.6666666665, ans=0.125
2023-11-24 23:37:26,089 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.balancer2.prob, batch_count=3042406.6666666665, ans=0.125
2023-11-24 23:37:31,586 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.336e+01 8.517e+01 9.262e+01 1.018e+02 1.314e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 23:37:34,186 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3042473.3333333335, ans=0.125
2023-11-24 23:37:42,295 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11500, loss[loss=0.06389, simple_loss=0.08712, pruned_loss=0.01115, audio_tagging_loss=0.00918, over 15510.00 frames. ], tot_loss[loss=0.06713, simple_loss=0.09132, pruned_loss=0.01285, audio_tagging_loss=0.008623, over 3040276.08 frames. ], batch size: 59, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:38:03,732 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456400
2023-11-24 23:38:10,942 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=3042673.3333333335, ans=0.05
2023-11-24 23:38:14,290 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=3042673.3333333335, ans=0.09899494936611666
2023-11-24 23:38:44,377 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11550, loss[loss=0.06997, simple_loss=0.101, pruned_loss=0.01231, audio_tagging_loss=0.00715, over 15151.00 frames. ], tot_loss[loss=0.06678, simple_loss=0.09101, pruned_loss=0.01269, audio_tagging_loss=0.008584, over 3046774.94 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:39:05,147 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456450
2023-11-24 23:39:13,580 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3043006.6666666665, ans=0.1
2023-11-24 23:39:14,685 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3043006.6666666665, ans=0.125
2023-11-24 23:39:19,211 WARNING [train_asr.py:1462] (0/4) Exclude cut with ID unbalanced/NeYOsnhOi4k_0.000_1.000.wav from training. Number of frames (before subsampling): 100. Number of frames (after subsampling): 23. Text: Dummy text added as a place holder. Please ignore this if possible. Tokens: ['▁D', 'ummy', '▁', 'text', '▁', 'added', '▁', 'as', '▁', 'a', '▁', 'place', '▁', 'holder.', '▁P', 'lease', '▁', 'ignore', '▁', 'this', '▁', 'if', '▁', 'possible']. Number of tokens: 24
2023-11-24 23:39:20,793 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward3.hidden_balancer.prob, batch_count=3043073.3333333335, ans=0.125
2023-11-24 23:39:24,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=3043073.3333333335, ans=0.5
2023-11-24 23:39:31,251 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=3043073.3333333335, ans=0.2
2023-11-24 23:39:34,179 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.52 vs. limit=22.5
2023-11-24 23:39:34,491 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.061e+01 8.770e+01 9.475e+01 1.009e+02 1.231e+02, threshold=1.895e+02, percent-clipped=0.0
2023-11-24 23:39:45,836 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11600, loss[loss=0.07061, simple_loss=0.09197, pruned_loss=0.0142, audio_tagging_loss=0.01043, over 14935.00 frames. ], tot_loss[loss=0.06699, simple_loss=0.09131, pruned_loss=0.01275, audio_tagging_loss=0.008592, over 3044287.47 frames. ], batch size: 57, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:39:47,340 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3043206.6666666665, ans=0.125
2023-11-24 23:40:00,716 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=384, metric=3.77 vs. limit=15.0
2023-11-24 23:40:04,194 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=3043273.3333333335, ans=0.125
2023-11-24 23:40:06,295 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456500
2023-11-24 23:40:15,859 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=512, metric=2.51 vs. limit=15.0
2023-11-24 23:40:25,367 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.79 vs. limit=12.0
2023-11-24 23:40:40,923 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.feed_forward1.out_proj.dropout_p, batch_count=3043473.3333333335, ans=0.1
2023-11-24 23:40:46,568 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11650, loss[loss=0.06381, simple_loss=0.08913, pruned_loss=0.01238, audio_tagging_loss=0.006863, over 15123.00 frames. ], tot_loss[loss=0.06646, simple_loss=0.09008, pruned_loss=0.01266, audio_tagging_loss=0.008753, over 3037951.09 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:41:01,127 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=3043606.6666666665, ans=0.035
2023-11-24 23:41:03,669 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3043606.6666666665, ans=0.1
2023-11-24 23:41:08,025 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456550
2023-11-24 23:41:15,703 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=3043673.3333333335, ans=0.0
2023-11-24 23:41:18,293 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=3043673.3333333335, ans=0.0
2023-11-24 23:41:36,903 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.669e+01 8.725e+01 9.251e+01 1.014e+02 1.338e+02, threshold=1.850e+02, percent-clipped=0.0
2023-11-24 23:41:38,388 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=3043806.6666666665, ans=0.0
2023-11-24 23:41:39,575 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.out_proj.dropout_p, batch_count=3043806.6666666665, ans=0.1
2023-11-24 23:41:42,983 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3043806.6666666665, ans=0.125
2023-11-24 23:41:43,085 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.3.encoder.layers.2.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:41:48,074 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11700, loss[loss=0.0637, simple_loss=0.08686, pruned_loss=0.009301, audio_tagging_loss=0.01097, over 14254.00 frames. ], tot_loss[loss=0.06658, simple_loss=0.09012, pruned_loss=0.01267, audio_tagging_loss=0.008848, over 3045954.50 frames. ], batch size: 54, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:42:09,641 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456600
2023-11-24 23:42:11,304 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=3043940.0, ans=0.125
2023-11-24 23:42:13,860 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3044006.6666666665, ans=0.0
2023-11-24 23:42:17,824 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.97 vs. limit=15.0
2023-11-24 23:42:22,280 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=9.56 vs. limit=15.0
2023-11-24 23:42:24,280 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=3044073.3333333335, ans=0.2
2023-11-24 23:42:44,554 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module1.balancer2.prob, batch_count=3044140.0, ans=0.125
2023-11-24 23:42:50,844 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11750, loss[loss=0.0471, simple_loss=0.0645, pruned_loss=0.008656, audio_tagging_loss=0.00619, over 14592.00 frames. ], tot_loss[loss=0.06708, simple_loss=0.09079, pruned_loss=0.01277, audio_tagging_loss=0.008915, over 3053436.88 frames. ], batch size: 55, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:42:52,287 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3044206.6666666665, ans=0.0
2023-11-24 23:42:57,100 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.feed_forward1.hidden_balancer.prob, batch_count=3044206.6666666665, ans=0.125
2023-11-24 23:43:00,621 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=3044206.6666666665, ans=0.0
2023-11-24 23:43:04,121 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.bypass.scale_min, batch_count=3044273.3333333335, ans=0.2
2023-11-24 23:43:11,140 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456650
2023-11-24 23:43:17,164 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3044340.0, ans=0.1
2023-11-24 23:43:22,838 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=512, metric=17.29 vs. limit=22.5
2023-11-24 23:43:23,697 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=3044340.0, ans=0.125
2023-11-24 23:43:33,630 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=3044406.6666666665, ans=0.0
2023-11-24 23:43:42,312 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.291e+01 8.689e+01 9.260e+01 1.011e+02 1.469e+02, threshold=1.852e+02, percent-clipped=0.0
2023-11-24 23:43:51,911 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11800, loss[loss=0.08491, simple_loss=0.1177, pruned_loss=0.01836, audio_tagging_loss=0.007726, over 15768.00 frames. ], tot_loss[loss=0.06656, simple_loss=0.08974, pruned_loss=0.01265, audio_tagging_loss=0.009035, over 3045213.57 frames. ], batch size: 60, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:44:12,672 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456700
2023-11-24 23:44:35,732 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=10.46 vs. limit=15.0
2023-11-24 23:44:38,041 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.67 vs. limit=10.0
2023-11-24 23:44:41,064 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=3044806.6666666665, ans=0.09899494936611666
2023-11-24 23:44:44,589 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=3044806.6666666665, ans=0.0
2023-11-24 23:44:53,002 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11850, loss[loss=0.05824, simple_loss=0.07628, pruned_loss=0.01122, audio_tagging_loss=0.008879, over 15132.00 frames. ], tot_loss[loss=0.06613, simple_loss=0.08909, pruned_loss=0.0125, audio_tagging_loss=0.009077, over 3049197.56 frames. ], batch size: 58, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:45:00,455 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.feed_forward2.hidden_balancer.prob, batch_count=3044873.3333333335, ans=0.125
2023-11-24 23:45:14,839 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456750
2023-11-24 23:45:23,285 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=3045006.6666666665, ans=0.2
2023-11-24 23:45:44,598 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.068e+01 8.737e+01 9.408e+01 9.939e+01 1.281e+02, threshold=1.882e+02, percent-clipped=0.0
2023-11-24 23:45:54,849 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11900, loss[loss=0.06946, simple_loss=0.1025, pruned_loss=0.01176, audio_tagging_loss=0.006456, over 14886.00 frames. ], tot_loss[loss=0.06646, simple_loss=0.08959, pruned_loss=0.01262, audio_tagging_loss=0.009042, over 3036911.33 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:45:58,333 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=288, metric=7.20 vs. limit=10.0
2023-11-24 23:46:05,162 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3045206.6666666665, ans=0.125
2023-11-24 23:46:12,370 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=3045273.3333333335, ans=0.125
2023-11-24 23:46:15,661 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456800
2023-11-24 23:46:15,934 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3045273.3333333335, ans=0.125
2023-11-24 23:46:51,957 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=3045473.3333333335, ans=0.1
2023-11-24 23:46:56,505 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 11950, loss[loss=0.07374, simple_loss=0.09858, pruned_loss=0.01446, audio_tagging_loss=0.009986, over 14613.00 frames. ], tot_loss[loss=0.06593, simple_loss=0.08867, pruned_loss=0.01243, audio_tagging_loss=0.009163, over 3036395.47 frames. ], batch size: 52, lr: 1.77e-03, grad_scale: 8.0
2023-11-24 23:46:56,913 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=3045540.0, ans=0.0
2023-11-24 23:47:11,010 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=3045606.6666666665, ans=0.125
2023-11-24 23:47:17,233 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456850
2023-11-24 23:47:31,941 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer1.prob, batch_count=3045740.0, ans=0.125
2023-11-24 23:47:41,986 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=3045740.0, ans=0.05
2023-11-24 23:47:43,387 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.3.encoder.layers.3.feed_forward3.out_whiten, num_groups=1, num_channels=512, metric=11.05 vs. limit=15.0
2023-11-24 23:47:47,343 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.445e+01 8.455e+01 9.152e+01 9.873e+01 1.157e+02, threshold=1.830e+02, percent-clipped=0.0
2023-11-24 23:47:50,529 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.55 vs. limit=10.0
2023-11-24 23:47:56,420 INFO [train_asr.py:1221] (0/4) Epoch 38, batch 12000, loss[loss=0.07588, simple_loss=0.1062, pruned_loss=0.01422, audio_tagging_loss=0.008565, over 14163.00 frames. ], tot_loss[loss=0.06665, simple_loss=0.08955, pruned_loss=0.0127, audio_tagging_loss=0.009174, over 3037313.69 frames. ], batch size: 56, lr: 1.77e-03, grad_scale: 16.0
2023-11-24 23:47:56,423 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 23:48:40,307 INFO [train_asr.py:1253] (0/4) Epoch 38, validation: loss=0.05738, simple_loss=0.0508, pruned_loss=0.005195, audio_tagging_loss=0.02678, over 4681554.00 frames. 
2023-11-24 23:48:40,308 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 23:48:42,841 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3045873.3333333335, ans=0.125
2023-11-24 23:48:45,148 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=3045873.3333333335, ans=0.0
2023-11-24 23:48:59,668 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456900
2023-11-24 23:49:06,726 INFO [checkpoint.py:75] (0/4) Saving checkpoint to multi_KD/exp_train_asr_full_libri1_do_audio_tagging1_as_unbalanced_scale1.0/epoch-38.pt
2023-11-24 23:49:37,961 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 0, loss[loss=0.07789, simple_loss=0.08696, pruned_loss=0.009117, audio_tagging_loss=0.02529, over 15170.00 frames. ], tot_loss[loss=0.07789, simple_loss=0.08696, pruned_loss=0.009117, audio_tagging_loss=0.02529, over 15170.00 frames. ], batch size: 58, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:49:37,964 INFO [train_asr.py:1244] (0/4) Computing validation loss
2023-11-24 23:50:14,453 INFO [train_asr.py:1253] (0/4) Epoch 39, validation: loss=0.0578, simple_loss=0.05083, pruned_loss=0.005244, audio_tagging_loss=0.02714, over 4681554.00 frames. 
2023-11-24 23:50:14,454 INFO [train_asr.py:1254] (0/4) Maximum memory allocated so far is 26396MB
2023-11-24 23:50:14,722 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.2.self_attn_weights.pos_emb_skip_rate, batch_count=3046020.0, ans=0.0
2023-11-24 23:50:14,743 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3046020.0, ans=0.125
2023-11-24 23:50:32,759 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.ff3_skip_rate, batch_count=3046086.6666666665, ans=0.0
2023-11-24 23:50:37,437 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3046153.3333333335, ans=0.1
2023-11-24 23:50:44,171 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=3046153.3333333335, ans=0.0
2023-11-24 23:50:51,665 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.26 vs. limit=12.0
2023-11-24 23:51:08,350 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.21 vs. limit=15.0
2023-11-24 23:51:09,892 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 456950
2023-11-24 23:51:11,364 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.conv_module1.balancer2.prob, batch_count=3046286.6666666665, ans=0.125
2023-11-24 23:51:15,704 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 50, loss[loss=0.08434, simple_loss=0.1132, pruned_loss=0.0167, audio_tagging_loss=0.01103, over 15523.00 frames. ], tot_loss[loss=0.07568, simple_loss=0.09209, pruned_loss=0.01267, audio_tagging_loss=0.01697, over 686606.58 frames. ], batch size: 56, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:51:18,694 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.2.feed_forward3.out_whiten, num_groups=1, num_channels=384, metric=11.55 vs. limit=15.0
2023-11-24 23:51:30,874 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=3046420.0, ans=0.2
2023-11-24 23:51:39,932 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 8.015e+01 9.557e+01 1.022e+02 1.093e+02 1.810e+02, threshold=2.044e+02, percent-clipped=0.0
2023-11-24 23:51:59,845 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.bypass.skip_rate, batch_count=3046553.3333333335, ans=0.07
2023-11-24 23:52:11,428 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457000
2023-11-24 23:52:16,247 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3046620.0, ans=0.1
2023-11-24 23:52:18,486 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 100, loss[loss=0.0744, simple_loss=0.09653, pruned_loss=0.01219, audio_tagging_loss=0.01395, over 14580.00 frames. ], tot_loss[loss=0.07435, simple_loss=0.09188, pruned_loss=0.01217, audio_tagging_loss=0.01624, over 1205874.49 frames. ], batch size: 56, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:52:21,326 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=3046686.6666666665, ans=0.0
2023-11-24 23:52:38,462 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3046753.3333333335, ans=0.125
2023-11-24 23:52:38,602 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=15.87 vs. limit=15.0
2023-11-24 23:53:14,562 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457050
2023-11-24 23:53:21,498 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 150, loss[loss=0.08578, simple_loss=0.1152, pruned_loss=0.01671, audio_tagging_loss=0.01145, over 14399.00 frames. ], tot_loss[loss=0.07128, simple_loss=0.08952, pruned_loss=0.01193, audio_tagging_loss=0.01459, over 1618580.66 frames. ], batch size: 52, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:53:24,564 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.67 vs. limit=22.5
2023-11-24 23:53:34,013 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=3047086.6666666665, ans=0.0
2023-11-24 23:53:35,815 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.97 vs. limit=22.5
2023-11-24 23:53:43,612 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=3047086.6666666665, ans=0.0
2023-11-24 23:53:45,773 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.772e+01 8.928e+01 9.394e+01 1.010e+02 1.309e+02, threshold=1.879e+02, percent-clipped=0.0
2023-11-24 23:54:06,692 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.35 vs. limit=15.0
2023-11-24 23:54:18,136 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457100
2023-11-24 23:54:21,886 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:54:24,059 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 200, loss[loss=0.06674, simple_loss=0.08978, pruned_loss=0.01507, audio_tagging_loss=0.006783, over 16247.00 frames. ], tot_loss[loss=0.06991, simple_loss=0.08984, pruned_loss=0.01222, audio_tagging_loss=0.01277, over 1932448.81 frames. ], batch size: 62, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:54:44,227 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.15 vs. limit=22.5
2023-11-24 23:55:01,545 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.3.attention_skip_rate, batch_count=3047553.3333333335, ans=0.0
2023-11-24 23:55:19,674 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457150
2023-11-24 23:55:21,291 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=384, metric=11.24 vs. limit=15.0
2023-11-24 23:55:26,008 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 250, loss[loss=0.06281, simple_loss=0.08408, pruned_loss=0.0121, audio_tagging_loss=0.008673, over 16100.00 frames. ], tot_loss[loss=0.0694, simple_loss=0.09065, pruned_loss=0.01252, audio_tagging_loss=0.01156, over 2177459.10 frames. ], batch size: 60, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:55:28,797 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=3047686.6666666665, ans=0.2
2023-11-24 23:55:30,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=3047686.6666666665, ans=0.2
2023-11-24 23:55:36,960 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3047753.3333333335, ans=0.0
2023-11-24 23:55:48,063 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=9.43 vs. limit=15.0
2023-11-24 23:55:50,701 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.067e+01 8.824e+01 9.475e+01 1.039e+02 1.300e+02, threshold=1.895e+02, percent-clipped=0.0
2023-11-24 23:55:59,789 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3047820.0, ans=0.125
2023-11-24 23:56:08,316 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=3047886.6666666665, ans=0.05
2023-11-24 23:56:16,992 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=3047953.3333333335, ans=0.025
2023-11-24 23:56:21,585 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457200
2023-11-24 23:56:28,287 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 300, loss[loss=0.07037, simple_loss=0.1024, pruned_loss=0.01407, audio_tagging_loss=0.00512, over 14989.00 frames. ], tot_loss[loss=0.06844, simple_loss=0.09046, pruned_loss=0.01251, audio_tagging_loss=0.01071, over 2372209.72 frames. ], batch size: 56, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:56:28,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=3048020.0, ans=0.0
2023-11-24 23:56:33,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=3048020.0, ans=0.0
2023-11-24 23:56:50,908 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-24 23:56:53,391 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3048153.3333333335, ans=0.125
2023-11-24 23:57:08,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=3048220.0, ans=0.2
2023-11-24 23:57:12,801 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=3048220.0, ans=0.125
2023-11-24 23:57:15,820 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=3048220.0, ans=0.125
2023-11-24 23:57:21,230 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3048286.6666666665, ans=0.1
2023-11-24 23:57:24,685 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457250
2023-11-24 23:57:25,167 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.99 vs. limit=6.0
2023-11-24 23:57:25,931 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=3048286.6666666665, ans=0.0
2023-11-24 23:57:26,123 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3048286.6666666665, ans=0.0
2023-11-24 23:57:30,467 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 350, loss[loss=0.07168, simple_loss=0.09548, pruned_loss=0.01536, audio_tagging_loss=0.008582, over 16026.00 frames. ], tot_loss[loss=0.06762, simple_loss=0.08993, pruned_loss=0.01241, audio_tagging_loss=0.01025, over 2523210.04 frames. ], batch size: 58, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:57:38,409 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.conv_module2.balancer1.prob, batch_count=3048353.3333333335, ans=0.125
2023-11-24 23:57:39,530 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3048353.3333333335, ans=0.125
2023-11-24 23:57:39,627 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=3048353.3333333335, ans=0.0
2023-11-24 23:57:48,177 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=3048420.0, ans=0.125
2023-11-24 23:57:56,165 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.520e+01 8.715e+01 9.321e+01 9.899e+01 1.393e+02, threshold=1.864e+02, percent-clipped=0.0
2023-11-24 23:58:01,147 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=3048486.6666666665, ans=0.0
2023-11-24 23:58:09,182 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3048553.3333333335, ans=0.1
2023-11-24 23:58:20,566 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.max_abs, batch_count=3048620.0, ans=10.0
2023-11-24 23:58:25,978 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457300
2023-11-24 23:58:32,354 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 400, loss[loss=0.08557, simple_loss=0.1185, pruned_loss=0.01918, audio_tagging_loss=0.007147, over 15643.00 frames. ], tot_loss[loss=0.068, simple_loss=0.09094, pruned_loss=0.01267, audio_tagging_loss=0.009864, over 2640823.12 frames. ], batch size: 56, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:59:06,234 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.48 vs. limit=10.0
2023-11-24 23:59:10,008 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=3048886.6666666665, ans=0.2
2023-11-24 23:59:14,559 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=3048886.6666666665, ans=0.0
2023-11-24 23:59:27,098 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=3048953.3333333335, ans=0.125
2023-11-24 23:59:27,950 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457350
2023-11-24 23:59:28,216 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module2.balancer2.prob, batch_count=3048953.3333333335, ans=0.125
2023-11-24 23:59:33,731 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 450, loss[loss=0.06522, simple_loss=0.09438, pruned_loss=0.01095, audio_tagging_loss=0.007081, over 14850.00 frames. ], tot_loss[loss=0.06706, simple_loss=0.09039, pruned_loss=0.01229, audio_tagging_loss=0.009577, over 2727001.52 frames. ], batch size: 55, lr: 1.75e-03, grad_scale: 32.0
2023-11-24 23:59:52,896 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=3049086.6666666665, ans=0.125
2023-11-25 00:00:00,320 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 7.663e+01 8.552e+01 9.288e+01 9.905e+01 1.638e+02, threshold=1.858e+02, percent-clipped=0.0
2023-11-25 00:00:07,195 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=3049153.3333333335, ans=0.125
2023-11-25 00:00:30,632 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457400
2023-11-25 00:00:36,783 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 500, loss[loss=0.06431, simple_loss=0.09341, pruned_loss=0.009117, audio_tagging_loss=0.008486, over 14748.00 frames. ], tot_loss[loss=0.0671, simple_loss=0.09064, pruned_loss=0.01244, audio_tagging_loss=0.009334, over 2802507.20 frames. ], batch size: 54, lr: 1.74e-03, grad_scale: 32.0
2023-11-25 00:00:37,207 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=3049353.3333333335, ans=0.125
2023-11-25 00:00:42,168 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.61 vs. limit=22.5
2023-11-25 00:00:46,893 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=3049353.3333333335, ans=0.2
2023-11-25 00:00:56,991 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-25 00:01:11,739 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3049486.6666666665, ans=0.1
2023-11-25 00:01:32,563 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457450
2023-11-25 00:01:35,719 INFO [scaling.py:1118] (0/4) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00
2023-11-25 00:01:39,083 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 550, loss[loss=0.06421, simple_loss=0.09179, pruned_loss=0.009506, audio_tagging_loss=0.008804, over 14507.00 frames. ], tot_loss[loss=0.06751, simple_loss=0.09108, pruned_loss=0.01273, audio_tagging_loss=0.009241, over 2857163.85 frames. ], batch size: 54, lr: 1.74e-03, grad_scale: 32.0
2023-11-25 00:01:39,453 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.2.conv_module1.balancer1.prob, batch_count=3049686.6666666665, ans=0.125
2023-11-25 00:01:49,890 INFO [scaling.py:1022] (0/4) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=384, metric=9.44 vs. limit=15.0
2023-11-25 00:02:03,689 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.3.encoder.layers.2.bypass_mid.scale_min, batch_count=3049820.0, ans=0.2
2023-11-25 00:02:05,697 INFO [optim.py:476] (0/4) Clipping_scale=2.0, grad-norm quartiles 6.951e+01 8.616e+01 9.440e+01 1.005e+02 1.281e+02, threshold=1.888e+02, percent-clipped=0.0
2023-11-25 00:02:35,478 INFO [model.py:792] (0/4) Freeze_encoder: False; Current batch idx: 457500
2023-11-25 00:02:41,281 INFO [train_asr.py:1221] (0/4) Epoch 39, batch 600, loss[loss=0.07421, simple_loss=0.09538, pruned_loss=0.01749, audio_tagging_loss=0.009028, over 14755.00 frames. ], tot_loss[loss=0.06761, simple_loss=0.09132, pruned_loss=0.01271, audio_tagging_loss=0.009236, over 2903554.15 frames. ], batch size: 57, lr: 1.74e-03, grad_scale: 32.0
2023-11-25 00:02:45,859 INFO [scaling.py:213] (0/4) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3050020.0, ans=0.1